# Bloom filter (strange behavior) **URL:** \ **Category:** troubleshooting\ **Tags:** garbage-collector\ **Created:** [April 30, 2024, 10:18am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028 "2024-04-30T10:18:30Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![elek](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/elek/32/9189_2.png) [@elek](https://forum.storj.io/u/elek)\ **Post date:** [May 7, 2024, 8:42am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/21 "2024-05-07T08:42:22Z")
I couldn’t reproduce the problem, and based on the code, it is possible only if it fails after the resume. Can you please check the logs for related errors? ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 7, 2024, 9:07am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/22 "2024-05-07T09:07:03Z")
> [@elek](#): > > I couldn’t reproduce the problem, and based on the code, it is possible only if it fails after the resume. Are you reproduce on Windows 10? So there is no problem? It seemed to us? 😀 > [@elek](#): > > Can you please check the logs for related errors? Unfortunately, there are no more errors in the log. Just what I already wrote ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [May 7, 2024, 9:16am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/23 "2024-05-07T09:16:03Z")
On one node I am seeing currently 2 files in the retain folder one with name `ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa` something and `v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa` something. I don’t know what the node is currently doing but should I restart it and see if these files get deleted? ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 7, 2024, 9:35am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/24 "2024-05-07T09:35:39Z")
Restart the node and see if the files remain in the /retain folder or not ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [May 7, 2024, 9:43am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/25 "2024-05-07T09:43:42Z")
> [@pdeline06](#): > > Restart the node and see Ok I have restarted it. The files are still there after restart. Both files have date 06/05. So I don’t know if they have been processed already, waiting to get processed or are under processing right now. ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 7, 2024, 10:11am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/26 "2024-05-07T10:11:01Z")
What windows are you use? ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [May 7, 2024, 10:12am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/27 "2024-05-07T10:12:41Z")
The nodes are on Linux. ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 7, 2024, 10:20am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/28 "2024-05-07T10:20:27Z")
😁 😁 😁 The problem only affects WINDOWS! ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 7, 2024, 10:26am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/29 "2024-05-07T10:26:13Z")
I recorded a video of how this happens! Link to YouTube [![](https://img.youtube.com/vi/wkM3ndNHLZU/hqdefault.jpg "ice video 20240507 131631") ](https://www.youtube.com/watch?v=wkM3ndNHLZU) ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [May 7, 2024, 10:47am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/30 "2024-05-07T10:47:13Z")
You should keep the log on info with custom log level for noise. You will see the start and finish of retain process. ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 7, 2024, 11:20am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/31 "2024-05-07T11:20:54Z")
I have log.level: info **When I start a node with a bloom filter file in the /retain folder:** INFO retain Prepared to run a Retain request. {“cachePath”: “X:\!STORJ913\WORKDIR/retain”, “Created Before”: “2024-04-30T20:59:59+03:00”, “Filter Size”: 3300354, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”} **The node runs for 5 minutes. Then I stop the node. (net stop storagenode):** INFO Stop/Shutdown request received. INFO lazyfilewalker.gc-filewalker subprocess exited with status {“satelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “status”: 1, “error”: “exit status 1”} ERROR pieces lazyfilewalker failed {“error”: “lazyfilewalker: exit status 1”, “errorVerbose”: “lazyfilewalker: exit status 1\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(\*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(\*Supervisor).WalkSatellitePiecesToTrash:160\n\tstorj.io/storj/storagenode/pieces.(\*Store).WalkSatellitePiecesToTrash:578\n\tstorj.io/storj/storagenode/retain.(\*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(\*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78”} ERROR filewalker failed to get progress from database {“error”: “gc\_filewalker\_progress\_db: context canceled”, “errorVerbose”: “gc\_filewalker\_progress\_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(\*gcFilewalkerProgressDB).Get:47\n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePiecesToTrash:154\n\tstorj.io/storj/storagenode/pieces.(\*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(\*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(\*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78”} ERROR filewalker failed to reset progress in database {“error”: “gc\_filewalker\_progress\_db: context canceled”, “errorVerbose”: “gc\_filewalker\_progress\_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(\*gcFilewalkerProgressDB).Reset:58\n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePiecesToTrash.func1:171\n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePiecesToTrash:244\n\tstorj.io/storj/storagenode/pieces.(\*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(\*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(\*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78”} ERROR retain retain pieces failed {“cachePath”: “X:\!STORJ913\WORKDIR/retain”, “error”: “retain: filewalker: context canceled”, “errorVerbose”: “retain: filewalker: context canceled\n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePiecesToTrash:178\n\tstorj.io/storj/storagenode/pieces.(\*Store).WalkSatellitePiecesToTrash:585\n\tstorj.io/storj/storagenode/retain.(\*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(\*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78”} **The bloom filter file is no longer in the /retain folder** ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [May 7, 2024, 11:51am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/32 "2024-05-07T11:51:41Z")
I see now. I believe on Linux/Docker it is saved and resumes at restart. I didn’t test it, but I saw the process starting and finishing 2 times, and BF was in Retain folder, but smaller. I guess it restarted for an update or something and split the BF, taking out the part that was processed already. Maybe I’m wrong, but this is what I remember. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [May 11, 2024, 4:35am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/33 "2024-05-11T04:35:47Z")
> [@snorkel](#): > > I guess it restarted for an update or something and split the BF I do not think the split is implemented, more like you received a second one for the same satellite. The suffix is a Linux epoch timestamp, so it will be different. ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 14, 2024, 7:21pm UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/34 "2024-05-14T19:21:23Z")
Why is there absolutely no reaction or solution to this problem? The problem has not gone away. The bloomfilter file is still deleted automatically when the node is stopped/restarted on Windows 10/11. My 46 nodes can’t be a fluke. The error appears on each of them! I provided logs and a video of the error manifestation. What else can I do? Or are the developers okay with this bug? I have pointed out many times that the problem only appears on Windows nodes! But for some reason everyone checks the problem on Linux nodes, where there is NO problem! Is it really impossible for a developer to check the problem on a Windows node? It’s been a long time since I posted about the problem, but nothing happens. All nodes has 104.1 for now ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [May 15, 2024, 7:13am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/35 "2024-05-15T07:13:39Z")
> [@pdeline06](#): > > The problem only affects WINDOWS! No, something is not right on Linux too. I just got 3 bloomfilters deleted instead of at least 2 of them being processed: > [@GC successful, bloomfilter not deleted, new bloomfilter not processed](https://forum.storj.io/t/gc-successful-bloomfilter-not-deleted-new-bloomfilter-not-processed/26194/22): > > Now I see the following: The node had restarted. GC has appeard to resume, meaning that it created a new trash date folder with current name. But when I look now there is only subfolder in it and all bloomfilter from the retain folder have vanished. This means that the bloomfilter for the US-1 satellite has vanished which was under processing as well as the 2 bloomfilters from EU-1 satellite are now gone. ---
**Author:** ![jtolio](https://storj.bcdn.literatehosting.com/letter_avatar/jtolio/32/5_5575768a8748004e209b776fc1b2916d.png) [@jtolio](https://forum.storj.io/u/jtolio)\ **Post date:** [May 16, 2024, 1:06pm UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/36 "2024-05-16T13:06:42Z")
@pdeline06 - we care, we’re just confused and so far unable to reproduce. It’s not obvious what to fix because we don’t understand what is wrong. Thank you for all of the information you’ve provided so far. We’ll keep trying to reproduce, but again, all of the debug logging and information you can provide is helpful, as is everyone else in this thread trying to understand what circumstances trigger this. One thing that might be interesting - if you notice a bloom filter exists, can you mark it read only in your filesystem, and then see what process fails or what new logs show up when something tries to delete it? ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 16, 2024, 6:33pm UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/37 "2024-05-16T18:33:28Z")
Thanks for the answer. As soon as the nodes receive the bloom filter file - I will do this ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [May 17, 2024, 5:44am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/38 "2024-05-17T05:44:49Z")
> [@It seems that the new feature “save-state-resume GC filewalker” isn’t functioning as expected](https://forum.storj.io/t/it-seems-that-the-new-feature-save-state-resume-gc-filewalker-isn-t-functioning-as-expected/25874/39): > > Here is another one that appears that it has been interrupted during processing of the bloomfilter file. The trash date folder only has a single aa subfolder: ls /storage/trash/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/2024-05-16 aa This suggests that there was a bloom filter starting or resuming yesterday on the 16th. But it only processed the aa folder. No other subfolders are showing. So I assume the process got interrupted during processing and would expect the bloom filter fi… ---
**Author:** ![pdeline06](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pdeline06/32/1129_2.png) [@pdeline06](https://forum.storj.io/u/pdeline06)\ **Post date:** [May 17, 2024, 11:53am UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/39 "2024-05-17T11:53:44Z")
I found a previously saved bloom filter file on one node and tried to play with it. I copied it to the /retain folder Next, in file security, I changed the permissions: - DENIED - ALL - DELETE - ALLOW - ALL - READ To check, I tried to remove it myself - it didn’t work, it’s forbidden. Next, I start service storagenode. The node has started. A line appeared in the log: INFO retain Prepared to run a Retain request. {“cachePath”: “X:\!STORJ913\WORKDIR/retain”, “Created Before”: “2024-04-30T20:59:59+03:00”, “Filter Size”: 3300354, "Satellite ID ": “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”} The node began collecting garbage. After 5 minutes I stop the service storagenode. As soon as the service storagenode is stopped, the bloom filter file is deleted from the /retain folder. Despite all the previously installed security attributes of the file, it was deleted! I’m shocked! The following lines appeared in the log: INFO Stop/Shutdown request received. ERROR filewalker failed to reset progress in database {“error”: “gc\_filewalker\_progress\_db: context canceled”, “errorVerbose”: “gc\_filewalker\_progress\_db: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(\*gcFilewalkerProgressDB).Reset:58 \n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePiecesToTrash.func1:171\n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePiecesToTrash:244\n\tstorj.io /storj/storagenode/pieces.(\*Store).WalkSatellitePiecesToTrash:565\n\tstorj.io/storj/storagenode/retain.(\*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.( \*Service).Run.func2:258\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78”} ERROR retain retain pieces failed {“cachePath”: “X:\!STORJ913\WORKDIR/retain”, “error”: “retain: filewalker: context canceled”, “errorVerbose”: “retain: filewalker: context canceled\n \tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePieces:74\n\tstorj.io/storj/storagenode/pieces.(\*FileWalker).WalkSatellitePiecesToTrash:178\n\tstorj.io/storj/storagenode /pieces.(\*Store).WalkSatellitePiecesToTrash:565\n\tstorj.io/storj/storagenode/retain.(\*Service).retainPieces:369\n\tstorj.io/storj/storagenode/retain.(\*Service). Run.func2:258\n\tgolang.org/x/sync/errgroup.(\*Group).Go.func1:78”} I do not know what else to say. I don’t see any more entries in the log. What does a bloom filter file remove? Naturally service storagenode when stopped. Nobody cares about this file anymore. How was it deleted if the file’s security attributes prohibit it??? Perhaps the service storagenode changes attributes and deletes it? I don’t understand. There is no normal logical explanation! Please note the last error in the log: ERROR retain retain pieces failed {“cachePath”: “X:\!STORJ913\WORKDIR/retain”… Could the non-standard (default) path to the /retain folder be the reason? All my paths are not the default ones. But they are registered in Config.yml ERROR filewalker failed to reset progress in database {“error”: "gc\_filewalker\_progress\_db… What does filewalker have to do with it? My settings: storage2.piece-scan-on-startup: false pieces.enable-lazy-filewalker: false My head is exploding because I can’t understand cause and effect! Ready to provide remote access to the node. Maybe this will help. ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [May 17, 2024, 12:15pm UTC](https://forum.storj.io/t/bloom-filter-strange-behavior/26028/40 "2024-05-17T12:15:05Z")
Does your node delete any file from this retain folder? Have you tried to put like three random text files in there and start and stop and see what happens? [Previous page](https://forum.storj.io/t/bloom-filter-strange-behavior/26028.md?page=1) [Next page](https://forum.storj.io/t/bloom-filter-strange-behavior/26028.md?page=3)