just used it, only because it was easiest to use:
i typed it in PowerShell with admin rights and after some HDD work, it deletes the 2 folders in Storj/blobs
(Wonder if i need additionally edit in the config.yaml? itâs Windows 10, GUI storagenode)
(i got 1,5TB on some node for those 2, just decomistioned satelities)
But the command donât work for the ever first test satelite, an old stefan-benten leftovers folder abforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaa (118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW)
and i want to tottaly eradicate all 3 of those from the memory of storagenode, so it even wont try FileWalker on those please, when i turn it full ON at start by command: storage2.piece-scan-on-startup: true
Yes just like in the example. Folders inside âabforhuxbzyd35blusvrifvdwmfx4hmocsva4vmpp3rgqaaaaaaaâ are mostly empty, but still some 249MB there some single files left.
Other folders from 2 recent test satelites are completly gone.
Iâm rolls up this command and it not works somehow:
docker exec -it storagenode3 ./storagenode forget-satellite --force 12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo --config-dir E:\data\ --identity-dir E:\identity\storagenode3\
INFO Anonymized tracing enabled {"process": "storagenode"}
FATAL Failed to load identity. {"process": "storagenode", "error": "file or directory not found: open E:\\identity\\storagenode3\\/identity.cert: no such file or directory", "errorVerbose": "file or directory not found: open E:\\identity\\storagenode3\\/identity.cert: no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdForgetSatellite:83\n\tmain.newForgetSatelliteCmd.func1:71\n\tstorj.io/private/process.cleanup.func1.4:393\n\tstorj.io/private/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:112\n\tmain.main:30\n\truntime.main:267"}
You need to provide the path to the identity and data inside the container, not your external paths. So, you need to execute it as is, i.e. literally without any modifications (except a container name):
Must be something wrong with these âbloom filtersâ again. My nodes did not âmissed the partyâ - they was running and online (>=99% online score) during summer satellite shutdown.
and IâVE SEEN massive automatic file deletion at that time back in summer. But for some unknown reason (more bugs in bloom filters?) it deleted only about 85-90% of the total data stored for these satellites on my nodes. The rest lay untouched until now. Now I have found this topic and am doing manual cleaning according to the instructions given in the first post.
As a result, about 500 thousand files (~600 GB) and ~ 8 thousand garbage folders ((1024 in blobs + 1024 in trash) x2 satellites x2 nodes) still remaining from 2 decommissioned satellites have been deleted on 2 nodes.
On the 3rd node i just deleted blob folders about a month ago before found this instruction. It was a small node (slightly more than 1 TB of data), so there was less debris from these satellites left on it. âOnlyâ about 40k useless files and 4k folders.
My nodes didnât miss the party and removed almost all data (and not only mine), so I guess our setups are different.
However, running this command should help to remove all remained data anyway.
This is how the bloom filter is configured. Itâs exactly tuned to delete not more than 90% at once (this is by design, otherwise it could delete something excess). So itâs required to run several bloom filters, this is mean that your node should not miss any of these requests (mine have them two times in the week). Satellites were running almost two months after the data were deleted before shutdown.
So, I guess your setup suffer from a slow disk response for a long time already.
I hope that the new version 1.93.2 is more robust even if the disk is too slow, however, if your node restarts/stops because of other errors (FATAL in particular), the filewalker may not finish its work in time.
The 90% parameter is a bit misleading here in this corner case. Usually it means: on average 10% of data to be removed will be mistaken for data to be kept. But for this to happen, there has to be some data to be kept, otherwise thereâs nothing for which a false positive can be mistaken for.
A bloom filter representing an empty set is just zeros, and a single run should be able to remove everything. As such, there must have been other operational problems at the time deletions and GC filters were being sent.
My nodes also kept around enough files for these satellites to account for about 3% of total data stored, despite having a pretty good online score. Might have just been problems on my side though, canât debug it anymore.
This problem with leftovers from defunct sats should be resolved by the code in the future, without user intervention, because you made possible the implemantation of private satellites, and the auto-maintenance of the stored data should be spot on, in order to make that ecosystem work flowlessly. FW, GC, bloom filters and all the other Santaâs elfs should take care of the files and trash, not requesting users to input commands for each private sat that they join and leave. So, put a pin in it and revisit this problem with the garbage/trash that is discussed on so many threads.
I believe the satellites have a protection against this, so I think it was sent as a normal bloom filter, not zeroed. Because my nodes have had a few GBs of not removed data in total after the satellites shutdown. So I think your nodes were ok.
Perhaps you are right, but you know - priorities⌠And we (SNOs) have a command to override.
Especially if the satellite operator will shutdown their satellite without waiting while data would be removed from nodes (operational costs, you know).
But I think, first we need to fix the leftovers of filewalkers.
i have a pi4 node .I have a pi node with docker. How is the command for docker forget-satellite --all-untrusted. Sry i am new in linux. And will the data be deleted or do I have to do something else?
2024-01-04T11:17:24Z INFO Anonymized tracing enabled {âprocessâ: âstoragenodeâ}
2024-01-04T11:17:24Z FATAL Failed to load identity. {âprocessâ: âstoragenodeâ, âerrorâ: âfile or directory not found: open mnt/storagenode-1/identity/identity.cert: no such file or directoryâ, âerrorVerboseâ: âfile or directory not found: open mnt/storagenode-1/identity/identity.cert: no such file or directory\n\tstorj.io/common/identity.Config.Load:326\n\tmain.cmdForgetSatellite:83\n\tmain.newForgetSatelliteCmd.func1:71\n\tstorj.io/private/process.cleanup.func1.4:393\n\tstorj.io/private/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomOptions:112\n\tmain.main:30\n\truntime.main:267â}