I am seeing the same thing on my node, its about 40TB and running v1.105.4. I see " ERROR lazyfilewalker.used-space-filewalker failed to start subprocess {“Process”: “storagenode”, “satelliteID”: “***”, “error”: “context canceled”}", only one of the satellite has finished. When i run the info command its says it failed to load the identity, i have attached a screenshot. I have confirmed that the container is able to access all of the identity files.
EDIT: My drives are part of a mdadm array, they have been running 24/7 for months, no reported disk errors.
Interesting that after restart i got on this node some free space but not from trash amount, looks like TTL data show free up on node only after restart(it is a theory yet)
I started my node with both:
storage2.piece-scan-on-startup=true
pieces.enable-lazy-filewalker=true
Using htop, I see it is finished scanning the blobs folders. However the usage according to the graph and df -h remain the same. using --si puts me at 20TB used.
I do get these kind of warnings, but I don’t think they’re related;
WARN console:service unable to get Satellite URL {“Process”: “storagenode”, “Satellite ID”: “12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo”, “error”: “console: trust: satellite is untrusted”, “errorVerbose”: “console: trust: satellite is untrusted\n\tstorj.io/storj/storagenode/trust.init:29\n\truntime.doInit1:6740\n\truntime.doInit:6707\n\truntime.main:249”
I think Im patient, but that the SNOs has to handle the deletion of decomissioned Satellites manually is a Joke … why the hell the SNO-Software couldnt do this?
Same Joke with the undeleted data which seems to be there forever even if TTL should be deleted. Same Joke with Trash and Filewalkers …
One bug and it might wipe the wrong satellite. That was the reason we didn’t implement something that can potentially destroy the entire network with just one update.
Your not seeing things, of course you’re spot on with your understanding of your NTFS volume. I’m guessing 64k clusters?
Quick explanation of your node history vs. now
By my recollection my average file size has been falling since my noted peak in 2023 of about 700k, down to about(~) 500k (death of 125 gb free) late 2023, and after recently killing off free tier data, all the way down to about 250k early 2024. This test data is perhaps 125k-135k distribution. You’ve got at least a 32TB disk there, ergo 16k or greater cluster size… prolly 64k. Your inefficiency now is over 16%… 37/32… Likely 1/4 of how efficient it used to be. Going forward, with this new workload - maybe go back to 16 or 32kb.
There are no bloom filters for your size, so I’d also suggest - especially if it’s true there’s a bug in deleting TTL, you go back to 16TB nodes, maybe even 4k clusters - maybe even with compression on, your only option to try that. Personally I use 32k, it’s a decent match. If you examine recent ‘test’ data, it’s 3k + 2meg polarized, imagine there’s an abundance of only 3k/64k efficiency in this new reality. Multiple nodes would sequence data better… inflow/outflow wize, without buggy prone to re-starts on unhealthy run-length file walker, etc. times.
Make sure all the filewalkers finish succesfuly, and restart node (latter part was necessary for me but may not be for you). No other solution is known