Disk usage discrepancy?

Node Operators troubleshooting
, , , ,
197

You are correct - all filewalkers should finish their job, only then the info will be updated in the local databases. The dashboard took this information from the local databases.
If the node is restarted before these filewalkers are finished, it will start over from the beginning. Databases are updated only after the successfully finished scan.
There are at least three filewalkers for each satellite:

  • used space filewalker (used-space-filewalker) - updates used space in the databases
  • garbage collector filewalker (gc-filewalker) - collect the garbage (deleted pieces)
  • retain filewalker (retain) - moves the garbage to the trash
  • collector - removes expired pieces (if the customer specified the expiration date for the piece)
  • piece:trash - cleans the trash

If your disk subsystem is slow (virtualized, USB, network filesystem, etc.), these filewalkers may works days. If the disk is slow to respond, the node may stop itself, if the write or read check would fail because of timeout.

1 Like
198

Hello - i have a HDD where the node dashboard claims 5.3TB used - but windows claims all 9TB is full. Maybe something with GC? can i do anything? there is not else than storj on the HDD

199

And no its not because GC has not run yet - there has been this difference for weeks. also its a problem because the disk is getting very full.

200

image

And dashboard:

image
image724×766 40.4 KB

201

what filesystem and what clustersize? disk type and manufacturer?

202

make new thread? storj version ? just confused this two together.

203

File system is nfts on a windows machine. i dont know what cluster size.

disk is a 3.5HDD from Seagate - its 10 TB drive.

204

Sorry about that - that is not a problem but user error - lets keep it to the original question.

205

Why did you delete the log ?

2023-08-11T23:23:15+02:00 ERROR pieces failed to lazywalk space used by satellite {“error”: “lazyfilewalker: context canceled”, “errorVerbose”: “lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:71\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:105\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:707\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:57\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:44\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”}

As per the log your used-space-filewalker didnt finish which is why there is discrepancy in space calculation.

Also have a look at this thread

1 Like
206

Let me upload the correct log - the one i posted was not the right one.

log:

This might be old log info… because server can ping satilites fine - it had issues earlier but i made an ip mistake. so im not sure what is happening with the GC

207

As per above log, your trash has an issue. Can you check the log for an error with gc-filewalker ?

208

I also see

"system clock is out of sync: system clock is out of sync with all trusted

May he check the system time and date?

209

Unfortunaly that is the complete log… there is no more in there.

210

System clock error is fixed. thats not an issue no more.

211

If your node is up then you can check your current log. Also to make space you can stop the node and delete *.partial files that are older than 2 days from temp folder.

212

The node is up. i will post logs again but i dont think its reporting anything (log level is also set to error - i hope thats fine.)

logs:

213

I might shut off the node as there is currently only 700mb left. or adjust the size of the node as a tmp solution.

214

New logs still dont report anything about QC.
https://pastebin.com/ECGR9Nji

Its annyoing because i now none of the files are there for any other reason but storj. so im getting “paid” for 4,5TB but using 9 :frowning:

215

You paid for what’s confirmed by your node and customers - it’s an average disk usage graph on the left. The right part is shown from the databases too, but databases are updated only when the filewalker is completed its scan for each satellite.
I also hope that you removed decommissioned satellites too (How To Forget Untrusted Satellites).
The best way can be to reduce the allocation and disable a lazy filewalker, to allow it to finish more quickly. However to completely clean your node from deleted data it should work like this several weeks.
See

https://forum.storj.io/t/disk-usage-discrepancy/24715/27?u=alexey

216

So i have tried to use the forget all untrusted within windows- but it does not see the command as valid:

storagenode forget-satellite --all-untrusted --config-dir C:\Program Files\Storj5\Storage Node --identity-dir E:\Identity\storagenode
Get-StorageNode : A positional parameter cannot be found that accepts argument ‘–all-untrusted’.
At line:1 char:1

  • storagenode forget-satellite --all-untrusted --config-dir C:\Program …
  •   + CategoryInfo          : InvalidArgument: (:) [Get-StorageNode], ParameterBindingException
  + FullyQualifiedErrorId : PositionalParameterNotFound,Get-StorageNode

I have dissabled filewalker on this sepecific node…