Dashboard reporting disk is almost full when in reality its half full?

[Dave]

I think when the test data was deleted and moved to trash its gone a bit wrong?

[nerdatwork]

Do you have used-space filewalker disabled ?

[Dave]

is this what i need to insert into my config file “pieces.enable-lazy-filewalker: true” ?
Thanks for replying

[nerdatwork]

Its this

# if set to true, all pieces disk usage is recalculated on startup
storage2.piece-scan-on-startup: true

[naxbc]

I actually have this one on startup and lazy file walker enabled and it’s been ongoing for over a week and still hasn’t updated the correct used space

[Bivvo]

I seem to have a similar topic …

Is used-space-filewalker finished for all satellites?

Filewalker status

Logs should be at least info level Used space filewalker docker logs storagenode -f 2>&1 | grep "used-space-filewalker" | grep -E "started|finished" GC filewalker docker logs storagenode -f 2>&1 | grep "gc-filewalker" | grep -E "started|finished" Trash-cleanup filewalker docker logs storagenode -f 2>&1 | grep "trash-cleanup-filewalker" | grep -E "started|finished" retain docker logs storagenode -f 2>&1 | grep "retain" collector docker logs storagenode -f 2>&1 | grep "collector" trash docke…

[Dave]

thanks, I’ve updated my config file and restarted the node. nothing has changed yet. i guess time will tell!

[tfoutfou]

i have pretty much the same behaviour , windows reporting a lot of free space , when in fact it should be all Red and almost full like it used to be , (less than 100 GB free per disk)

i have around 1/3 free space on each disk now

All my 14 node dashboard report 5GB of empty space , and my ingress is basically under 1GB per day , wich seem logic since the node “think” it is full

been runing windows docker on hyperv for each node and working like a charm for more than a year , but now i have this behavior that seem to coincide with recent update

what should i read about new “lazy filewalker” and other important stuff that i may have miss to fix this ?

thanks

[Alexey]

Hello @tfoutfou,
Welcome back!

You may try to check status of all filewalkers, they should finish without an errors for each satellite:

Storagenode.exe stopping for too long, for 10h+

Windows/PowerShell Logs should be at least info level Used space filewalker Regulated by the parameter in the config.yaml file or as a command line argument (add -- before the parameter, remove :, place it after the image name in your docker run command): # if set to true, all pieces disk usage is recalculated on startup (default true) storage2.piece-scan-on-startup: true to watch for a progress: cat "C:\Program Files\Storj\Storage Node\*.log" -Wait | sls "used-space-filewalker" | sls "sta…

[tfoutfou]

i dont get what i should do exactly sorry

but the only reference to filewalker i see in my logs are related to

2024-06-09T02:31:19Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”, “dateBefore”: “2024-06-02T02:31:19Z”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”, “bytesDeleted”: 0, “numKeysDeleted”: 0}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO pieces:trash emptying trash finished {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “elapsed”: “85.5156ms”}

not a single line with “used-space-filewalker” or “gc-filewalker”

i would like to remind that i dont know anything about the relatively new lazy filewalker , and my “old” config files dont even mention it

[donald.m.motsinger]

@tfoutfou

not a single line with “used-space-filewalker”

Then you have disabled it. Check your config.yaml for

storage2.piece-scan-on-startup:

[tfoutfou]

of course this is set to false

storage2.piece-scan-on-startup: false

never had a problem with that before for more than a year , dashboard storage and windows reported free space always close match , this is recent behavior

i have 12 TB node , and hamering the disk like crazy during 2-3-4 days at every single windows update reboot or automatic node update or small change is not very acceptable in my opinion

how can i be sure if i enable it again that it will use the “lazy” filewalker and not the old one ?

i tried re enable it on a smaller 2TB node , restart node , 12h later still not see any line about space-filewalker

sorry for many question , but need help to understand how this work

[Alexey]

Please make sure that’s enabled and restart the node. The used-space-filewalker should update your databases.
Please, do not interrupt it (otherwise you will be forced to start over).

[tfoutfou]

will try on a smaller node first (all affected the same way anyway)

and what about this ?

i have 12 TB node , and hamering the disk like crazy during 2-3-4 days at every single windows update reboot or automatic node update or small change is not very acceptable in my opinion

is this bad habit and irrelevant with the new filewalker ?

sorry for not following the update release note in due time and learn about the new stuff , now i am lost

and maybe i have to add new parameter that must be added to my old config.yaml that are necessary by today standard ?

summary of my config.yaml

healthcheck.details: false
healthcheck.enabled: true
identity.cert-path: config/ident/identity.cert
identity.key-path: config/ident/identity.key
log.level: info
log.output: “/app/ssd/storagenode.log”
metrics.app-suffix: -alpha
metrics.interval: 30m0s
operator.email: xxxxxxxxxxxx@xxx.com
operator.wallet: 0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
operator.wallet-features: [“zksync”]
storage2.database-dir: “/app/ssd/database”
version.server-address: https://version.storj.io
storage2.piece-scan-on-startup: true
pieces.enable-lazy-filewalker: true

and the docker line that go with it

docker run -d --name node14 -p 20014:20014/tcp -p 20014:20014/udp -p 10014:10014/tcp --restart unless-stopped --stop-timeout 300 --mount type=bind,source=“M:\node14”,destination=/app/config --mount type=bind,source=“D:\nvmeSSD\node14”,destination=/app/ssd storjlabs/storagenode:latest --console.address=:10014 --contact.external-address=xxx.ddns.net:20014 --server.address=:20014 --server.private-address=:8014 --storage.allocated-disk-space=1.70TB

[Alexey]

@tfoutfou

is this bad habit and irrelevant with the new filewalker ?

no
Just make sure, that’s running. That’s all.

[tfoutfou]

obvisouly there is no other way right ?

i hate to ear my HDD screaming in pain , and be at 100% for days

why did it worked ok before , while now the reported size is wrong ?
is this all due to the most recent update from the last few week/month ?

i am very concerned about the filewalker runing for days at every windows reboot , or storagenode update , or any hickups , that’s why i disabled scan on startup a long time ago

after a succesful scan , can i disable it again for month ? or the subsequent scan will be really shorter maybe ?

[Alexey]

@tfoutfou

why did it worked ok before , while now the reported size is wrong ?

Because now it cannot keep up with the load, likely produced by recent tests:

Updates on Test Data

Hello Storj Community, We are excited to announce some changes that we have begun to implement regarding how we are using test data on the network! What is changing We are currently seeing exponential growth in our paid customer data and expect that trend to continue based on our sales pipeline. Because of this we want to ensure that we are sending timely demand signals to the node community so that the capacity and other resources we need for our growing customer base is available when they …

@tfoutfou

after a succesful scan , can i disable it again for month ?

likely yes, if you do not have errors related to databases in your logs and if that disk is not used for other stuff.
Each successful upload or deletion should update databases properly, so the usage is expected to be accurate. However, if you would notice that it’s differ - you would need to enable the scan again and restart, then disable it until the next usage issue.

[Bivvo]

@Alexey

However, if you would notice that it’s differ - you would need to enable the scan again and restart, then disable it until the next usage issue.

So it’s recommended to do it on startup only rather than lazy? (Because lazy is running permanently and causing RAM / CPU issues on small hardware setups)

[Alexey]

These are not mutual exclusive, the scan is needed to fix a discrepancy issues, the lazy mode should reduce I/O load during scans (any filewalker, perhaps except expired pieces collector - it doesn’t support a lazy mode yet).

[Dave]

It’s finally sorted itself out! Thanks for everyone’s input. Do I need to remove the lazy file walker line from the config file now?