Dashboard reporting disk is almost full when in reality its half full?

I think when the test data was deleted and moved to trash its gone a bit wrong?
image

Do you have used-space filewalker disabled ?

is this what i need to insert into my config file “pieces.enable-lazy-filewalker: true” ?
Thanks for replying

Its this :point_down:

# if set to true, all pieces disk usage is recalculated on startup
storage2.piece-scan-on-startup: true

I actually have this one on startup and lazy file walker enabled and it’s been ongoing for over a week and still hasn’t updated the correct used space :frowning:

1 Like

I seem to have a similar topic …

Is used-space-filewalker finished for all satellites?

1 Like

thanks, I’ve updated my config file and restarted the node. nothing has changed yet. i guess time will tell!

3 Likes

i have pretty much the same behaviour , windows reporting a lot of free space , when in fact it should be all Red and almost full like it used to be , (less than 100 GB free per disk)

i have around 1/3 free space on each disk now

All my 14 node dashboard report 5GB of empty space , and my ingress is basically under 1GB per day , wich seem logic since the node “think” it is full

been runing windows docker on hyperv for each node and working like a charm for more than a year , but now i have this behavior that seem to coincide with recent update

what should i read about new “lazy filewalker” and other important stuff that i may have miss to fix this ?

thanks

Hello @tfoutfou,
Welcome back!

You may try to check status of all filewalkers, they should finish without an errors for each satellite:

i dont get what i should do exactly sorry

but the only reference to filewalker i see in my logs are related to

2024-06-09T02:31:19Z INFO pieces:trash emptying trash started {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker starting subprocess {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”, “dateBefore”: “2024-06-02T02:31:19Z”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess Database started {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker.subprocess trash-filewalker completed {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Process”: “storagenode”, “bytesDeleted”: 0, “numKeysDeleted”: 0}

2024-06-09T02:31:19Z INFO lazyfilewalker.trash-cleanup-filewalker subprocess finished successfully {“Process”: “storagenode”, “satelliteID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”}

2024-06-09T02:31:19Z INFO pieces:trash emptying trash finished {“Process”: “storagenode”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “elapsed”: “85.5156ms”}

not a single line with “used-space-filewalker” or “gc-filewalker”

i would like to remind that i dont know anything about the relatively new lazy filewalker , and my “old” config files dont even mention it

Then you have disabled it. Check your config.yaml for

storage2.piece-scan-on-startup:

3 Likes

of course this is set to false

storage2.piece-scan-on-startup: false

never had a problem with that before for more than a year , dashboard storage and windows reported free space always close match , this is recent behavior

i have 12 TB node , and hamering the disk like crazy during 2-3-4 days at every single windows update reboot or automatic node update or small change is not very acceptable in my opinion

how can i be sure if i enable it again that it will use the “lazy” filewalker and not the old one ?

i tried re enable it on a smaller 2TB node , restart node , 12h later still not see any line about space-filewalker

sorry for many question , but need help to understand how this work

1 Like

Please make sure that’s enabled and restart the node. The used-space-filewalker should update your databases.
Please, do not interrupt it (otherwise you will be forced to start over).

1 Like

will try on a smaller node first (all affected the same way anyway)

and what about this ?

i have 12 TB node , and hamering the disk like crazy during 2-3-4 days at every single windows update reboot or automatic node update or small change is not very acceptable in my opinion

is this bad habit and irrelevant with the new filewalker ?

sorry for not following the update release note in due time and learn about the new stuff , now i am lost

and maybe i have to add new parameter that must be added to my old config.yaml that are necessary by today standard ?

summary of my config.yaml

healthcheck.details: false
healthcheck.enabled: true
identity.cert-path: config/ident/identity.cert
identity.key-path: config/ident/identity.key
log.level: info
log.output: “/app/ssd/storagenode.log”
metrics.app-suffix: -alpha
metrics.interval: 30m0s
operator.email: xxxxxxxxxxxx@xxx.com
operator.wallet: 0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
operator.wallet-features: [“zksync”]
storage2.database-dir: “/app/ssd/database”
version.server-address: https://version.storj.io
storage2.piece-scan-on-startup: true
pieces.enable-lazy-filewalker: true

and the docker line that go with it

docker run -d --name node14 -p 20014:20014/tcp -p 20014:20014/udp -p 10014:10014/tcp --restart unless-stopped --stop-timeout 300 --mount type=bind,source=“M:\node14”,destination=/app/config --mount type=bind,source=“D:\nvmeSSD\node14”,destination=/app/ssd storjlabs/storagenode:latest --console.address=:10014 --contact.external-address=xxx.ddns.net:20014 --server.address=:20014 --server.private-address=:8014 --storage.allocated-disk-space=1.70TB

no
Just make sure, that’s running. That’s all.

obvisouly there is no other way right ? :sleepy: :cry:

i hate to ear my HDD screaming in pain , and be at 100% for days

why did it worked ok before , while now the reported size is wrong ?
is this all due to the most recent update from the last few week/month ?

i am very concerned about the filewalker runing for days at every windows reboot , or storagenode update , or any hickups , that’s why i disabled scan on startup a long time ago

after a succesful scan , can i disable it again for month ? or the subsequent scan will be really shorter maybe ?

Because now it cannot keep up with the load, likely produced by recent tests:

likely yes, if you do not have errors related to databases in your logs and if that disk is not used for other stuff.
Each successful upload or deletion should update databases properly, so the usage is expected to be accurate. However, if you would notice that it’s differ - you would need to enable the scan again and restart, then disable it until the next usage issue.

So it’s recommended to do it on startup only rather than lazy? (Because lazy is running permanently and causing RAM / CPU issues on small hardware setups)

These are not mutual exclusive, the scan is needed to fix a discrepancy issues, the lazy mode should reduce I/O load during scans (any filewalker, perhaps except expired pieces collector - it doesn’t support a lazy mode yet).

It’s finally sorted itself out! Thanks for everyone’s input. Do I need to remove the lazy file walker line from the config file now?

1 Like