Ok. I was able to reproduce in storj-up.
Just need to keep it running for more than 24h.
4 Likes
@mahir Do you use a dedicated disk feature?
I observed this behavior, with file walker finished in ms on one of my nodes too, but I didn’t pay to much importance to it.
Ubuntu Server, 32GB RAM, no dedicated disk, no lazzy, no badger, 2 nodes on 2 drives, Docker, both with 3TB+ data.
I started the scan, and short after it finished, an update restarted the node and the scan started again, finishing very quickly. At that time I thought it was normal because the metadata was already cached in RAM. Now it seems there is a bug. I can’t remember if this happened before moving dbes to storage drive, or after. They were on OS m2 SSD. Normaly, the scan takes like 7+ hours.
The quick scan is not a bug, it’s a feature to speedup the used-space-filewalker even more.
Do you have a discrepancy after it’s finished?
Didn’t check at that moment. It seemed right. I believe it looses a few GB, and they are hard to spot on a 3TB node. They are visible on new small nodes.
Yes, this is only confirms what I have tested.
The quick used-space-filewalker is not a bug.
However, there are some circumstances, where the difference may occur.
what do you mean with dedicated disk feature?
its running inside a VM with no extra disk mounted.
tried workaround on my node with ~410gb of data, got 357 in dashboard (didn’t help at all)
seems like it’s showing usage after first restart
is what I’m seeing also the same problem?
- satellites report .99TB used
- used space on dashboard reports 1.4TB used.
- df --si reports 1.8TB used
- with a restart the used space filewalker finishes in 1 second per node.
- delete these used_space_per_prefix.db and restart, and filewalker takes normal times (a few hours)
- dashboard still showing same numbers 0.99TB vs 1.4TB
and i’m using the badger cache on this node, btw.
“seems like it’s showing usage after first restart” my nodes have same behaviour.
if I check database corruption it say OK but something happend because it fix the issue.
If you deleted a database and restarted the node, then you need to wait until the used-space-filewalker would finish the scan for all trusted satellites (search for used-space and completed).
No, there is discussed a difference between the usage on the piechart and the usage reported by OS. In your case 1.8TB vs 1.4TB.
The difference between the satellites reports and the actual usage is likely related either to a missing reports from the satellites or not collected garbage (for example, your node didn’t receive or yet processed the Bloom Filters from the satellites).
The quick used-space-filewalker is a feature, not a bug. But it may have a bug in the processing.
Does deleting the database eliminate the difference between 1.8TB (OS) and 1.4TB (dashboard)?
This is a new feature which allows to do not calculate the used space but use the usage reported by OS directly. However, this is an experimental feature and the usage on the dashboard will be wrong.
upd: all settled down after i opened dashboard after several hours
now it shows right disk usage
1 Like
No it’s been more than a day since restart, filewalkers are done, and it’s still showing the 1.8 / 1.4 / 0.9 discrepancies. I’m using badger cache which i didn’t touch, btw.
This case I cannot reproduce yet.
Could you please try to recreate a piece_spaced_used.db too?
Seems same for me as well, every time node is restarted - used disk value goes to value, that was after 1st restart. I’m using TrueNas Scale.
to clarify, you want me to delete piece_spaced_used.db and restart and see what happens?
Using this procedure:
because it wouldn’t be able to recreate only one missing database (because there are migrations).
Okay I did the sqlite3 database check (Everyone was ok), and deleted BOTH the pieces piece_spaced_used.db and used_space_per_prefix.db, and the dashboard is now showing nearly zero disk space usage but going up as the filewalker is running.
So… we’ll so how that goes!