Hello, I have a node that I once assigned around 10.7TB of storage. The cli dashboard shows that 10.59TB is in use and the node has stopped accepting uploads, while df -h shows that only 6TB is in use. What could be wrong?
Update: The used-space filewalker finished successfully last week. But the problem persists.
I’ve read that post. But what happens to my node is different. Because on my node the actual disk usage is much smaller than what the dashboard reports.
This is exactly the same issue. Your databases are not updated with correct values.
Please search for errors related to the databases in your logs, I would also recommend to check them:
Or do you mean that the CLI dashboard shows different values from the web dashboard?
Thank you. I will check it out.
I’ve checked and integrity of all databases are ok. What else can I try?
The next step is to search for any errors related to databases in your logs.
And by the way, it would be good if you also check that used-space-filewalkers are finished too for each trusted satellite.
You need also to remove the data from the untrusted ones:
I’ve searched the logs for [Dd]atabase and db[.] but didn’t find any error message. Also, I can see only 4 dirs under storage/blobs dir. So data for the 2 decommissioned satellites has been deleted.
Also, last week, I saw those used-space-filewalkers finished messages for each satellite in the log.
Then your used space on the dashboard should pretty match the numbers from the OS.
However, you need to request the usage with a --si option, not -h to have the same measure units as a dashboard.
I.e.
df --si -T
and/or
du --si -d 1 /mnt/storj/storagenode/storage
But df --si shows 7.0TB, which is still far from 10.59TB on cli dashboard.
Could you please show the result of the
df --si -T
and the result of the
docker exec -it storagenode ./dashboard.sh
And how much do you have as an allocation?
For now it clearly look like a problem either with a used-space-filewalker failed during the way or problems with databases.
I would recommend to check your logs for errors related to a filewalker and databases.
If you disabled a scan on startup (it’s enabled by default), please enable it back and restart the node. If you have errors with databases (e.g. “database is locked” or “malformed”), please consider to fix them if they are corrupted and/or move them to another disk/SSD if they are locked.
It’s used to be set to 10.7TB. But now I’m migrating the node to another drive. So I lowered it to 7TB.
used-space-filewalker finished successfully 1-2 weeks ago. But I will re-run it after the migration finishes.
Is there any tool to rebuild the database? For example, from the blobs/ or pull the data from satellites.
Depends on the situation. There is a method to fix corrupted DB and another method to recreate them fresh. Did you find your databse to be corrupt ?
I followed this guide but found no corrupted database.
I found a relevant post here: Disk usage discrepancy? - #787 by Vadim
Then you do not need to re-create these databases, you may just allow the filewalker to fix the discrepancy.
If you wouldn’t have any errors related to a filewalker and/or the database - it should update the usage. Please note, you need to have all filewalkers successfully finished for all trusted satellites.
Since you did change the allocation to 7TB, it’s expected to see a 10.44TB used and -3.44TB Available (i.e. it’s an overusage now).
However the difference between usage on the disk and the usage on the dashboard perhaps related to the wrongly updated usage for the trash (the 1.104.5 version has this bug), but it can be fixed by a filewalker as well.
