So my free space went from 40GB to 1.97TB and the trash went from 550GB to 910GB. All I did was add an additional port to the router (for a second node) for the pc the node is running on (Linux). Note that this has also happened before for the exact same values it reset to.
Why did this happen? Twice!
Perhaps the trash filewalker removed the expired data from the trash and the used-space-filewalker finally updated databases to reflect the actual used space, and the TTL based data has been removed (it is deleted by nodes without using GC or the trash), also the garbage collector finally moved detected garbage to the trash.
You may check all of that:
or for Windows:
Your change in the router is not related to this, unless you also unblocked connections from the satellites and customers to your node.
If you have some “smart” protection feature or “DDoS protection” on your router - I would recommend to disable these features.
The only coincidence here is a timing in my opinion, because almost all filewalkers are time based.
So for the “smart” protection, I did have that turned on since I got a new ISP and it was on by default but I just turned it off, called (advanced security).
as for the docker logs command, I get the following:
2024-05-05T19:08:14Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T19:08:14Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T19:09:58Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-05T19:09:58Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T19:09:58Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-05T23:56:29Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-05T23:56:29Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-05T23:56:29Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
Edit: I’d like to also add that this isn’t new trash. The same exact thing happened about a week ago, reverting back to the same exact free and trash values.
If do not see all “finished” yet.
this is usually mean that you have problems with a database (database is locked or malformed or any other error, related to a database in your logs).
I would also recommend to check databases:
There’s “finished successfully” in there
and
but not other two, please wait until they are finished too.
Right, I just stopped watching so I stopped monitoring it after I sent that reply and went to bed so I don’t know if they got completed.
It still doesn’t explain why I lost nearly 2TB of data (that didn’t go to the trash) out of nowhere twice in a month with the exact same values as last time. That’s super frustrating
This is usually mean that your databases doesn’t have the current collected usage.
Please search for “database is locked” errors, also any other errors related to “database” in your logs.
Please also check the databases integrity and fix them if needed:
Could have the wrong command format but using sudo docker logs storagenode 2>&1 | grep "database is locked" doesn’t return anything, doesn’t even finish the command execution. I also tried using the -f flag.
For the integrity check for each DB, the following did not return ok, did not finish running either so no error was returned:
piece_expiration.db
piece_expiration.db-shm - returned: Error: in prepare, file is not a database (26)
Everything else returned ok
Edit: Also, values were reset again after I started the node back up. I’m back to 1.97TB free and 910GB trash (was at 1.65TB free and 550GB trash before doing all this).
This is mean that it’s doing something. It should either return an output or finish the command.
You should run the test only when the node is stopped. This file is exist if the node is not stopped or if it was stopped unclean.
Of course. Now you need to check for database is locked errors in your logs.
Note: Sorry, I don’t know how to directly reply to certain parts of a reply like you did.
So checking the integrity of that DB file took a little over an hour but it returned ok so everything is good there.
I stopped the node properly with the following so I believe I have stopped it cleanly:
sudo docker stop -t 300 storagenode
What would be the proper command for searching for database is locked?
Select the line, there would be a menu, where you can click Quote:
docker logs storagenode 2>&1 | grep -i error | grep database | tail
This command returned nothing after a while. Node was running when I ran it.
This is mean that the node do not have errors related to databases since the last creation of the container.
Because when you remove the container, all logs will be removed with it.
Now you need to wait until the used-space-filewalker would be finished successfully for all trusted satellites. You may search for used-space-filewalker and finished.
See also for a more detailed progress: Filewalker status - #5 by Alexey
Please also make sure that there would not be errors related to the databases when the used-space-filewalker is finished.
I ran the following command again as you advised to wait and all 4 satellites finished successfully (output below command):
docker logs storagenode -f 2>&1 | grep "used-space-filewalker" | grep -E "started|finished"
2024-05-05T19:08:14Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-05T19:08:14Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T19:08:14Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T19:09:58Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-05T19:09:58Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T19:09:58Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-05T23:16:58Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-05T23:56:29Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-05T23:56:29Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-05T23:56:29Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-05T23:56:29Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-07T06:08:36Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-07T06:08:36Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-07T06:08:36Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-07T09:31:50Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-07T09:31:50Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-07T09:31:50Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-07T09:31:50Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-07T09:50:07Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-07T09:50:07Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-07T09:50:07Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-07T09:50:07Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-09T04:39:36Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-09T04:39:36Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-09T04:39:36Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-09T04:39:36Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-09T05:28:44Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-19T14:38:44Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-19T14:38:44Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-19T14:38:44Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-19T14:38:58Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-19T14:38:58Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-19T14:38:58Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Process": "storagenode"}
2024-05-19T16:47:51Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-19T16:47:51Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-19T16:47:51Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-19T16:47:51Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Process": "storagenode"}
2024-05-19T16:51:44Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-19T16:51:44Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-19T16:51:44Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-19T16:51:44Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Process": "storagenode"}
2024-05-21T11:22:15Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-21T11:22:15Z INFO lazyfilewalker.used-space-filewalker subprocess started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-21T11:22:15Z INFO lazyfilewalker.used-space-filewalker.subprocess Database started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-21T11:22:15Z INFO lazyfilewalker.used-space-filewalker.subprocess used-space-filewalker started {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Process": "storagenode"}
2024-05-21T11:59:46Z INFO lazyfilewalker.used-space-filewalker subprocess finished successfully {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
As for checking for errors regarding the DB after used-space-filewalker:
sudo docker logs storagenode 2>&1 | grep -i error | grep database | tail
Still no output so all still seems good there.
yes, the used-space-filewalker is finished successfully in May.
But not in June.
Do the usage on the dashboard matches what’s OS reported?
du --si -d 1 /mnt/storj/storagenode/storage/
my node is defined in the run command as such:
--mount type=bind,source="/home/pi/identity/drive1/",destination=/app/identity \
--mount type=bind,source="/mnt/drive1/data",destination=/app/config \
--mount type=bind,source="/home/pi/database/drive1/",destination=/app/dbs \
Would the command for me be?
du --si -d 1 /mnt/drive1/data/storage/
Just want to make sure I understand that I am running it correctly before doing so.
yes. It should calculate the size for each subfolder in SI units, like the dashboard do.
Alright, I have it running now and it’s showing the garbage dir so far. The node is still running so is that ok and proper? Also, is this a long running command because it has to calculate all the used space in each category?
It’s ok
Yes. It’s exactly what filewalkers usually doing. But since you run it with a normal priority it shouldn’t take days to complete. I hope.