I have been reading this post, also about garbage collection problems.
This node shows different capacities, on the right one capacity appears, on the left another capacity appears.
Why doesn’t the trash appear in the trash can? There is garbage?
The hard drive is full with the capacity on the right side.
Why does less capacity appear on the left side?
Is the capacity measurement on the left side false?
What could be the problem?
Last month this node was full.
Other nodes are the same, I have other nodes that have less capacity on the left side.
I am running Storj on a Windows Server 2022 vm. I have my pool connected to the vm over iSCSi hosted by a zfs pool in truenas. Up until late last year, my Avg Disk Usage each month stayed fairly consistent with my Disk Space used metricover my first 1.5 years. These also lined up with what Windows reported the drive was using. Over the last 4 months or so, things have grown apart to the tune of avg space used being 9.9 TB currently, while diskspace used and windows come in around 15.5 TB used.
Assuming the following 2 commands are correct, i get no results for either. “sls “walk” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls “error|failed” | select -Last 10
sls “retain” “C:\Program Files\Storj\Storage Node\storagenode.log” | sls “error|failed” | select -Last 10”. Any ideas where to start looking for the cause of this large difference? Should i wait for the latest version of the node software to be released and see if that gets things back in check?
Then they never start. Any ideas on troubleshooting if they never start?
If the used-space-filewalker is enabled (it’s enabled by default), it should update databases with the actual usage.
You may check the progress:
All filewalkers should be finished without errors for each satellite, you also should not have errors related to databases (“database is locked”, “database is malformed”, “not a database”). If all conditions are met - the node should show the usage very close to the actual usage on the disk and to the usage reported by your node to the satellites.
ok, so my logging was set to fatal. i have moved it to info. i do not see any entries in the config.yaml file talking about filewalker. is it ok to add the “# if set to true, all pieces disk usage is recalculated on startup (default true)
storage2.piece-scan-on-startup: true” anywhere in the config yaml file, or does it need to go somewhere specific.
Dosen’t matter the location. Just check it to not have it in 2 locations, uncommented.
Thanks. i double checked the config file and found no entry for the scan on start up. i have added it, made sure it was not commented out, and set to true. i rebooted the server and now i guess its just a waiting game? Is there any way to check progress, or do i just wait a few days and reboot and see if things are getting better?
If it finish, You will get a info in the log file.
Also You can tell by looking at HDD working time (like windows manager has),
if its stable 100% use time, it means its filewalking.
its DEFINNITELY NOT filewalking if its fluctuates 0-50% even up to 80% all the time
can confirm, because of the low amount of difference then i did not suspect anything, now things go way out of sync.
I reinstalled and filewalker started working again. Kept getting install version errors when I was trying to use my previous DBs so I just let it build new ones and everything seems to work for now. Checked old DBs for corruption and everything was good too.
You usually do not need to rebuild databases unless they are corrupted. The re-installation is also not needed. But perhaps you installed a newer version, which has improvements related to filewalkers:
Hi @Alexey, this discussion is too long, it make it hard to follow some good advise in the middle, those should advise be extract to pin somewhere for new folk follow this topic?
Click the ellipsis (…) and it will expand to show a bookmark tool as shown below.
Click the bookmark button for any post that you seem fit to save as a bookmark.
Click your profile to show list of bookmarks you have saved by clicking the same bookmark button.
Hi @Alexey.
The 2TB Data discrepancy in my node is still there.
Databases check all came out ok, and used space filewalker is successfull.
What shoud I do?
Hello,
I am seeking advice regarding the big difference in real disk space used by node and the average disk space used for payouts. I understand there have been several threads regarding this issue but I am out of ideas. I moved the dbs to SSD and ran the forget untrusted satelites commands. These steps have not made any significant difference. I am running my node in a docker container, the node runs on a 10 TB internal HDD, ext4 FS. Is there anything I could try? Or this is simply it and this is how it works with larger nodes? I was planning to move the node to 20TB drive (or start a new node after 10TB fill up), but not sure how much of a difference there would be with a 20TB node.
Thanks for any advice.
1.99.3 node here:
many others have this problem, missing serveral TB on full nodes.
No fix afaik. Just keep going.
See “Disk usage discreppancie” and “Debugging disk usage discreppancies” threads… or something like that.
Thank you for the replies. I have looked at different threads, but there is very little conclusive info on the matter. If it is a software problem on the side of Storj, an announcement in easy to understand language would be appreciated. There is a thread on a problem with garbage collection and I assume it is related, but for someone not familiar with all the technical matters, it is really hard to understand what is going on. If this is a problem that affects many SNOs, it would be nice to have information from Storj team acknowledging the issue and with a road map for solving it. Otherwise we spend a lot of time reading through forum posts (and asking questions) without finding any solution.
Recently some larger nodes have not received bloom filters or garbage collection on US1 since Mar 1, and judging from your 150GB trash it has not made its way to your node yet. I would search your log files for the keyword gc-filewalker and report here all lines.