Maybe run used-space-filewalker 
?
3 Likes
maybe what happened was a used space filewalker actually finished somewhere and updated space that was erroneously thought to be unused and is actually used.
1 Like
Just a fun data point:
On one of my slower nodes with a lot of trash, just the “emptying trash” job (aka deleting files) for just the SLC node took 22 hours to run, and deleted around 300GB of files. (I don’t know how many files it was, probably approximately a bajillion).
You only had 300GB trash? I have 10TB to be deleted. It takes about 20 minutes for each xx folder with ~10GB in it and is at folder d3 right now.
3 Likes
No I had about 3.3TB of trash before. The one pass of emptying trash for SLC reduced it by around .3TB and took 22 hours.
This may happen, if nodes have had a disk usage discrepancy, and when the used-space-filewalker finished the scan, it updated the databases and some nodes are now aware that they used more, than was allocated.
Unfortunately, I do not know, how much data was deleted there, since the lazy mode is off (the lazy trash-filewalker has had a messages about amount of pieces at least):
2024-09-03T10:35:14Z INFO pieces:trash emptying trash finished {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "elapsed": "16h19m58.7272235s"}
The emptying for SLC is still in progress:
2024-09-03T18:15:15Z INFO pieces:trash emptying trash started {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
deleting is not random? how can I check the progess? no lazy walker
This is a good question, accordingly @jammerdan, the folder selection which would start to be deleted is random or, more like, the latest one, but the folders inside deleted in an ordered way, see
This is true for Linux and ext4. Windows behave differently, their system calls sorting the output by default. So - from oldest to newest, but I can confirm (or at least I noted that), that the folders inside the folders with the date names are deleted in the alphabetical order, too.
I found an easy way:
$ sudo ls -1Rltu /mnt/x/storagenode2/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/ | head
/mnt/x/storagenode2/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/:
total 0
drwx------ 1 root root 4096 Sep 5 10:09 2024-08-29
drwx------ 1 root root 4096 Sep 5 09:52 2024-08-26
drwx------ 1 root root 4096 Sep 5 09:50 2024-09-02
/mnt/x/storagenode2/storage/trash/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2024-08-29:
total 0
drwx------ 1 root root 4096 Sep 3 15:27 v3
drwx------ 1 root root 4096 Sep 3 15:18 v2
If you would add a date folder to the satellite’s subfolder path, you would see a progress If you have atime enabled, otherwise, the u option can be removed.
And to do not extend this with a discussing the sorting, you may refer the linked thread, thanks.
Inside the date folder it’s not random and rather a-z and then 2-7. You can check with lsof or the modification time with ls.
However, the sort order doesn’t affect the amount of the deleted data as far as we know for any system. So, seems it is irrelevant.
See also
To put some numbers into perspective, I’m not sure exactly how much trash was accumulated, but it was significantly less than 5.6 TB. This is on a full node, which means it’s unable to accept new ingress due to the full trash:
Upon reviewing the date folders, I noticed that the trash garbage collection started on August 26th. The trash removal process began after a week on September 2nd and is still ongoing. Fortunately, it appears to be nearing completion, running on the 6* folders right now. Given the progress, I’m hopeful that it will be finished today, September 9th.
The node has been full and unpaid for that space for at least since then, unable to receive new uploads for at least two weeks. This is significantly longer than the typical 7-day trash removal period that’s often cited. Because it not just takes the 7-days unpaid grace period until removal starts but also the time to finally physically remove the data.
So with 5.6Tb the duration could be even worse, of course depending on the SNOs situation.
1 Like
I seem to have similar issue. I’m running a small node of only 3TB, and unfortunately most of that was filled up saltlake. About two months ago, even though my Used space still says 3.0TB, my Average Disk Space Used This Month dropped all the way down to 111.37, with saltlake showing 0 storage and activity, while that 111.37 is split among ap1, us1, and eu1.
Based on what Alexey said in previous threads, I believed that I would still get a full payout for those 3TB on saltlake, but payment just came in and it was only $0.06, with saltlake paying nothing. What really sucks is that I have almost 3TB of saltlate data that isn’t paying anything, and the other three nodes that do work don’t have any free space to add anything.
The key thing to understand is that you don’t get paid for the actual space used on your disk. Instead, you get paid based on the satellite’s calculation of what should be stored on your disk. This can lead to a significant discrepancy, as we’ve seen happen multiple times in the past. The satellite’s estimate of disk usage can be vastly different from the actual space occupied, resulting in a substantial difference in earnings and believing how your node is doing.
I would hope that the sequent GC would collect not so much amount of data to be moved to the trash and then deleted. We accumulated it for months before the fix. We also increased the frequency of BF, so it should help to distribute the deletion load more evenly.
My node is still deleting the trash too since 2024-09-06 for SLC.
Hello @Rassah,
Welcome to the forum!
You need to check the payout history for a previous month. The satellites doesn’t use the estimation showed on your dashboard to calculate payouts (but the node is doing so), they calculates what’s exactly the customer used and paid, thus your node paid accordingly the orders, submitted by your node. Each order has an exact amount of data stored or downloaded, so no place for any estimation, it’s pretty accurate. The only downside, that you can see these numbers only when the payout is completed (after the first two weeks of every month), and they may not match the estimations, calculated by your node for the same period.
$1.83 two months ago when it was working ok, $0.92 for July when reported disk space usage suddenly dropped in the middle of the month, and $0.06 for August that just got paid a few days ago, currently in Undistributed payout. I left it alone thinking that the estimation is wrong and I would get paid the real amount as you suggested. But with payment being $0 from saltlake despite saltlake data taking up almost3 gigs, I posted this.
I had 10TB trash of which less than a quarter got emptied in 7 days. Each xx directory took betwenn 15 minutes and 1 hour to get emptied, depending on other filewalkers, garbage collection etc.
I stopped this node now and moved the 2024-08-25 folder out of the trash folder somewhere else on the same filesystem. This should finish instantly. Then I started the node again and let the used space filwalker run.
Now I’m running
find /path/to/moved_folder/ -delete
in a tmux session, which deletes ~10 xx folders per hour. The remaining folders should be deleted in less than 4 days now. If required I can interrupt it so it doesn’t compete with the filewalkers etc.
DISCLAIMER: DO THIS ON YOUR OWN RISK! If you specify the wrong folder in the find command or have any spaces at the wrong position you could wipe your whole system! Also, do this only with a trash folder where the empty trash has already started. Otherwise you could get disqualified!
3 Likes
So there have been “emptying trash” jobs running from 9/2 to today 9/9 continuously? Is this just for one satellite or multiple satellites?
Yes, for me it was continuously since 2024-09-02 and it deleted about a quarter (~250 xx folders) of the trash from SLC (10TB in total). This morning I stopped it and started deleting with find as written above. It already deleted > 100 xx folders in less than 12 hours.
3 Likes