After the recent announcements of payout cuts, i took a look at the dashboard of my node.
I quickly noticed that the “Average Disk Space Used This Month” was around 20% lower than the “Total Disk Space”.
After checking the payouts of the last month and the actual disk usage of the storage folder on my hdd, i concluded that this was not just some display issue and there was around 2.2TB of unaccounted data on my node.
After looking trough the forum, i found this post which describes a similar issue: 4 TB node has been full the past few months, still only 2.17 TBm storage paid - #48 by Ottetal
However this seems to have been solved by a satellite update months ago and there is nothing i can/need to do.
I also found this post about forgetting the recently removed test satellites: How To Forget Untrusted Satellites
I did the steps described in there and indeed around 1 TB of data was removed for the satellites. I also repeated these steps to remove any possible data from the decommissioned stefan-benten satellite as my node is old enough to have used this one.
That still left me with >1TB of unaccounted data. I used ncdu and the satellite info post to check which satellite was using how much disk space and compared that to the averages that these satellites reported in the dashboard. I also used the earnings.py script with the debug lines to verify my findings.
The numbers are correct for all satellites except us1.storj.io, which consistently reports 1.1TB less than it actually uses:
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤
│ us1.storj.io:7777 (OK) │ │ │ $ 1.49/TBm $ 6.00/TB $ 6.00/TB 0% 100% │
│ 2019-05-14 54 │ $ 0.03 │ 0.00% 0.00% 0.05% │ $ 3.2232 $ 0.7501 $ 0.0855 -$ 0.0000 $ 4.0588 │
Disk Average So Far (debug): 2.29 TB >> 68% of expected 3.38 TB <<
├────────────────────────────────┼─────────────┼──────────────────────────┼─────────────────────────────────────────────────────────────┤ +
I run the node on a Linux host (Ubuntu 20.04) with docker and watchtower. The file system is ext4 and there is still more than enough space left on the disk as well as the configured max storage for the node.
Looking trough the logs, i only found one retain request from that satellite, and that one failed to delete a piece:
e[36mstoragenode |e[0m 2023-10-25T12:47:55Z INFO retain Prepared to run a Retain request. {"process": "storagenode", "Created Before": "2023-10-18T17:59:59Z", "Filter Size": 2097155, "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
e[36mstoragenode |e[0m 2023-10-25T14:01:25Z WARN retain failed to delete piece {"process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "NFYNEPEQ23263EBAUG3W5CL7JCVMAY3JJ3BTD4L7PFOCVU7I4VYA", "error": "pieces error: filestore error: file does not exist", "errorVerbose": "pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:110\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:245\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:290\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:404\n\tstorj.io/storj/storagenode/retain.(*Service).trash:364\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:341\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:221\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
Retain requests for all other satellites work as expected.
Is there anything i can do to fix this? Is there any way to tell the storage node to reindex what is there and compare to what should be there?