And yet my filesystem saids 14.49TB is being used. Why don’t the dashboard’s figures even agree, and why are they both under what’s actually used?
Hello @kchiem
Welcome to the Forum!
I’ve forwarded your question and someone will return with an answer soon.
1 Like
Please check the following, this can have various causes.
Yeah, I believe this should be documented better. @Alexey, please consider the following text as public domain/CC0 and if you find the text below accurate (please verify!), feel free to edit it accordingly and incorporate it in storage node documentation.
Average Disk Space Used This Month is the sum of bytes of pieces that the satellite considers as stored. This is what you get paid for.
Used Disk Space is the sum of bytes of pieces that the node believes it is supposed to store. This includes files that were deleted from the satellite’s perspective, but not yet garbage-collected by your node. Garbage collection happens once a week per satellite. This number is updated during garbage collections and during the file walker executed each time your node is started (unless you disabled it). Sometimes the node cannot update these numbers properly, for example when the I/O load is so high that the bandwidth.db file becomes locked, and as such this number may be imprecise. It may also include outdated satellites, though I admit I don’t know how true this is. If you see a large discrepancy here, please make sure the initial file walker is enabled (this is the default), restart your node, and wait until the file walker finishes. And if you see that your node still contains data for satellites no longer trusted, run the forget satellite procedure.
On my nodes the difference between the last Average Disk Space Used This Month value and the Used Disk Space number are usually on the order of low hundreds of gigabytes on an active node, but it did happen sometimes that I’ve seen larger discrepancies. As such, I’m not really surprised that you see this kind of difference, but if it stays at this level for a month, then it would be suspicious.
Both of the above numbers are reported using SI units, i.e. 1kB = 1000 bytes, 1MB = 1000000 bytes, and so on.
Each piece is a separate file on disk. For each file your file system needs to store some metadata (file name, location on disk, attributes, permissions, and so on, usually between 150 and ~1 kB per file). There is also some additional overhead involved: the file itself is usually rounded up to full sectors/clusters (usually 4 kB, sometimes as large as 2 MB though!), as in most file systems a single cluster cannot be shared between two files, even if the file contents do not fill it. The specific numbers depend on the file system used.
I would expect the total overhead of 10 TB worth of node data, assuming around 36 M files, to be:
- around 85 GB for a typical ext4 file system,
- at least 110 GB for NTFS with cluster size of 4 kB,
- at least 340 GB for NTFS with cluster size of 16 kB,
- at least 2.4 TB for default-formatted exFAT (cluster size of 128 kB; by the way, this is a nice description of the relation of cluster size to space efficiency in exFAT).
(Though I admit I do not enough about NTFS/exFAT to accurately estimate its overhead).
In addition to that, some operating systems report file system usage using IEC 60027-2 units, i.e. 1 kiB = 1024 bytes, 1 MiB = 1048576 bytes. Sometimes even they do that while using SI names (kB instead of kiB, MB instead of MiB, etc.) which adds to confusion. Here I’m only using SI units.
The number for exFAT kinda matches your observations, so… are you using exFAT?
3 Likes
I also wonder about this. One of my nodes which has been at capacity for a while (thus not growing or fluctuating much) uses 3.13TB of actual space, shows 3.19TB on the “My Nodes” section of the multinode dashboard, 2.7TB in “Total Disk Space” (3.03TB used, 161GB Trash, 494GB overused), and 1.78TB in the “Average Disk Space Used This Month”. The payouts screen lists it as 1.83TBm for last month. Where is all this space going?
I added some comments about untrusted satellites, please verify whether this is the case for you.
1 Like
I’m not looking at the average for the month. I’m looking at the most recent data point in that average, which I would think should match the current usage on the right.
I’m using zfs, and the 14.49TB figure I provided for the filesystem is only the total of the files, not what’s used on the disks with all the filesystem overhead.
It might have something to do with garbage collection and the file walker activity, as the pool it’s on tends to be io-bound lately. I’m going to move this to another more idle pool to see if things improve.
1 Like
Oh dear, I have zero idea how zfs lays out data, especially with all the possible ways zfs can be configured. Sorry! Though, there are some vocal zfs users here on the forum, and they are sometimes even helpful.
yes, they usually should match, unless you have errors related to gc-filewalker, lazyfilewalker and retain in your logs.
I suspect context canceled errors (high disks load) and/or FATAL errors (the filewalker update the database only when it’s successfully finished, otherwise it will start from the beginning on each restart).
I’m also using ZFS. I had some issues in the past that were causing a lot of wasted space, but I have since fixed those. Even for new nodes where I never had those issues to begin with, I still see a discrepancy.
Please search for these errors in your logs
https://forum.storj.io/t/disk-usage-discrepancy/24715/10?u=alexey
I am having something similar where the local filesystem is at 16 TB while the node claims to be at 12 TB and no space left. I have not started to investigate what is going on there, I am still hoping it will resolve itself with a complete and successful filewalker pass.
not until the node is full.
Even at my fully cached 20TB drive there is a 0.13TB Difference (2.2TB data)
I only see about a dozen INFO lines each for gc-filewalker and lazyfilewalker in the logs, and about a dozen ‘retain’ WARN lines, something like:
WARN retain failed to delete piece {“process”: “storagenode”, “Satellite ID”: “XXX”, “Piece ID”: “YYY”, “error”: “pieces error: filestore error: file does not exist”, “errorVerbose”: “pieces error: filestore error: file does not exist\n\tstorj.io/storj/storagenode/blobstore/filestore.(*blobStore).Stat:110\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).pieceSizes:245\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).Trash:290\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:404\n\tstorj.io/storj/storagenode/retain.(*Service).trash:364\n\tstorj.io/storj/storagenode/retain.(*Service).retainPieces:341\n\tstorj.io/storj/storagenode/retain.(*Service).Run.func2:221\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75”}
Nothing for “context canceled”.
There were some filewalker processes running yesterday, but they’re not running now, so I’ll assume they completed. Still, the discrepancy remains.
And what’s going on with the download and repair/audit rates?
Hasn’t it been $6/tb for those for a while? When will the dashboard be updated to reflect that?
Also, why is the Payout estimated at $2/tb?
Edit: nevermind. It looks like the updater only updated the binary component and not the web component. I updated that and it shows a link to the forum post about the rate reduction.
It should be completed for the each satellite. The same for retain.
yes.
when you (or the updater) will restart the node. It doesn’t load most of payout-related info until restart.
it’s estimated as $9.72 as a payout for 6.52 TBm, or $9.72 / 6.52TBm ~ $1.5/TBm
What do you mean for each satellite? I’m only running one storage node.
I already stated why the dashboard wasn’t updated. The updater restarts the storagenode process itself.
Not what I was referring to, but that’s irrelevant now.
Are there supposed to be nearly 42 million files under the storage/blobs folder? The total of the files there is only 10.5 TB, which makes the average file size only 271 kb.
Isn’t possible that the satellites have wrong data? You all throw the blame on nodes, but maybe there is a bug in code or something bad designed, that makes sats to have wrong data.
All my nodes with ext4 and 18GB RAM shows this discrepancy, but not the 1GB one. This one has 2x 6TB data (2nodes), the other ones have over 9TB of data. Maybe the discrepancy increases with more data? I admit I had the FW off, but for all, including the small one. I reactivated it a week ago and I don’t see any improvements. I will start to take screen shots one week apart, for all, to document the behavior. I recommend others do the same. But again, you should check the sats as well. As we seen it from a SNO perspective, we are loosing money.
1 Like