I don’t mess around with my storj setup, I just leave it to do its thing. So I had not noticed that storj started using more and more storage 2nd half of July. The web UI shows that it is not using more than it should. My file system begs to differ. According to my monitoring system, usage was more or less along a flat line until beginning of July when it decreased by ½. Then around the 20th of July usage started increasing and is now using 50% more than it should. System was up since before the increase in storage started, so I rebooted the server last night. Now 24 hours later, storage use is still growing.
Screenshot from monitoring platform attached to show the growth.
Any suggestions as to what is potentially going on? What I should check? OS is Linux, filesystem is ZFS, storj (v1.87.1) is running in docker container.
I’ve suspected for a while it just checks out the free space on the storage partition and goes for it based on what’’s available not what you advertise as available. This makes sense in that the system needs proof of what’s available and can’t rely on published values which can change over time or are inappropriately set.
This looks like a difference between a record size and size of pieces. Do you have a compression enabled on this dataset/pool? What’s you ashift?
@Unique Storj doesn’t work like this. We do not use a “proof of storage” any kind.
If we would implement something like this, it likely will be on the dashboard and I hope it will be paid. But now it’s not the case.
That’s the problem. Turn compression back on, and then zfs send | zfs receive to adopt that for the existing files (follow node migration guide, adapted for zfs, and then you can rename the dataset back to the original name in the last step)
Generally keep settings at default, unless you have a very good reason to change them. Compression is on by default.
You can check the (hopefully, left at default) record size with zfs get recordsize pool1/storagenode. If you dont’ have compression enabled, you are wasting massive amount of storage, as most of storj files are very small.
Compression was never on for this pool. This system has been running as a storj node since May 2020. Once it filled up with the 2 TB of data I allocated for storj, it has been pretty constant in terms of storage used. Beginning of July this year, it first started loosing a lot of data (I’m guessing this was when storj stopped sending test data). It dropped to around 1 TB, and then mid-July started eating storage. The pool now has almost 3 TB of data. As per the Web UI, storj only knows about 2 TB of data.
Have you tried comparing actual size with apparent size of the blob folder? If this does not match the discrepancy — then it’s probably indeed stale data, it shall be taken care of filewalker/garbage collector.
Compression or no compression. I’ve told storj to use no more than 2 TB. It has kept its side of the bargain.Suddenly, the pool given to storj is using ever more space. Something isn’t right.
Well, file size and space the file takes are not the same things. Check the difference on the blobs folder. It’s pointless to hypothesize when you can just check.
