Using the free trial on the EU1 satellite with the old 150GB (150*10^9) allowance, I have a single bucket which a recursive s3 ls shows as 142 objects, 148832439976 bytes used. When I try to upload a 1GiB (2^30) file (16 segments of 64MiB), which would take the total to 149906181800 (under 150GB), I receive the error “An error occurred (StorageLimitExceeded) when calling the UploadPart operation: You have reached your project storage limit.”
Looking at the same bucket from the satellite dashboard, I see 142 objects, 149.15GB used. I guess this higher used amount explains why I’m receiving the error, but I don’t understand where the higher storage value is coming from. That value does not decrease after not touching the bucket for a couple days, so I don’t think it’s just requests that haven’t settled.
Is there some hidden metadata storage usage that’s being counted in the dashboard but that isn’t visible through s3, and is there some way to view the storage usage programmatically other than with aws s3 ls --recursive like I’ve been doing?
There also could be objects with different encryption phrase
uplink ls --recursive --encrypted sj://my-bucket | wc -l
and
uplink ls --recursive sj://my-bucket | wc -l
should be the same.
But most likely that there is a problem on the edge of the limit. Your client is trying to allocate more, than available. I also do not exclude a rounding error somewhere though.
You may also try to use rclone ncdu us1: (us1 here is a remote’s name) and toggle to bytes with u.
You may upgrade your account to PRO and set the old limits yourself. This way you would not be charged until you go over $1.65 coupon.
Thanks for the welcome, the answer, and the project in general!
uplink ls --pending sj://my-bucket gives no output.
uplink ls --recursive --encrypted sj://my-bucket | wc -l gives 143 uplink ls --recursive sj://aws | wc -l also gives 143
After configuring rclone (with native storj), it shows the remote as 148832439976 (matching s3 view).
To double check that my backup script isn’t sending too large of a chunk (was just using head -c), I tried copying a file rather than reading from a FIFO on stdin. ls shows file size as 1073741824.
It gave the same error when copying with aws s3 CLI, progress showed about 998 MiB of the copy completed before failing.
So seems like there might actually be some small error in the storage limit checking?
Good idea of creating a pro account, I didn’t realize they could set limits, I might go ahead and do that since it would also give flexibility on downloading the backup in full, in case a single transfer is interrupted I could pay a couple cents instead of waiting weeks to finish recovering. I might also just add a 200MB fudge factor to my project sizes since that seems to be about what I need so far.
Storage is calculated based on bytes uploaded, including any encryption-based overhead which is not visible through s3. In the example on the billing page a 1TB file with the encryption overhead is stored as 1.001TB, which is 0.1% overhead. If the same encryption overhead percentage holds true on your files it is enough to put you over the limit. 149906181800 * 1.001 = 150056087982
Aha thanks pwilloughby, that probably explains it. That gives me some new questions,
How does segment size tracking/billing relate to this? I’ve been uploading segments of exactly 64MiB in my 1GiB files, but after encryption overhead wouldn’t each of these be slightly over 64MiB? It doesn’t seem like those are billed, but does it create inefficiency for the network and SNOs?
Is there any way to view the encrypted size? Even with uplink ls --encrypted it shows the same file size as the unencrypted versions. This makes it a bit hard to programmatically view free space in a project. I guess this is minor since butting up against project limits is probably more of a free trial problem.
it will, but it’s better than use 5MiB segments anyway. You may provide a less chunk size of course, and it will utilize the same or greater number of segments.
I don’t think so. The size is taken from the object’s System Metadata (ContentLength), it’s not measured on the fly in the uplink.