Exceeded usage limit, with no files in bucket - 1 TB of Zombie segments?

[Th3Van]

I’ve been running a script to fill a bucket to the 1TB limit and remove all the files again for a few days now.
But for some reason I’m not able to upload anymore.
Waited like 7 hours and still cant upload any files.

Overview page on the satellite shows 1TB used, despite no files exist in the bucket.

32.00 KiB / 9.31 GiB [>____________________________________________] 0.00% ? p/s
Error: metainfo error: Exceeded Usage Limit; metainfo error: Exceeded Usage Limit

root@server030:~# date -u
Mon Feb 10 03:32:37 UTC 2020
root@server030:~# 
root@server030:~# uplink ls 
BKT 2020-01-29 02:22:18 test
root@server030:~# 
root@server030:~# uplink ls sj://test
root@server030:~# uplink --log.level debug  ls sj://test
2020-02-10T04:33:22.236+0100    INFO    Configuration loaded from: /root/.local/share/storj/uplink/config.yaml
2020-02-10T04:33:22.236+0100    DEBUG   debug server listening on 127.0.0.1:40031
2020-02-10T04:33:23.189+0100    DEBUG   allowed minimum version from control server is: v0.26.0
2020-02-10T04:33:23.189+0100    INFO    running on version v0.31.12
root@server030:~# 

7 hours later …

root@server030:~# date -u  
Mon Feb 10 10:11:57 UTC 2020
root@server030:~# 
root@server030:~# uplink ls
BKT 2020-01-29 02:22:18 test
root@server030:~# uplink ls sj://test                   
root@server030:~# uplink --log.level debug  ls sj://test
2020-02-10T11:12:16.753+0100    INFO    Configuration loaded from: /root/.local/share/storj/uplink/config.yaml
2020-02-10T11:12:16.753+0100    DEBUG   debug server listening on 127.0.0.1:35875
2020-02-10T11:12:17.298+0100    DEBUG   allowed minimum version from control server is: v0.26.0
2020-02-10T11:12:17.298+0100    INFO    running on version v0.31.12
root@server030:~#

32.00 KiB / 9.31 GiB [>____________________________________________] 0.00% ? p/s
Error: metainfo error: Exceeded Usage Limit; metainfo error: Exceeded Usage Limit

Could this be a result of 1 TB Zombi segments?

Support has been notified, and I’ll post in here as it progresses.

[Th3Van]

Looks like it’s a known issue by the dev team.
They are currently working on a fix.

[blu]

@Th3Van Hello I am going through the same problem as well and was wondering if there has been a fix yet. I was also wondering if this problem is within a single API key level or the whole account level, thank you.

[Th3Van]

Hello @blu, and welcome the forum!

I have not tried with a new API key, but I think the fix are on its way :
https://review.dev.storj.io/c/storj/storj/+/764

[kinor]

In our case, we did remove all the buckets and files testing the system and it seems according to https://review.dev.storj.io/c/storj/storj/+/764 our accounting is not working due to a bug related to unaccountable transactions. Since we are at a stand still, what is a potential short-term solution?

[heunland]

Our dev team is currently working on a fix, unfortunately I do not currently have an ETA for when the solution will be available yet. Thanks for your patience!

[Th3Van]

Any progress on this issue?

Six days ago, I did a 30 x 1TB concurrent upload test that all failed @ ~24TB → Uploads limited to 24 hours?

The upload issue has since been fixed, but I have not preformed the upload test again, because I’m being charged ~$8 each day since the upload test failed.
I’m currently at ~$116.88 in cost for storing 0 object and 32TB of “zombie/failed” storage on the network, that I’m not able to remove/delete myself.

If the 30TB upload test fails again, I could end up being charged ~$16 daily, and the $125 I’ve deposit to the EU satellite, will all be spend on zombie/failed storage, and not on legit storage

//Th3Van