I’m using signed URLs to allow users to upload files from their browser. I just had a user get a 500 error on a PUT request signed URL upload (some fields redacted)
I’ve probably close to hundreds of other uploads go without issue. Any idea why this one may have failed?
EDIT: It seems that I’m getting the issue as well. My upload seems to be stuck without completing, I also can’t seem to open any of my buckets in the UI, they’re just timing out with a spinner. Is Storj having an outage??
The network request in the UI eventually returned after ~2m with:
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InternalError</Code><Message>We encountered an internal error, please try again.: cause(uplink: metaclient: rpc: tcp connector failed: rpc: dial tcp 184.108.40.206:7777: i/o timeout)</Message><BucketName>uploads</BucketName><Resource>/uploads</Resource><RequestId>1707CC67C0F00544</RequestId><HostId></HostId></Error>
The outgoing request which is timing out with the error is:
If that helps in any way
Ok, service now seems to be restored.
The downtime by my estimate was from 08:25:45 UTC time until 10:08 am (now).
What was going on??
Shoot, sorry this affected you! A victim of our own success to some degree - we’ve had a lot of load recently! Some error rates have been a bit higher (largely due to internal timeouts).
We’ve been working on scaling up but I’m always bummed when scaling challenges are noticeable. We’re bringing on a lot more Satellite and gateway capacity soon. Thanks for your patience!
All good. It’s a bit stressful when stuff like this happens but it’s unclear that Storj knows theirs an issue / the status page isn’t updating