Node not starting, ERROR 503: Backend unavailable, connection timeout

Node Operators troubleshooting
, , , ,
1

Hi,

I am trying to restart my up to 3 year old docker nodes. Some started a few minutes ago, but now I get the below error on all nodes that I try to start. Looks like it is not able to download the latest storage node version?
Is someone experiencing the same issue?

Best regards

--2023-06-29 17:45:18--  https://github.com/storj/storj/releases/download/v1.80.10/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/c4517808-8dc9-4e95-a4bf-fb5767001680?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T174448Z&X-Amz-Expires=300&X-Amz-Signature=1805662a03aea9aaf50df5e6310771037d22955e4041600385312d1d955cec59&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
--2023-06-29 17:45:18--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/c4517808-8dc9-4e95-a4bf-fb5767001680?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T174448Z&X-Amz-Expires=300&X-Amz-Signature=1805662a03aea9aaf50df5e6310771037d22955e4041600385312d1d955cec59&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 503 Backend unavailable, connection timeout
2023-06-29 17:45:19 ERROR 503: Backend unavailable, connection timeout.

downloading storagenode-updater
--2023-06-29 17:45:45--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)... 34.173.164.90
Connecting to version.storj.io (version.storj.io)|34.173.164.90|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93 [text/plain]
Saving to: 'STDOUT'

     0K                                                       100%  355M=0s

2023-06-29 17:45:46 (355 MB/s) - written to stdout [93/93]

--2023-06-29 17:45:46--  https://github.com/storj/storj/releases/download/v1.80.10/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/c4517808-8dc9-4e95-a4bf-fb5767001680?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T174448Z&X-Amz-Expires=300&X-Amz-Signature=1805662a03aea9aaf50df5e6310771037d22955e4041600385312d1d955cec59&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
--2023-06-29 17:45:46--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/c4517808-8dc9-4e95-a4bf-fb5767001680?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230629%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230629T174448Z&X-Amz-Expires=300&X-Amz-Signature=1805662a03aea9aaf50df5e6310771037d22955e4041600385312d1d955cec59&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 503 Backend unavailable, connection timeout
2023-06-29 17:45:47 ERROR 503: Backend unavailable, connection timeout.
1 Like
2

same issue here

obrazek
obrazek1528×235 19.4 KB

i have a power outage, 18 containers start bud 2 have this error

3

I’m having the same issue on my docker container.

Edit: Looks like it is fixed now

4

Okay if nobody is able to start nodes anymore, that might quickly become a risk for the whole network…

5

windows gui nodes working fine

1 Like
6

Yes my nodes are also back online again :slight_smile:

But that shows that this is a single point of failure, that is a high risk for the decentralised network.
Wouldn‘t it be possible to distribute the binaries via the STORJ DCS?

1 Like
7

Thank @striker43, for reporting the issue.

It’s something that we should fix. If the node cannot download the latest binary, it should try to use the current one locally.

Wouldn‘t it be possible to distribute the binaries via the STORJ DCS?

It’s possible, but it isn’t straightforward because cheeky node operators could abuse it.

We have contemplated several times and have some thoughts about stopping the abuse, but we had other priorities.

2 Likes
8

Ah, you mean people could continuously download the binary on a large scale in order to artificially increase their node’s egress?
That would be bad, but might be solved by a kind of rate limiting or something similar.
Great to hear that you are working on a solution! Thank you

1 Like
9

Right ont the point.

That would be bad, but might be solved by a kind of rate limiting or something similar.

Yes, but we need to put a proxy in front and then it will be a single point of failure as Github is now.

2 Likes
10

That’s true…
(20 characters)

11

I created an issue in our repository Storagenode should run with the local binary when latest binary cannot be downloaded · Issue #6009 · storj/storj · GitHub

4 Likes
12

I think it could be useful to mark this object download as unpaid. This will prevent an abuse. But it’s hard to implement right now.

1 Like