Storj container unable to download storagenode-updater

Node Operators troubleshooting
,
1

Hi all,

I just tried to setup my first node, however, I can’t get it to work.
When I run the container on my debian 11 machine, I get the following logs:

downloading storagenode-updater
--2022-10-22 02:00:28--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)... 35.232.172.28
Connecting to version.storj.io (version.storj.io)|35.232.172.28|:443... connected.
ERROR: The certificate of 'version.storj.io' is not trusted.
ERROR: The certificate of 'version.storj.io' doesn't have a known issuer.
The certificate's owner does not match hostname 'version.storj.io'
http://: Invalid host name.

I used this command:

docker run -d --stop-timeout 300 \
    -p 28967:28967/tcp \
    -p 28967:28967/udp \
    -p 127.0.0.1:14002:14002 \
    -e WALLET="0x0..." \
    -e EMAIL="XXX" \
    -e ADDRESS="XXX.duckdns.org:28967" \
    -e STORAGE="600GB" \
    --user $(id -u):$(id -g) \
    --mount type=bind,source="/mnt/raid/apps/storj/identity",destination=/app/identity \
    --mount type=bind,source="/mnt/raid/apps/storj/config",destination=/app/config \
    --name storagenode docker.io/storjlabs/storagenode:latest

I also tried it with podman, but it is exactly the same.
Adding the

--dns 1.1.1.1

did not change anything, either.

Do you have any ideas what could be wrong?
Thanks
Reodaudo

2

Hello @Reodaudo,
Welcome to the forum!

Please install/update ca-certificates package in your host system. The Certificate on version.storj.io is valid, just your OS do not know the root certificates or have the old ones.

3

For some reason this is not required for unRAID. Does anyone know the reason for this?

4

Hello Alexey,

thanks for your reply. An old version of ca-certificates does not seem to be the reason for my troubles. I updated (update-ca-certificates) and reinstalled ca-certificates, both did not change the behaviour. However, curl https://version.storj.io:443 works without issues on the host.

5

I cannot reproduce that

$ docker pull storjlabs/storagenode
Using default tag: latest
latest: Pulling from storjlabs/storagenode
c32ce6654453: Pull complete
9e3f12e9f0b9: Pull complete
baabb14e0092: Pull complete
b23156875490: Pull complete
5907f5f7011c: Pull complete
dff5c6ba6652: Pull complete
efc8248e2380: Pull complete
e9c76c2e1d75: Pull complete
951fc6673570: Pull complete
a1334c8c6f6b: Pull complete
c365a770c3ef: Pull complete
Digest: sha256:f5614a2b086801c1cc834cd5120442eccf92e83272aae660865467bf4caa010c
Status: Downloaded newer image for storjlabs/storagenode:latest
docker.io/storjlabs/storagenode:latest

$ docker run -it --rm storjlabs/storagenode:latest
downloading storagenode-updater
--2022-10-22 10:59:29--  https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)... 35.232.172.28
Connecting to version.storj.io (version.storj.io)|35.232.172.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 92 [text/plain]
Saving to: 'STDOUT'

-                             100%[=================================================>]      92  --.-KB/s    in 0s

2022-10-22 10:59:30 (36.6 MB/s) - written to stdout [92/92]

--2022-10-22 10:59:30--  https://github.com/storj/storj/releases/download/v1.64.1/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/c508a985-e222-4940-9de7-98a576854bc5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221022%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221022T105829Z&X-Amz-Expires=300&X-Amz-Signature=22e841a9e2a44ebf55961a149ed98d479c84f2eaf38d5ac57f9892fb7505606e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
--2022-10-22 10:59:31--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/c508a985-e222-4940-9de7-98a576854bc5?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221022%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221022T105829Z&X-Amz-Expires=300&X-Amz-Signature=22e841a9e2a44ebf55961a149ed98d479c84f2eaf38d5ac57f9892fb7505606e&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8438278 (8.0M) [application/octet-stream]
Saving to: '/tmp/storagenode-updater.zip'

/tmp/storagenode-updater.zip  100%[=================================================>]   8.05M  3.98MB/s    in 2.0s

2022-10-22 10:59:34 (3.98 MB/s) - '/tmp/storagenode-updater.zip' saved [8438278/8438278]

downloading storagenode
--2022-10-22 10:59:34--  https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)... 35.232.172.28
Connecting to version.storj.io (version.storj.io)|35.232.172.28|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 84 [text/plain]
Saving to: 'STDOUT'

-                             100%[=================================================>]      84  --.-KB/s    in 0s

2022-10-22 10:59:35 (7.28 MB/s) - written to stdout [84/84]

--2022-10-22 10:59:35--  https://github.com/storj/storj/releases/download/v1.64.1/storagenode_linux_amd64.zip
Resolving github.com (github.com)... 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/2a4327a3-6729-4344-83b4-ff9724f24e2b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221022%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221022T105936Z&X-Amz-Expires=300&X-Amz-Signature=af8b651663cbb27b3272c55f9b09006cb5c374b001fa41693176079e55a7375d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
--2022-10-22 10:59:35--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/2a4327a3-6729-4344-83b4-ff9724f24e2b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221022%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221022T105936Z&X-Amz-Expires=300&X-Amz-Signature=af8b651663cbb27b3272c55f9b09006cb5c374b001fa41693176079e55a7375d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16097553 (15M) [application/octet-stream]
Saving to: '/tmp/storagenode.zip'

/tmp/storagenode.zip          100%[=================================================>]  15.35M  8.81MB/s    in 1.7s

2022-10-22 10:59:38 (8.81 MB/s) - '/tmp/storagenode.zip' saved [16097553/16097553]

2022-10-22 10:59:38,643 INFO Set uid to user 0 succeeded
2022-10-22 10:59:38,651 INFO RPC interface 'supervisor' initialized
2022-10-22 10:59:38,651 INFO supervisord started with pid 1
2022-10-22 10:59:39,656 INFO spawned: 'processes-exit-eventlistener' with pid 42
2022-10-22 10:59:39,666 INFO spawned: 'storagenode' with pid 43
2022-10-22 10:59:39,673 INFO spawned: 'storagenode-updater' with pid 44

Please try to pull the latest version of the image

docker pull storjlabs/storagenode:latest
6

Yeah, I can’t reproduce it on my laptop either.
Neither deleting and downloading the image again nor removing the arguments from the docker run command makes any difference.

7

I just tried running the image with podman and the option --network=host, and it fixes the problem. However, that is not a setup I feel comfortable running with. And it doesn’t help me to understand what is going on on that machine.

8

Hi @Reodaudo,

Thank you for the info.
Is it correct to assume that’s failing when you’re using the network=bridge which is the default option?

9

Oh, sorry, I forgot to update this threat with my new finding.
Yes, it is correct that it fails with the bridge option. With the option --network=slirp4netns the download starts as expected.
I know to little about container networking to explain this - I always assumed that bridge was the most robust option.

10

I think with Ubuntu it will work without additional hacks

11

downloading storagenode-updater
–2023-04-21 11:34:44-- https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)… 34.173.164.90
Connecting to version.storj.io (version.storj.io)|34.173.164.90|:443… failed: Connection timed out.
Retrying.

discovered that it is related to some IP`s subnet blocked at server side (anti DDoS ? ). storj need provide alternate backup https servers to distribute updates.
Multiple nodes was affected, so ip need to pack them on single IP to keep it running and do not lose user data.

12

You need to unblock this IP or disable the extended security on your router or ISP.

13

it is blocked on 34.173.164.90 (version.storj.io) - storj hosts its stuff on google. so google banned entrie subnet (tested). maybe some of other ISP`s customers have malware or bots which triggers google protection, i dont know. but this behaviour is weak point - auto update need to use multiple mirrors.

14

It’s reverse. Your ISP is blocked this IP, not Google. Or Google could block your subnet - but it’s a rare case, like connecting from sanctioned countries.

I would suggest to contact your ISP first to unblock this IP and all our IPs, you may check them by nslookup.

15

i have the same issue. 34.173.164.90 is not blocked by isp, ping and tracepath work fine… but port 443 doesnt return packets

16

Hello @obvoaadd,
Welcome to the forum!

You may check it on Open Port Check Tool - Test Port Forwarding on Your Router
image

So, it is blocked in your location. And there are not many options, either your firewall, or smart security on your router or your ISP (or somewhere in between your ISP and this IP).

17

If i had port 443 blocked i wouldn’t be able to write you this message. I stated that i can reach the server in question. You must have some geoip or whatever rules set on the firewall which blocks “wrong” source ip. Please just notify someone competent in you company. Some network admin and sysops/devops who handles this and he will know what to do/check.

18

It could have filter only against sanctioned countries. What’s your IP?
In other cases it’s a client side only.
I’m not talking about blocking 443 port, but this IP and port or just this IP (or even subnet).

19

I can ping and tracepath the server successfully so nothin is blocked on my side. I can write you my ip in private message

20

Erm it seems this forum doesnt support direct/private messaging.

188.159.235.xxx