Node fails auto update

Node Operators troubleshooting
, , ,
1

Have 3 nodes, all setup the same running Debian and the auto version update now always fails. I would say it’s been the last 3 or 4 updates so started a few months ago. Error output below. A restart gives the same failure but if I reload the container it runs fine and has the new update installed.

2024-08-02T22:55:52Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.108.3”}
2024-08-02T22:55:52Z INFO Download started. {“Process”: “storagenode-updater”, “From”: “https://github.com/storj/storj/releases/download/v1.109.2/storagenode_linux_arm64.zip”, “To”: “/tmp/storagenode_linux_arm64.1444416309.zip”}
2024-08-02T22:55:58Z INFO Download finished. {“Process”: “storagenode-updater”, “From”: “https://github.com/storj/storj/releases/download/v1.109.2/storagenode_linux_arm64.zip”, “To”: “/tmp/storagenode_linux_arm64.1444416309.zip”}
2024-08-02T22:55:59Z INFO Restarting service. {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2024-08-02T22:56:00Z INFO Service restarted successfully. {“Process”: “storagenode-updater”, “Service”: “storagenode”}2024-08-02T22:56:00Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.108.3”}
2024-08-02T22:56:00Z INFO Download started. {“Process”: “storagenode-updater”, “From”: “https://github.com/storj/storj/releases/download/v1.109.2/storagenode-updater_linux_arm64.zip”, “To”: “/tmp/storagenode-updater_linux_arm64.3921128788.zip”}
2024-08-02 22:56:00,358 INFO exited: storagenode (exit status 0; expected)
2024-08-02 22:56:01,365 INFO spawned: ‘storagenode’ with pid 20442
2024-08-02 22:56:01,379 WARN received SIGQUIT indicating exit request
2024-08-02 22:56:01,380 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2024-08-02T22:56:01Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2024-08-02T22:56:01Z ERROR Error updating service. {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “error”: “Get "https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/a530643f-e1de-4889-a518-f34c5d1069d7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240802%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240802T225600Z&X-Amz-Expires=300&X-Amz-Signature=182ac014af3b67fed473e990bc0ca64486850c5268bd49e42f623c4065b35191&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm64.zip&response-content-type=application%2Foctet-stream\”: context canceled", “errorVerbose”: “Get "https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/a530643f-e1de-4889-a518-f34c5d1069d7?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240802%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240802T225600Z&X-Amz-Expires=300&X-Amz-Signature=182ac014af3b67fed473e990bc0ca64486850c5268bd49e42f623c4065b35191&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_arm64.zip&response-content-type=application%2Foctet-stream\”: context canceled\n\tmain.downloadBinary:58\n\tmain.update:39\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tmain.cmdRun:138\n\tstorj.io/common/process.cleanup.func1.4:393\n\tstorj.io/common/process.cleanup.func1:411\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:271"}
2024-08-02 22:56:01,386 INFO stopped: storagenode-updater (exit status 0)
2024-08-02 22:56:01,390 INFO stopped: storagenode (terminated by SIGTERM)
2024-08-02 22:56:01,391 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

2

Something is blocking your connection to GitHub.
Can you download it from the host?

wget https://github.com/storj/storj/releases/download/v1.109.2/storagenode-updater_linux_arm64.zip
3

It download manually fine.
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 14847118 (14M) [application/octet-stream]
Saving to: ‘storagenode-updater_linux_arm64.zip’

storagenode-updater_linux_arm6 100%[====================================================>] 14.16M 26.8MB/s in 0.5s

2024-08-06 15:03:06 (26.8 MB/s) - ‘storagenode-updater_linux_arm64.zip’ saved [14847118/14847118]

4

Could you please try to add --network host before the image name in your docker run command?
Please note - this will expose all container’s ports to the host. So, if you have several nodes here, you need to change their console, server and the private addresses too:

I suspect that Debian has did something with the docker’s NAT or with routing tables.
This is not a first Debian where docker container has a connectivity issues to GitHub.

5

They currently all run with —network host. Each host is on a seperate ARM server.

6

Could you please try to run a container with Debian Bookworm and execute the same wget command?

Also, you may try to remove the --network host and check - did it help?

7

They run bookwork with the latest LTS kernel 6.6. I did test it both inside the container and outside and it worked fine. Also the files are already downloaded as I can see then in the containers app directory. I don’t think it is a problem downloading the files as reinstalling the docker container it comes up with the new version. If I just restart the container it goes into the update sequence again and fails.

8

Please try to delete an image

docker stop storagenode
docker rm storagenode
docker rmi storjlabs/storagenode

After that try to run it again

9

I will try that. Won’t know if it works until they are next updated. ta.