Upgrade to v1.68.2 failed, container crashed

Node Operators troubleshooting
,
1

I have two nodes running with docker image: storjlabs/storagenode:67a9c1135-v1.66.1-go1.17.12-arm64v8, both of them crashed today, some logs here

2022-12-07T07:47:10.066Z	INFO	Download finished.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.68.2/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.3043625810.zip"}
2022-12-07T07:47:10.125Z	INFO	Restarting service.	{"Process": "storagenode-updater", "Service": "storagenode"}
2022-12-07T07:47:10.281Z	INFO	piecestore	uploaded	{"Process": "storagenode", "Piece ID": "65HKV7W3OJEDLKH36AIUWUZQLTI5HYKHSKCHKLGA33Y32J6C2CPA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 435200}
2022-12-07T07:47:11.142Z	INFO	Service restarted successfully.	{"Process": "storagenode-updater", "Service": "storagenode"}
2022-12-07T07:47:11.142Z	INFO	Got a signal from the OS: "interrupt"	{"Process": "storagenode"}
2022-12-07T07:47:11.145Z	INFO	piecestore	upload canceled	{"Process": "storagenode", "Piece ID": "XT72RAFCBQZMNVP6OIMAJCBMK5HBYK4N6LPROZAXVLQYMGLFG3HQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 0}
2022-12-07T07:47:11.164Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.67.3"}
2022-12-07T07:47:11.164Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.68.2/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.2895538794.zip"}
2022-12-07 07:47:11,270 INFO exited: storagenode (exit status 0; expected)
2022-12-07 07:47:12,276 INFO spawned: 'storagenode' with pid 7642
2022-12-07 07:47:12,308 WARN received SIGQUIT indicating exit request
2022-12-07 07:47:12,309 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2022-12-07T07:47:12.310Z	INFO	Got a signal from the OS: "terminated"	{"Process": "storagenode-updater"}
2022-12-07T07:47:12.313Z	ERROR	Error updating service.	{"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "context canceled", "errorVerbose": "context canceled\n\tmain.downloadBinary:58\n\tmain.update:39\n\tmain.loopFunc:32\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tmain.main:20\n\truntime.main:250"}

after manual restart it works again, but turns out there’s an issue for the updater.

2

Seems stopping the service took too long on your hardware. How many nodes, how many CPU cores, how many RAM, what is filesystem on your HDD and how is it connected?

3

probably, I ran it on raspberry pi 4c8g external HDD connected via USB, ext4 filesystem

4

Does it happen before? Is your disk healthy?
Also, do you have --restart unless-stopped or --restart always in your docker run command?

5

I have a node on a RPi 3B with an external HDD and I also see strange behaviour. As soon as the node check for update, it stops responding. Single node on this RPi, nothing else is running on it. It has version 1.67.3. Just today it stopped 2 times for 3 and 5 hours… :frowning:

2022-12-07T12:32:08.115335419Z 2022-12-07T12:30:44.962Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2022-12-07T17:05:37.508231590Z 2022-12-07T17:05:36.509Z ERROR Error retrieving version info. {“Process”: “storagenode-updater”, “error”: “version checker client: Get "https://version.storj.io": context deadline exceeded (Client.Timeout exceeded while awaiting headers)”, “errorVerbose”: “version checker client: Get "https://version.storj.io": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tmain.loopFunc:21\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:377\n\tstorj.io/private/process.cleanup.func1:395\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:92\n\tmain.main:20\n\truntime.main:250”}
2022-12-07T17:05:37.662331195Z 2022-12-07T17:05:36.657Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}
2022-12-07T17:05:42.105626358Z 2022-12-07T17:05:42.104Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode”, “Version”: “v1.67.3”}
2022-12-07T17:05:42.105780784Z 2022-12-07T17:05:42.104Z INFO New version is being rolled out but hasn’t made it to this node yet {“Process”: “storagenode-updater”, “Service”: “storagenode”}
2022-12-07T17:05:42.272539827Z 2022-12-07 17:05:42,242 INFO exited: storagenode (terminated by SIGKILL; not expected)
2022-12-07T17:05:42.542417492Z 2022-12-07T17:05:42.538Z INFO Current binary version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.67.3”}
2022-12-07T17:05:42.542633376Z 2022-12-07T17:05:42.538Z INFO New version is being rolled out but hasn’t made it to this node yet {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”}
2022-12-07T17:05:42.547784484Z 2022-12-07 17:05:42,546 INFO spawned: ‘storagenode’ with pid 4751
2022-12-07T17:05:42.829517948Z 2022-12-07 17:05:42,828 WARN received SIGQUIT indicating exit request
2022-12-07T17:05:42.831032833Z 2022-12-07 17:05:42,829 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die
2022-12-07T17:05:42.832766778Z 2022-12-07T17:05:42.830Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}
2022-12-07T17:05:42.838423925Z 2022-12-07 17:05:42,837 INFO stopped: storagenode-updater (exit status 0)
2022-12-07T17:05:42.847244485Z 2022-12-07 17:05:42,846 INFO stopped: storagenode (terminated by SIGTERM)
2022-12-07T17:05:42.855279010Z 2022-12-07 17:05:42,854 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)
2022-12-07T17:05:56.713878484Z 2022-12-07 17:05:56,713 INFO Set uid to user 0 succeeded
2022-12-07T17:05:56.789434455Z 2022-12-07 17:05:56,788 INFO RPC interface ‘supervisor’ initialized
2022-12-07T17:05:56.790564049Z 2022-12-07 17:05:56,790 INFO supervisord started with pid 1
2022-12-07T17:05:57.799179771Z 2022-12-07 17:05:57,797 INFO spawned: ‘processes-exit-eventlistener’ with pid 11
2022-12-07T17:05:57.810333241Z 2022-12-07 17:05:57,809 INFO spawned: ‘storagenode’ with pid 12
2022-12-07T17:05:57.817548813Z 2022-12-07 17:05:57,816 INFO spawned: ‘storagenode-updater’ with pid 13

My other node which is on a RPi 4 8GB automatically updated itself to 1.68.2 and running without any issue.

6

This suggests some network issues. It cannot download even a small json file from the version.storj.io site. And I believe you cannot access it in the same time.
Maybe some hardware issue?