ERROR Error updating service

Node Operators troubleshooting
, , ,
21

i have no idea how did my node been upgraded to that version. im just basing it to the logs im seeing

2024-10-11T12:57:39Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.113.2"}
2024-10-11T12:57:39Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.114.6/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.3586801199.zip"}
2024-10-11T12:57:39Z	ERROR	Error updating service.	{"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "Get \"https://github.com/storj/storj/releases/download/v1.114.6/storagenode-updater_linux_arm64.zip\": context canceled", "errorVerbose": "Get \"https://github.com/storj/storj/releases/download/v1.114.6/storagenode-updater_linux_arm64.zip\": context canceled\n\tmain.downloadBinary:58\n\tmain.update:40\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tmain.cmdRun:138\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:271"}
2024-10-11 12:57:40,351 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-10-11 12:57:40,352 INFO stopped: storagenode-updater (exit status 0)
2024-10-11 12:57:41,489 WARN stopped: storagenode (exit status 1)
2024-10-11 12:57:41,492 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
22

thank you for this my node is back online. i need to download these two: storagenode and storagenode-updater, and override the existing files in the bin folder.

idk why my node is running on v1.114.6

image
image1259Ɨ139 34.7 KB

23

Seems because it cannot download it for some reason. But the previous log lines suggests that your node should be upgraded to 1.114.6 accordingly its NodeID.

24

Hi, my node keeps restarting in a loop every time a new version is released. I have to manually download the storagenode and storagenode-updater to get it working again. This has happened with at least three versions, or possibly more. Iā€™m wondering if thereā€™s a way to fix this on my side or node, so I donā€™t have to do this manually every time.

25

That is odd. Can you provide a log of the errors causing the reset?

26

Could you please remove the bin subfolder from the data location and re-create the container?

27

I tried this, but it didnā€™t resolve the issue.

28

from container

2025-01-10 10:41:56,937 INFO RPC interface 'supervisor' initialized
2025-01-10 10:41:56,938 INFO supervisord started with pid 1
2025-01-10 10:41:57,946 INFO spawned: 'processes-exit-eventlistener' with pid 36
2025-01-10 10:41:57,958 INFO spawned: 'storagenode' with pid 37
2025-01-10 10:41:57,963 INFO spawned: 'storagenode-updater' with pid 38
2025-01-10T10:41:58Z	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.details"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.path"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features.0"}
2025-01-10T10:41:58Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2025-01-10T10:41:58Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.117.8"}
2025-01-10T10:41:58Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-01-10T10:41:58Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.117.8"}
2025-01-10T10:41:58Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.2840522335.zip"}
2025-01-10 10:41:59,815 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-10 10:41:59,816 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-10 10:41:59,816 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-10T10:42:04Z	INFO	Download finished.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.2840522335.zip"}
2025-01-10T10:42:05Z	INFO	Restarting service.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-01-10T10:42:05Z	INFO	Service restarted successfully.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-01-10T10:42:05Z	INFO	Cleaning up old binary.	{"Process": "storagenode-updater", "Service": "storagenode", "Path": "/app/bin/storagenode.old.v1.117.8"}
2025-01-10T10:42:05Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.117.8"}
2025-01-10T10:42:05Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.171671116.zip"}
2025-01-10T10:42:15Z	INFO	Download finished.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.171671116.zip"}
2025-01-10T10:42:15Z	INFO	Restarting service.	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-01-10T10:42:15Z	INFO	Cleaning up old binary.	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Path": "/app/bin/storagenode-updater.old"}
2025-01-10 10:42:16,744 WARN exited: storagenode-updater (exit status 1; not expected)
2025-01-10 10:42:17,754 INFO spawned: 'storagenode-updater' with pid 169
2025-01-10 10:42:27,039 WARN received SIGQUIT indicating exit request
2025-01-10 10:42:27,042 INFO waiting for processes-exit-eventlistener, storagenode, storagenode-updater to die
2025-01-10 10:42:27,050 WARN stopped: storagenode-updater (terminated by SIGTERM)
2025-01-10 10:42:30,055 INFO waiting for processes-exit-eventlistener, storagenode to die

from storagenode

2025-01-10T10:44:59Z	FATAL	Unrecoverable error	{"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: database: info opening file \"/app/config/database/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:715\n\tstorj.io/storj/private/migrate.(*Migration).Run:168\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:415\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272", "errorVerbose": "Error migrating tables for database on storagenode: migrate: database: info opening file \"/app/config/database/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:715\n\tstorj.io/storj/private/migrate.(*Migration).Run:168\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:415\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272"}
29

The FATAL error could be a consequence of the terminating process.
The tail of the logs look like a usual upgrading sequence, except this one:

It shouldnā€™t quit with exit 1. There should be a some issue.
Could you please give me a result of the command:

docker run -it --rm debian:bookworm-slim uname -a
30

Linux 7c55cae2e5bc 6.6.63-v8+ #1821 SMP PREEMPT Mon Nov 25 13:50:21 GMT 2024 aarch64 GNU/Linux

31

So, it shouldnā€™t be a compatible issue then.
Could you please do:

docker stop storagenode
docker rm storagenode
docker rmi storjlabs/storagenode:latest
docker pull storjlabs/storagenode:latest

and then run your full docker run command.