ERROR Error updating service

Node Operators troubleshooting
, , ,
21

i have no idea how did my node been upgraded to that version. im just basing it to the logs im seeing

2024-10-11T12:57:39Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.113.2"}
2024-10-11T12:57:39Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.114.6/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.3586801199.zip"}
2024-10-11T12:57:39Z	ERROR	Error updating service.	{"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "Get \"https://github.com/storj/storj/releases/download/v1.114.6/storagenode-updater_linux_arm64.zip\": context canceled", "errorVerbose": "Get \"https://github.com/storj/storj/releases/download/v1.114.6/storagenode-updater_linux_arm64.zip\": context canceled\n\tmain.downloadBinary:58\n\tmain.update:40\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tmain.cmdRun:138\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:271"}
2024-10-11 12:57:40,351 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-10-11 12:57:40,352 INFO stopped: storagenode-updater (exit status 0)
2024-10-11 12:57:41,489 WARN stopped: storagenode (exit status 1)
2024-10-11 12:57:41,492 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
22

thank you for this my node is back online. i need to download these two: storagenode and storagenode-updater, and override the existing files in the bin folder.

idk why my node is running on v1.114.6

image
image1259×139 34.7 KB

23

Seems because it cannot download it for some reason. But the previous log lines suggests that your node should be upgraded to 1.114.6 accordingly its NodeID.

24

Hi, my node keeps restarting in a loop every time a new version is released. I have to manually download the storagenode and storagenode-updater to get it working again. This has happened with at least three versions, or possibly more. I’m wondering if there’s a way to fix this on my side or node, so I don’t have to do this manually every time.

25

That is odd. Can you provide a log of the errors causing the reset?

26

Could you please remove the bin subfolder from the data location and re-create the container?

27

I tried this, but it didn’t resolve the issue.

28

from container

2025-01-10 10:41:56,937 INFO RPC interface 'supervisor' initialized
2025-01-10 10:41:56,938 INFO supervisord started with pid 1
2025-01-10 10:41:57,946 INFO spawned: 'processes-exit-eventlistener' with pid 36
2025-01-10 10:41:57,958 INFO spawned: 'storagenode' with pid 37
2025-01-10 10:41:57,963 INFO spawned: 'storagenode-updater' with pid 38
2025-01-10T10:41:58Z	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.details"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.path"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2025-01-10T10:41:58Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features.0"}
2025-01-10T10:41:58Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2025-01-10T10:41:58Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.117.8"}
2025-01-10T10:41:58Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-01-10T10:41:58Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.117.8"}
2025-01-10T10:41:58Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.2840522335.zip"}
2025-01-10 10:41:59,815 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-10 10:41:59,816 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-10 10:41:59,816 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-10T10:42:04Z	INFO	Download finished.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.2840522335.zip"}
2025-01-10T10:42:05Z	INFO	Restarting service.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-01-10T10:42:05Z	INFO	Service restarted successfully.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-01-10T10:42:05Z	INFO	Cleaning up old binary.	{"Process": "storagenode-updater", "Service": "storagenode", "Path": "/app/bin/storagenode.old.v1.117.8"}
2025-01-10T10:42:05Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.117.8"}
2025-01-10T10:42:05Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.171671116.zip"}
2025-01-10T10:42:15Z	INFO	Download finished.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.119.15/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.171671116.zip"}
2025-01-10T10:42:15Z	INFO	Restarting service.	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-01-10T10:42:15Z	INFO	Cleaning up old binary.	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Path": "/app/bin/storagenode-updater.old"}
2025-01-10 10:42:16,744 WARN exited: storagenode-updater (exit status 1; not expected)
2025-01-10 10:42:17,754 INFO spawned: 'storagenode-updater' with pid 169
2025-01-10 10:42:27,039 WARN received SIGQUIT indicating exit request
2025-01-10 10:42:27,042 INFO waiting for processes-exit-eventlistener, storagenode, storagenode-updater to die
2025-01-10 10:42:27,050 WARN stopped: storagenode-updater (terminated by SIGTERM)
2025-01-10 10:42:30,055 INFO waiting for processes-exit-eventlistener, storagenode to die

from storagenode

2025-01-10T10:44:59Z	FATAL	Unrecoverable error	{"Process": "storagenode", "error": "Error migrating tables for database on storagenode: migrate: database: info opening file \"/app/config/database/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:715\n\tstorj.io/storj/private/migrate.(*Migration).Run:168\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:415\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272", "errorVerbose": "Error migrating tables for database on storagenode: migrate: database: info opening file \"/app/config/database/info.db\" failed: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabaseWithStat:390\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).openDatabase:367\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Migration.func1:715\n\tstorj.io/storj/private/migrate.(*Migration).Run:168\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).MigrateToLatest:415\n\tmain.cmdRun:100\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:102\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272"}
29

The FATAL error could be a consequence of the terminating process.
The tail of the logs look like a usual upgrading sequence, except this one:

It shouldn’t quit with exit 1. There should be a some issue.
Could you please give me a result of the command:

docker run -it --rm debian:bookworm-slim uname -a
30

Linux 7c55cae2e5bc 6.6.63-v8+ #1821 SMP PREEMPT Mon Nov 25 13:50:21 GMT 2024 aarch64 GNU/Linux

31

So, it shouldn’t be a compatible issue then.
Could you please do:

docker stop storagenode
docker rm storagenode
docker rmi storjlabs/storagenode:latest
docker pull storjlabs/storagenode:latest

and then run your full docker run command.

32

i tried this but still having the same issue.
this is the logs now.

2025-02-05T00:27:39Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.120.4"}
2025-02-05T00:27:39Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-02-05T00:42:39Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-02-05T00:42:39Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.120.4"}
2025-02-05T00:42:39Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-02-05T00:42:39Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.120.4"}
2025-02-05T00:42:39Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-02-05T00:57:39Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-02-05T00:57:40Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.120.4"}
2025-02-05T00:57:40Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-02-05T00:57:40Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.120.4"}
2025-02-05T00:57:40Z	INFO	New version is being rolled out but hasn't made it to this node yet	{"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-02-05T01:12:39Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-02-05T01:12:39Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.120.4"}
2025-02-05T01:12:39Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.121.2/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.1503519217.zip"}
2025-02-05T01:12:47Z	INFO	Download finished.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.121.2/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.1503519217.zip"}
2025-02-05T01:12:47Z	INFO	Restarting service.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-02-05T01:12:49Z	INFO	Service restarted successfully.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-02-05T01:12:49Z	INFO	Cleaning up old binary.	{"Process": "storagenode-updater", "Service": "storagenode", "Path": "/app/bin/storagenode.old.v1.120.4"}
2025-02-05T01:12:49Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.120.4"}
2025-02-05T01:12:49Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.121.2/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.2972573594.zip"}
2025-02-05 01:12:50,157 INFO exited: storagenode (exit status 0; expected)
2025-02-05 01:12:51,175 INFO spawned: 'storagenode' with pid 16256
2025-02-05 01:12:52,180 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-05 01:12:52,181 WARN received SIGQUIT indicating exit request
2025-02-05 01:12:52,182 INFO waiting for processes-exit-eventlistener, storagenode, storagenode-updater to die
2025-02-05T01:12:52Z	INFO	Got a signal from the OS: "terminated"	{"Process": "storagenode-updater"}
2025-02-05T01:12:52Z	ERROR	Error updating service.	{"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "context canceled", "errorVerbose": "context canceled\n\tmain.downloadBinary:62\n\tmain.update:42\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:272"}
2025-02-05 01:12:52,202 INFO stopped: storagenode-updater (exit status 0)
2025-02-05 01:12:52,270 WARN stopped: storagenode (exit status 1)
2025-02-05 01:12:52,271 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2025-02-05 01:12:59,059 INFO RPC interface 'supervisor' initialized
2025-02-05 01:12:59,060 INFO supervisord started with pid 1
2025-02-05 01:13:00,069 INFO spawned: 'processes-exit-eventlistener' with pid 34
2025-02-05 01:13:00,078 INFO spawned: 'storagenode' with pid 35
2025-02-05 01:13:00,084 INFO spawned: 'storagenode-updater' with pid 36
2025-02-05T01:13:00Z	INFO	Configuration loaded	{"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "contact.external-address"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.details"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.address"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.email"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage.path"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "operator.wallet-features.0"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "server.private-address"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "console.address"}
2025-02-05T01:13:00Z	INFO	Invalid configuration file key	{"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2025-02-05T01:13:00Z	INFO	Anonymized tracing enabled	{"Process": "storagenode-updater"}
2025-02-05T01:13:00Z	INFO	Running on version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.120.4"}
2025-02-05T01:13:00Z	INFO	Downloading versions.	{"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-02-05T01:13:01Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.120.4"}
2025-02-05 01:13:01,139 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-05 01:13:01,140 INFO success: storagenode entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-05 01:13:01,141 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-02-05T01:13:01Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.121.2/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.3084383543.zip"}
2025-02-05T01:13:09Z	INFO	Download finished.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.121.2/storagenode_linux_arm64.zip", "To": "/tmp/storagenode_linux_arm64.3084383543.zip"}
2025-02-05T01:13:09Z	INFO	Restarting service.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-02-05T01:13:10Z	INFO	Service restarted successfully.	{"Process": "storagenode-updater", "Service": "storagenode"}
2025-02-05T01:13:10Z	INFO	Cleaning up old binary.	{"Process": "storagenode-updater", "Service": "storagenode", "Path": "/app/bin/storagenode.old.v1.120.4"}
2025-02-05T01:13:10Z	INFO	Current binary version	{"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.120.4"}
2025-02-05T01:13:10Z	INFO	Download started.	{"Process": "storagenode-updater", "From": "https://github.com/storj/storj/releases/download/v1.121.2/storagenode-updater_linux_arm64.zip", "To": "/tmp/storagenode-updater_linux_arm64.1587975237.zip"}
2025-02-05 01:13:12,652 INFO exited: storagenode (exit status 0; expected)
2025-02-05 01:13:13,658 INFO spawned: 'storagenode' with pid 101
2025-02-05 01:13:13,659 WARN received SIGQUIT indicating exit request
2025-02-05 01:13:13,660 INFO waiting for processes-exit-eventlistener, storagenode, storagenode-updater to die
2025-02-05T01:13:13Z	INFO	Got a signal from the OS: "terminated"	{"Process": "storagenode-updater"}
2025-02-05T01:13:13Z	ERROR	Error updating service.	{"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "context canceled", "errorVerbose": "context canceled\n\tmain.downloadBinary:65\n\tmain.update:42\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:272"}
2025-02-05 01:13:13,698 INFO stopped: storagenode-updater (exit status 0)
2025-02-05 01:13:13,703 WARN stopped: storagenode (terminated by SIGTERM)
2025-02-05 01:13:13,705 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2025-02-05 01:13:38,200 INFO RPC interface 'supervisor' initialized
2025-02-05 01:13:38,201 INFO supervisord started with pid 1
2025-02-05 01:13:39,208 INFO spawned: 'processes-exit-eventlistener' with pid 33
2025-02-05 01:13:39,216 INFO spawned: 'storagenode' with pid 34
2025-02-05 01:13:39,223 INFO spawned: 'storagenode-updater' with pid 35


2025-02-05T02:16:57Z	ERROR	failure during run	{"Process": "storagenode", "error": "Error during preflight check for storagenode databases: preflight: database: \"used_serial\": failed inserting test value: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:496\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:421\n\tmain.cmdRun:115\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272", "errorVerbose": "Error during preflight check for storagenode databases: preflight: database: \"used_serial\": failed inserting test value: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:496\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:421\n\tmain.cmdRun:115\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:117\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272"}
2025-02-05T02:16:57Z	FATAL	Unrecoverable error	{"Process": "storagenode", "error": "Error during preflight check for storagenode databases: preflight: database: \"used_serial\": failed inserting test value: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:496\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:421\n\tmain.cmdRun:115\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272", "errorVerbose": "Error during preflight check for storagenode databases: preflight: database: \"used_serial\": failed inserting test value: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).preflight:496\n\tstorj.io/storj/storagenode/storagenodedb.(*DB).Preflight:421\n\tmain.cmdRun:115\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272\n\tmain.cmdRun:117\n\tmain.newRunCmd.func1:33\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tmain.main:34\n\truntime.main:272"}
33

It looks like a slow drive. What’s it? How is it connected?

34

I have an SSD for databases and two hard disks for storage, all housed in an enclosure and connected via USB 3. The network is via Ethernet.

35

Did you try to stop and remove the container and run it back?
Also, could you please check the filesystem on SSD and databases.

36

I would strongly suggest avoid usb in your storage stack.

1 Like
37

Sometimes it’s not avoidable, if they would use some SOC like Raspberry PI. In that case they need a quality expansion board USB-SATA, or at least a not cheap power USB hub and normal cables, also quality external independent power supply. But the expansion board is preferred.