Storagenode crashing during startup

Node Operators troubleshooting
1

Hello,

Today my node started crashing during startup, it’s displaying the following log, which seems to be something related to updates:

2024-11-29T15:52:20Z ERROR Error updating service.

2024-11-29T15:52:19Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2024-11-29T15:52:19Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-11-29T15:52:19Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-11-29T15:52:19Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-29T15:52:19Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-11-29T15:52:20Z    INFO    Command output. {"Process": "storagenode-updater", "Output": ""}
2024-11-29T15:52:20Z    ERROR   Error updating service. {"Process": "storagenode-updater", "Service": "storagenode", "error": "signal: segmentation fault", "errorVerbose": "signal: segmentation fault\n\tmain.update:21\n\tmain.loopFunc:26\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:271"}
2024-11-29T15:52:20Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-29T15:52:20Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-11-29 15:52:21,173 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-29 15:52:21,176 INFO spawned: 'storagenode' with pid 61
2024-11-29 15:52:21,177 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-29 15:52:21,181 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-29 15:52:23,187 INFO spawned: 'storagenode' with pid 62
2024-11-29 15:52:23,192 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-29 15:52:26,196 INFO spawned: 'storagenode' with pid 63
2024-11-29 15:52:26,200 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-29 15:52:27,202 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-11-29 15:52:29,204 WARN received SIGQUIT indicating exit request
2024-11-29 15:52:29,205 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2024-11-29T15:52:29Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}

Let me know if you need more information about the issue,

Regards

2

Is this a Raspberry Pi node ?

3

It is a intel NUC with a i5 processor

4

How much RAM does this NUC have ?

1 Like
5

Yeah I’d grep dmesg or /var/log/syslog (or whereever system messages are for your distro) for “oom” messages.

6

RAM Memory is 16GB, almost 2GB being used, I don’t think it’s the computer, because I have another storagenode running in the same machine, but different HDD, and is doing good.

image
image921×171 6.45 KB

7

If HDD cannot keep up with IO the node will attempt to buffer the io, run out of ram, and get killed. While doing so it will evict filesystem caches, further destroying performance and dying faster.

The solution here is to not use solitary disks, but arrays with caching. Naked disks can only sustain 200iops. This is nothing, especially if you keep databases on the same disk, haven’t disabled sync, and atime updates. Using solitary disks also contradicts the core principle of this project of utilizing unused capacity. Is this disk spinning just for storj? Shut it down then.

Today load is very low, and if your setup can’t keep up now — imagine what will happen when load picks up.

8

Try the --memory docker option for this node.

1 Like
9

Please search for FATAL or Unrecoverable errors in the node’s logs (seems you have redirected them).

10

Here is the full log:

2024-11-30T12:23:42Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2024-11-30T12:23:42Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-11-30T12:23:42Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-11-30T12:23:42Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-30T12:23:42Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-11-30T12:23:42Z    INFO    Command output. {"Process": "storagenode-updater", "Output": ""}
2024-11-30T12:23:42Z    ERROR   Error updating service. {"Process": "storagenode-updater", "Service": "storagenode", "error": "signal: segmentation fault", "errorVerbose": "signal: segmentation fault\n\tmain.update:21\n\tmain.loopFunc:26\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:271"}
2024-11-30T12:23:42Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-30T12:23:42Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-11-30 12:23:43,655 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-30 12:23:43,657 INFO spawned: 'storagenode' with pid 60
2024-11-30 12:23:43,658 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-30 12:23:43,662 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:23:45,666 INFO spawned: 'storagenode' with pid 61
2024-11-30 12:23:45,670 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:23:48,675 INFO spawned: 'storagenode' with pid 62
2024-11-30 12:23:48,680 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:23:49,680 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-11-30 12:23:51,682 WARN received SIGQUIT indicating exit request
2024-11-30 12:23:51,682 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2024-11-30T12:23:51Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-11-30 12:23:51,686 INFO stopped: storagenode-updater (exit status 0)
2024-11-30 12:23:52,687 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-11-30 12:23:54,666 INFO Set uid to user 0 succeeded
2024-11-30 12:23:54,670 INFO RPC interface 'supervisor' initialized
2024-11-30 12:23:54,671 INFO supervisord started with pid 1
2024-11-30 12:23:55,674 INFO spawned: 'processes-exit-eventlistener' with pid 43
2024-11-30 12:23:55,677 INFO spawned: 'storagenode' with pid 44
2024-11-30 12:23:55,679 INFO spawned: 'storagenode-updater' with pid 45
2024-11-30 12:23:55,682 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30T12:23:55Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-11-30T12:23:55Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-11-30T12:23:55Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-11-30T12:23:55Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-30T12:23:55Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-11-30T12:23:56Z    INFO    Command output. {"Process": "storagenode-updater", "Output": ""}
2024-11-30T12:23:56Z    ERROR   Error updating service. {"Process": "storagenode-updater", "Service": "storagenode", "error": "signal: segmentation fault", "errorVerbose": "signal: segmentation fault\n\tmain.update:21\n\tmain.loopFunc:26\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:271"}
2024-11-30T12:23:56Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-30T12:23:56Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-11-30 12:23:57,089 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-30 12:23:57,091 INFO spawned: 'storagenode' with pid 61
2024-11-30 12:23:57,092 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-30 12:23:57,096 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:23:59,101 INFO spawned: 'storagenode' with pid 62
2024-11-30 12:23:59,105 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:24:02,110 INFO spawned: 'storagenode' with pid 63
2024-11-30 12:24:02,115 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:24:03,117 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-11-30 12:24:05,120 WARN received SIGQUIT indicating exit request
2024-11-30 12:24:05,120 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2024-11-30T12:24:05Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-11-30 12:24:05,124 INFO stopped: storagenode-updater (exit status 0)
2024-11-30 12:24:06,126 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-11-30 12:24:08,103 INFO Set uid to user 0 succeeded
2024-11-30 12:24:08,107 INFO RPC interface 'supervisor' initialized
2024-11-30 12:24:08,108 INFO supervisord started with pid 1
2024-11-30 12:24:09,110 INFO spawned: 'processes-exit-eventlistener' with pid 41
2024-11-30 12:24:09,112 INFO spawned: 'storagenode' with pid 42
2024-11-30 12:24:09,115 INFO spawned: 'storagenode-updater' with pid 43
2024-11-30 12:24:09,118 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30T12:24:09Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.database-dir"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-bandwidth"}
2024-11-30T12:24:09Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2024-11-30T12:24:09Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2024-11-30T12:24:09Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-30T12:24:09Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2024-11-30T12:24:09Z    INFO    Command output. {"Process": "storagenode-updater", "Output": ""}
2024-11-30T12:24:09Z    ERROR   Error updating service. {"Process": "storagenode-updater", "Service": "storagenode", "error": "signal: segmentation fault", "errorVerbose": "signal: segmentation fault\n\tmain.update:21\n\tmain.loopFunc:26\n\tstorj.io/common/sync2.(*Cycle).Run:102\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.4:392\n\tstorj.io/common/process.cleanup.func1:410\n\tgithub.com/spf13/cobra.(*Command).execute:983\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1115\n\tgithub.com/spf13/cobra.(*Command).Execute:1039\n\tstorj.io/common/process.ExecWithCustomOptions:112\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:77\n\tmain.main:22\n\truntime.main:271"}
2024-11-30T12:24:09Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.116.7"}
2024-11-30T12:24:09Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2024-11-30 12:24:10,528 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-30 12:24:10,533 INFO spawned: 'storagenode' with pid 60
2024-11-30 12:24:10,534 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-30 12:24:10,546 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:24:12,550 INFO spawned: 'storagenode' with pid 61
2024-11-30 12:24:12,555 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:24:15,559 INFO spawned: 'storagenode' with pid 62
2024-11-30 12:24:15,563 WARN exited: storagenode (terminated by SIGSEGV; not expected)
2024-11-30 12:24:16,565 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-11-30 12:24:18,568 WARN received SIGQUIT indicating exit request
2024-11-30 12:24:18,568 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2024-11-30T12:24:18Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2024-11-30 12:24:18,571 INFO stopped: storagenode-updater (exit status 0)
11

Could you please remove the bin subfolder from the data location and then try to run the node?

12

It worked, thank you @Alexey !

1 Like