2 month old node crashed and wont work now

Node Operators troubleshooting
1

Hi,

I was wondering if someone could give me some pointers with this node. I run 4 nodes. One of them decided to fall over and would not start. The logs don’t seem to point to anything obvious (not to me at least). Many thanks!

Here is my docker log:

2025-09-08 01:52:29,310 INFO Set uid to user 0 succeeded
2025-09-08 01:52:29,313 INFO RPC interface 'supervisor' initialized
2025-09-08 01:52:29,314 INFO supervisord started with pid 1
2025-09-08 01:52:30,316 INFO spawned: 'processes-exit-eventlistener' with pid 41
2025-09-08 01:52:30,319 INFO spawned: 'storagenode' with pid 42
2025-09-08 01:52:30,322 INFO spawned: 'storagenode-updater' with pid 43
2025-09-08T01:52:30Z    INFO    Configuration loaded    {"Process": "storagenode-updater", "Location": "/app/config/config.yaml"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "pieces.enable-prealloc"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.address"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.temporary-dir"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.path"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "console.address"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.email"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "contact.external-address"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage2.piece-scan-on-startup"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "server.private-address"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "operator.wallet-features"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.details"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "storage.allocated-disk-space"}
2025-09-08T01:52:30Z    INFO    Invalid configuration file key  {"Process": "storagenode-updater", "Key": "healthcheck.enabled"}
2025-09-08T01:52:30Z    INFO    Anonymized tracing enabled      {"Process": "storagenode-updater"}
2025-09-08T01:52:30Z    INFO    Running on version      {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.136.4"}
2025-09-08T01:52:30Z    INFO    Downloading versions.   {"Process": "storagenode-updater", "Server Address": "https://version.storj.io"}
2025-09-08 01:52:30,449 WARN exited: storagenode (exit status 1; not expected)
2025-09-08T01:52:30Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode", "Version": "v1.136.4"}
2025-09-08T01:52:30Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode"}
2025-09-08T01:52:30Z    INFO    Current binary version  {"Process": "storagenode-updater", "Service": "storagenode-updater", "Version": "v1.136.4"}
2025-09-08T01:52:30Z    INFO    Version is up to date   {"Process": "storagenode-updater", "Service": "storagenode-updater"}
2025-09-08 01:52:31,810 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 01:52:31,811 INFO spawned: 'storagenode' with pid 68
2025-09-08 01:52:31,812 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-09-08 01:52:31,899 WARN exited: storagenode (exit status 1; not expected)
2025-09-08 01:52:33,902 INFO spawned: 'storagenode' with pid 75
2025-09-08 01:52:33,995 WARN exited: storagenode (exit status 1; not expected)
2025-09-08 01:52:36,999 INFO spawned: 'storagenode' with pid 82
2025-09-08 01:52:37,084 WARN exited: storagenode (exit status 1; not expected)
2025-09-08 01:52:38,085 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2025-09-08 01:52:40,087 WARN received SIGQUIT indicating exit request
2025-09-08 01:52:40,087 INFO waiting for processes-exit-eventlistener, storagenode-updater to die
2025-09-08T01:52:40Z    INFO    Got a signal from the OS: "terminated"  {"Process": "storagenode-updater"}
2025-09-08 01:52:40,090 INFO stopped: storagenode-updater (exit status 0)
2025-09-08 01:52:41,091 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2025-09-08 01:52:42,988 INFO Set uid to user 0 succeeded
2

dumb hunch, out of RAM or out of disk space?

3

Hi, thanks for ideas. Sadly no.

Mem:           4.0Gi       249Mi       3.0Gi       216Ki       789Mi       3.8Gi
Swap:          512Mi          0B       512Mi


/dev/sdd1                          17T  6.3T   11T  38% /mnt/storj-01
4

got it working eventually by adding more binds and splitting the directories like my other nodes, ie

    --mount type=bind,source="${SSD_PATH}",destination=/app/config \
    --mount type=bind,source="${NODE_PATH}/storage",destination=/app/config/storage \
    --mount type=bind,source="${SSD_PATH}/databases",destination=/app/config/databases \
    --mount type=bind,source="${NODE_PATH}/identity",destination=/app/identity \
    --mount type=bind,source="${LOG_PATH}",destination=/app/logs \

but the actual cause remains unknown. Nothing has changed in my environment, nothing even restarted.

I suspect storagenode binary self-updated in the last few days. There must have been a change resulting in some path not being found.

The node was down for at least a day so I imagine there will be massive reputation hit… sigh… Time to implement -e STORAGENODE_UPDATER_ENABLED=“false”

5

Don’t worry about it. It’s just online time, it will return.

1 Like
6

The reason likely missing bind for logs, which you redirected (since any output from the node is missing in your initial post).