Node crashed today and doesn‘t start again

Node Operators troubleshooting
,
1

Today in the morning my monitoring started throwing warnings that my StorJ Node keeps restarting. It‘s the first time after over a year of the node running.

The log:

storj-storagenode-1  | 2025-07-31 10:41:12,791 INFO RPC interface 'supervisor' initialized
storj-storagenode-1  | 2025-07-31 10:41:12,792 INFO supervisord started with pid 1
storj-storagenode-1  | 2025-07-31 10:41:13,794 INFO spawned: 'processes-exit-eventlistener' with pid 43
storj-storagenode-1  | 2025-07-31 10:41:13,796 INFO spawned: 'storagenode' with pid 44
storj-storagenode-1  | supervisor: couldn't exec /app/bin/storagenode: ENOEXEC
storj-storagenode-1  | supervisor: child process was not spawned
storj-storagenode-1  | 2025-07-31 10:41:13,818 WARN exited: storagenode (exit status 127; not expected)
storj-storagenode-1  | 2025-07-31 10:41:14,820 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
storj-storagenode-1  | 2025-07-31 10:41:14,821 INFO spawned: 'storagenode' with pid 45
storj-storagenode-1  | supervisor: couldn't exec /app/bin/storagenode: ENOEXEC
storj-storagenode-1  | supervisor: child process was not spawned
storj-storagenode-1  | 2025-07-31 10:41:14,838 WARN exited: storagenode (exit status 127; not expected)
storj-storagenode-1  | 2025-07-31 10:41:16,841 INFO spawned: 'storagenode' with pid 46
storj-storagenode-1  | supervisor: couldn't exec /app/bin/storagenode: ENOEXEC
storj-storagenode-1  | supervisor: child process was not spawned
storj-storagenode-1  | 2025-07-31 10:41:16,858 WARN exited: storagenode (exit status 127; not expected)
storj-storagenode-1  | 2025-07-31 10:41:19,863 INFO spawned: 'storagenode' with pid 47
storj-storagenode-1  | supervisor: couldn't exec /app/bin/storagenode: ENOEXEC
storj-storagenode-1  | supervisor: child process was not spawned
storj-storagenode-1  | 2025-07-31 10:41:19,878 WARN exited: storagenode (exit status 127; not expected)
storj-storagenode-1  | 2025-07-31 10:41:20,880 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
storj-storagenode-1  | 2025-07-31 10:41:22,883 WARN received SIGQUIT indicating exit request
storj-storagenode-1  | 2025-07-31 10:41:22,883 INFO waiting for processes-exit-eventlistener to die
storj-storagenode-1  | 2025-07-31 10:41:24,885 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
storj-storagenode-1 exited with code 0

I already tried deleting the binaries and the whole bin folder. It re-downloads everything and crashes again like above. My node ran great and I haven‘t touched anything, so I‘m totally out of ideas. Suspension time is ticking.

Edit: It‘s running as a Docker container on a Synology Diskstation. x86_64 platform.

Edit2:
I checked the *.db files with sqlite3. The ones in the /config/storage folder are all okay. The revocations.db seems to be no sqlite3 database.
After all those restarts the error changed:


storj-storagenode-1  | 2025-07-31 12:28:06,479 INFO RPC interface 'supervisor' initialized
storj-storagenode-1  | 2025-07-31 12:28:06,480 INFO supervisord started with pid 1
storj-storagenode-1  | 2025-07-31 12:28:07,482 INFO spawned: 'processes-exit-eventlistener' with pid 46
storj-storagenode-1  | 2025-07-31 12:28:07,485 INFO spawned: 'storagenode' with pid 47
storj-storagenode-1  | 2025-07-31 12:28:07,506 WARN exited: storagenode (terminated by SIGSEGV; not expected)
storj-storagenode-1  | 2025-07-31 12:28:08,508 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
storj-storagenode-1  | 2025-07-31 12:28:08,509 INFO spawned: 'storagenode' with pid 48
storj-storagenode-1  | 2025-07-31 12:28:08,524 WARN exited: storagenode (terminated by SIGSEGV; not expected)
storj-storagenode-1  | 2025-07-31 12:28:10,528 INFO spawned: 'storagenode' with pid 49
storj-storagenode-1  | 2025-07-31 12:28:10,543 WARN exited: storagenode (terminated by SIGSEGV; not expected)
storj-storagenode-1  | 2025-07-31 12:28:13,547 INFO spawned: 'storagenode' with pid 50
storj-storagenode-1  | 2025-07-31 12:28:13,561 WARN exited: storagenode (terminated by SIGSEGV; not expected)
storj-storagenode-1  | 2025-07-31 12:28:14,563 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
storj-storagenode-1  | 2025-07-31 12:28:16,565 WARN received SIGQUIT indicating exit request
storj-storagenode-1  | 2025-07-31 12:28:16,565 INFO waiting for processes-exit-eventlistener to die
storj-storagenode-1  | 2025-07-31 12:28:18,567 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
storj-storagenode-1 exited with code 0

From what I found online I tried renaming the revocations.db and the trust-cache.json so the node recreates them. Still no change.

2

If the data directory of the node is mounted with noexec, try mounting it without that parameter.

3

I‘m using normal bind mounts without any additional parameters.

4

I found the issue. Dumb past me set a quota on the storage that was reached. I removed it and now it works again.
Thanks for the help!

5

This should not be possible. Node always leaves a buffer of a few GB. Or did you set a quota after the fact, lower than actually used space? And DSM let you do that?!

6

@SplashmasterBee Thanks for the good read, I was about to add a quota to mine and now I will hold off and see if the size I told the Storj node to use, if it stops there.