Spawnerr: no permission to run command

[Toyoo]

@Alexey

It doesn’t have issues on a default setup

The requirements so far did not include ensuring that the storage directory is not mounted with noexec. Frankly, I would see it as a nice security feature to make sure that the node does not accidentally receive a malicious binary to store. I myself did that long time ago already.

@Scriptiee

Are you taking me for a fool here? You were aware of this since at least 30th of July, not 2 days ago via announcement … jeez man, dismissive snob does not look good on you…

While I applaud anyone working towards more consistent communication on node operation changes, please change the tone.

I know Alexey implies here that unless you spend an hour per day going through almost all threads on the forum, and in other thread he did imply you need to carefully study almost all commits made to the code, calling people names will not bring change.

[Alexey]

@Toyoo

The requirements so far did not include ensuring that the storage directory is not mounted with noexec. Frankly, I would see it as a nice security feature to make sure that the node does not accidentally receive a malicious binary to store. I myself did that long time ago already.

Thank you,

added exec permission requirement for static mount

Seems the most secure option would be either use a BINARY_DIR variable, or mount a docker volume to that path if you use docker.

[Alexey]

@Scriptiee

You were aware of this since at least 30th of July, not 2 days ago via announcement

Yes, I submitted this issue and may have received a notification from GitHub. I don’t always have time to test something before it’s ready.

So I tested the solution only when I received an announcement from the forum. I’m always trying to be a SNO to be aware of a potential issues. Unfortunately I do not have my RPI anymore, so test was only for remaining docker nodes in the Docker Desktop for Windows.
I also tested it in the Ubuntu VM for both cases - when the drive is mounted with defaults and for the docker volume, no issues have been found. Here I used an ARM64 emulation, because usually ARM64 may introduce some issues. But the node was not “live”, it did not have an external address.

[Pentium100]

After updating the docker image, my node does not start. How do I revert the image to the previous version?

2024-09-25 10:59:37 (50.5 MB/s) - '/tmp/storagenode.zip' saved [34209717/34209717]

2024-09-25 10:59:38,645 INFO Set uid to user 0 succeeded
2024-09-25 10:59:38,650 INFO RPC interface 'supervisor' initialized
2024-09-25 10:59:38,650 INFO supervisord started with pid 1
2024-09-25 10:59:39,654 INFO spawned: 'processes-exit-eventlistener' with pid 23
2024-09-25 10:59:39,655 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:39,656 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:40,670 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 10:59:40,671 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:40,672 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:42,675 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:42,676 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:45,681 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:45,681 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 10:59:45,681 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:45,682 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 10:59:47,685 WARN received SIGQUIT indicating exit request
2024-09-25 10:59:47,686 INFO waiting for processes-exit-eventlistener to die
2024-09-25 10:59:49,690 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-25 10:59:51,186 INFO Set uid to user 0 succeeded
2024-09-25 10:59:51,190 INFO RPC interface 'supervisor' initialized
2024-09-25 10:59:51,190 INFO supervisord started with pid 1
2024-09-25 10:59:52,194 INFO spawned: 'processes-exit-eventlistener' with pid 16
2024-09-25 10:59:52,196 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:52,197 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:53,207 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 10:59:53,208 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:53,209 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:55,213 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:55,213 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:58,218 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 10:59:58,218 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 10:59:58,219 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 10:59:58,219 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 11:00:00,222 WARN received SIGQUIT indicating exit request
2024-09-25 11:00:00,223 INFO waiting for processes-exit-eventlistener to die
2024-09-25 11:00:02,227 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-25 11:00:05,827 INFO Set uid to user 0 succeeded
2024-09-25 11:00:05,830 INFO RPC interface 'supervisor' initialized
2024-09-25 11:00:05,831 INFO supervisord started with pid 1
2024-09-25 11:00:06,834 INFO spawned: 'processes-exit-eventlistener' with pid 9
2024-09-25 11:00:06,837 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:06,837 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:07,846 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 11:00:07,847 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:07,848 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:09,851 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:09,851 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:12,856 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:12,856 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 11:00:12,857 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:12,857 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 11:00:14,861 WARN received SIGQUIT indicating exit request
2024-09-25 11:00:14,862 INFO waiting for processes-exit-eventlistener to die
2024-09-25 11:00:16,866 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-25 11:00:18,274 INFO Set uid to user 0 succeeded
2024-09-25 11:00:18,277 INFO RPC interface 'supervisor' initialized
2024-09-25 11:00:18,277 INFO supervisord started with pid 1
2024-09-25 11:00:19,281 INFO spawned: 'processes-exit-eventlistener' with pid 16
2024-09-25 11:00:19,283 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:19,284 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:20,293 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 11:00:20,294 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:20,295 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:22,300 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:22,300 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:25,305 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:25,306 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 11:00:25,306 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:25,306 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 11:00:27,311 WARN received SIGQUIT indicating exit request
2024-09-25 11:00:27,311 INFO waiting for processes-exit-eventlistener to die
2024-09-25 11:00:29,315 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-25 11:00:30,561 INFO Set uid to user 0 succeeded
2024-09-25 11:00:30,567 INFO RPC interface 'supervisor' initialized
2024-09-25 11:00:30,567 INFO supervisord started with pid 1
2024-09-25 11:00:31,571 INFO spawned: 'processes-exit-eventlistener' with pid 9
2024-09-25 11:00:31,574 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:31,575 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:32,583 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 11:00:32,584 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:32,584 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:34,588 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:34,589 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:37,594 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:37,594 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 11:00:37,595 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:37,595 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 11:00:39,598 WARN received SIGQUIT indicating exit request
2024-09-25 11:00:39,599 INFO waiting for processes-exit-eventlistener to die
2024-09-25 11:00:41,602 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-25 11:00:42,795 INFO Set uid to user 0 succeeded
2024-09-25 11:00:42,798 INFO RPC interface 'supervisor' initialized
2024-09-25 11:00:42,799 INFO supervisord started with pid 1
2024-09-25 11:00:43,802 INFO spawned: 'processes-exit-eventlistener' with pid 9
2024-09-25 11:00:43,804 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:43,805 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:44,810 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 11:00:44,811 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:44,811 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:46,814 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:46,814 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:49,819 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:49,820 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 11:00:49,820 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:49,820 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 11:00:51,824 WARN received SIGQUIT indicating exit request
2024-09-25 11:00:51,824 INFO waiting for processes-exit-eventlistener to die
2024-09-25 11:00:53,829 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-25 11:00:55,096 INFO Set uid to user 0 succeeded
2024-09-25 11:00:55,099 INFO RPC interface 'supervisor' initialized
2024-09-25 11:00:55,099 INFO supervisord started with pid 1
2024-09-25 11:00:56,103 INFO spawned: 'processes-exit-eventlistener' with pid 9
2024-09-25 11:00:56,106 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:56,106 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:57,116 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 11:00:57,117 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:57,118 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:00:59,121 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:00:59,122 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:01:02,127 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 11:01:02,127 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 11:01:02,127 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 11:01:02,127 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 11:01:04,130 WARN received SIGQUIT indicating exit request
2024-09-25 11:01:04,131 INFO waiting for processes-exit-eventlistener to die
2024-09-25 11:01:06,134 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)

[Roxor]

The “no permission to run” thing reminds me of this. There are also a few other “spawnerr” posts recently…

[nerdatwork]

Spawnerr: no permission to run command

Hi, My storagenode is offline with permissions errors and attempts to recover the node by recreating the docker container with root permissions have not been successful. The errors are: INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode’ INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode-updater’ INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode’ INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode-update…

[Pentium100]

So yeah, now the node data filesystem needs to be mounted with exec.

[Xade]

All of my 12 nodes went down on these errors after an update…this is a pure joke…

2024-09-25 20:45:07,165 INFO waiting for processes-exit-eventlistener to die
2024-09-25 20:45:09,171 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-25 20:45:11,413 INFO Set uid to user 0 succeeded
2024-09-25 20:45:11,422 INFO RPC interface 'supervisor' initialized
2024-09-25 20:45:11,423 INFO supervisord started with pid 1
2024-09-25 20:45:12,428 INFO spawned: 'processes-exit-eventlistener' with pid 9
2024-09-25 20:45:12,430 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 20:45:12,431 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 20:45:13,443 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-25 20:45:13,444 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 20:45:13,445 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 20:45:15,450 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 20:45:15,452 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 20:45:18,459 INFO spawnerr: no permission to run command '/app/config/bin/storagenode'
2024-09-25 20:45:18,459 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-25 20:45:18,461 INFO spawnerr: no permission to run command '/app/config/bin/storagenode-updater'
2024-09-25 20:45:18,461 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-25 20:45:20,466 WARN received SIGQUIT indicating exit request
2024-09-25 20:45:20,467 INFO waiting for processes-exit-eventlistener to die
2024-09-25 20:45:22,471 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)

[EasyRhino]

could be this problem about the node binaries being stored on your main storage folder and needing execute permission now in whatever directory maps to /app/config or /app/config/bin

Storagenode Docker Update: Changes To Binaries Location For Persistence Between Restarts - Node Operators - Storj Community Forum (official)

[Pentium100]

Yeah, I got hit by this as well

Node does not start after docker image update

After updating the docker image, my node does not start. How do I revert the image to the previous version? 2024-09-25 10:59:37 (50.5 MB/s) - '/tmp/storagenode.zip' saved [34209717/34209717] 2024-09-25 10:59:38,645 INFO Set uid to user 0 succeeded 2024-09-25 10:59:38,650 INFO RPC interface 'supervisor' initialized 2024-09-25 10:59:38,650 INFO supervisord started with pid 1 2024-09-25 10:59:39,654 INFO spawned: 'processes-exit-eventlistener' with pid 23 2024-09-25 10:59:39,655 INFO spawnerr: no…

[snorkel]

But… the users don’t have to do nothing, after pushing this update in a short notice.
Why so many nodes went down?

[Pentium100]

Some people (like me) had mounted their data partitions/drives as noexec. This was for added security as I did not expect to have any executables there. Storj changed the docker image and now puts the node executables in the data drive, so now the “noexec” parameter has to be removed.
It would have been even more fun if the node updated automatically while I was away.

[Alexey]

Not so many, accordingly the statistic so far, only QNAP, WD cloud, Synology so far, but not all, too.
Some pis too, but it’s a different issue, related to the docker update (I think), because now they need to be run with a --privileged option, see

Storagenode stopped working on Pi4 - ERROR: The certificate of 'version.storj.io' is not trusted

Ah, I does work, I had two problems: I can’t spell - I’d added an extra d to --privileged I’d put it last, putting it before the --mount and --name flags made it work I’m back up and running, thanks for your help

[Alexey]

We prepared a new image, could you please check before it would become a latest?

Storagenode Docker Update: Changes To Binaries Location For Persistence Between Restarts

I just pushed new images containing this patch. Not marked as latest yet: https://hub.docker.com/layers/storjlabs/storagenode/9109fd2/images/sha256-17c83b5a16a2938364e14e1a0cc62fabcd4a468bc3bab19fe31abe3c361e2a1b?context=repo