Storagenode repeated storage/storagenode-update is not executable

mi5key · September 27, 2024, 7:13pm

I’m still locked in this death spiral of ‘not executable’. I’ve stopped the container, rm’d it, pulled latest, but still the same thing. Directories have r/x, bins have r/x. Uid of dockerlimited is 1028 and is set in the docker run command. I went through the other topic but mine is still in crisis.

drwxrwxrwx+ 7 dockerlimited users 4096 Sep 27 02:19 storage

drwxrwxrwx+ 2 dockerlimited users 4096 Sep 27 03:18 bin

-rwxrwxrwx+ 1 dockerlimited users 78855672 Sep 27 03:18 storagenode
-rwxrwxrwx+ 1 dockerlimited users 32505920 Sep 27 03:18 storagenode-updater

2024-09-27 11:18:49,890 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-27 11:18:49,890 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-27 11:18:50,905 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-27 11:18:50,906 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-27 11:18:50,906 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-27 11:18:52,909 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-27 11:18:52,909 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-27 11:18:55,912 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-27 11:18:55,912 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-27 11:18:55,912 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-27 11:18:55,912 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-27 11:18:57,915 WARN received SIGQUIT indicating exit request
2024-09-27 11:18:57,915 INFO waiting for processes-exit-eventlistener to die
2024-09-27 11:18:59,918 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)

jbew81 · September 27, 2024, 7:26pm

Same problem for me, i can’t understand where the problem is.

mi5key · September 27, 2024, 8:26pm

Now that I typed all that out, it’s working now. Concerned about problems deeper down, I a ran a "sudo chown -R dockerlimited:users * on my storj top dir.

donald.m.motsinger · September 27, 2024, 9:02pm

How is the partition mounted? With “noexec”?

Alexey · September 28, 2024, 6:26am

Hello @jbew81,
Welcome to the forum!

Please, apply x permission to the bin subfolder of your storage location from the source value of the --mount option for /app/config/ in your docker run command, e.g.

sudo chmod +x /mnt/y/storagenode2/bin

jbew81 · September 28, 2024, 7:04am

Thks Alexey,

Before :
root@NAS:~#ls -l /volume1/my-storj-data/node1/bin/
total 108756
----------+ 1 root root 78855672 Sep 27 01:31 storagenode
----------+ 1 root root 32505920 Sep 27 01:31 storagenode-updater
Change :
root@NAS:~#chmod +x /volume1/my-storj-data/node1/bin/*

After:
root@NAS:~#ls -l /volume1/my-storj-data/node1/bin/
total 108756
—x------ 1 root root 78855672 Sep 27 01:31 storagenode
—x------ 1 root root 32505920 Sep 27 01:31 storagenode-updater
Everything is fine now.
Thank you.

brolin76 · September 28, 2024, 7:11am

Thanks.
This works.

Just to help people who are using docker with generic installation information.
My path is

/volume1/Storj/storage/bin

So the command as provided by Alexey is (for my path)

sudo chmod +x /volume1/Storj/storage/bin

Working fine now.

therealmko · September 29, 2024, 7:01am

I had the same issue and it is fixed changing the permissions as indicated above. Still how is this possible all of a sudden for multiple people apparently? Anything to change in the setup to prevent this from happening?

Alexey · September 29, 2024, 8:14am

Hello @therealmko,
Welcome back!

Perhaps, more testing on NASes. We do not have them, and the usual setups are ok.

However, I created a PR, which should help to solve this problem:

github.com/storj/storagenode-docker

fixed exec issue for restricted systems

storj:main ← AlexeyALeonov:bugfix-binary-dir-for-restricted-systems

opened 03:24AM - 26 Sep 24 UTC

AlexeyALeonov

+37 -27

Related to #23 How to test: * build the docker image: ``` docker buildx …build -f Dockerfile -t node:latest . ``` * Create an image file and mount it with `noexec` to emulate a restricted mount: ``` truncate -s 100M temp.img mkfs -t ext4 temp.img mkdir temp-disk sudo mount -o rw,noexec temp.img temp-disk sudo chown $(id -i):$(id -g) temp-disk ``` ## testing a normal flow 1. Run the container, and wait till the binaries are downloaded: ``` docker run --rm --name test --mount type=bind,source="${PWD}/test-disk",destination=/app/config node:latest ``` 2. Check that there were no permission errors 3. Check that the `test-disk` contains both binaries in "test-disk/bin" 4. Rerun the container; binaries should not be downloaded again, and the container should be running without permission errors. ## storagenode has been updated in the ephemeral storage, but not in a persistent storage 1. download an old storagenode binary ``` wget -O /tmp/storagenode.zip https://github.com/storj/storj/releases/download/v1.111.4/storagenode_linux_amd64.zip mkdir -p test-disk/bin unzip -p /tmp/storagenode.zip > test-disk/bin/storagenode chmod u+x test-disk/bin/storagenode ``` 2. run the container ``` docker run --rm --name test --mount type=bind,source="${PWD}/test-disk",destination=/app/config node:latest ``` 3. check that a new version of storagenode has been downloaded and replaced both in the persistent and ephemeral storages 4. check that there were no permissions errors 5. Check that the `test-disk` contains both binaries in "test-disk/bin" 6. Rerun the container; binaries should not be downloaded again, and the container should be running without permission errors. 7. Make sure, that the old version is updated to a minimum version, if the update to the suggested version is not allowed yet. ## cleanup ``` sudo umount test-disk rm -rf test-disk test.img ```

ryan · September 30, 2024, 12:40pm

I just had this happen myself when I was out of town for 3 days. Now one of my NAS is suspended on a satellite. I did get it fixed by deleting the image, redownloading and then reinputting the commands to run. But this is discouraging as my longest running node is now suspended. It’s back up and running now and will heal itself but out of 3 Synology NAS devices local, this one had the problem.

Fabianoaspro · September 30, 2024, 6:01pm

Hello!
I’m new here. I hosted a (test) node off 500Gb and everything is OK. BUT when I edit the config.yaml file to change the line “operator.wallet-features: [‘zksync-era’]” when I restart the Storj hangs with this log:

2024-09-30 14:59:32,351 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 14:59:32,352 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-30 14:59:35,355 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 14:59:35,355 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-30 14:59:35,356 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-30 14:59:35,356 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-30 14:59:37,358 WARN received SIGQUIT indicating exit request
2024-09-30 14:59:37,359 INFO waiting for processes-exit-eventlistener to die
2024-09-30 14:59:39,361 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-30 14:59:48,732 INFO Set uid to user 0 succeeded
2024-09-30 14:59:48,735 INFO RPC interface ‘supervisor’ initialized
2024-09-30 14:59:48,735 INFO supervisord started with pid 1
2024-09-30 14:59:49,738 INFO spawned: ‘processes-exit-eventlistener’ with pid 9
2024-09-30 14:59:49,743 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 14:59:49,744 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-30 14:59:50,745 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-30 14:59:50,746 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 14:59:50,747 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-30 14:59:52,749 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 14:59:52,750 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-30 14:59:55,753 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 14:59:55,753 INFO gave up: storagenode entered FATAL state, too many start retries too quickly
2024-09-30 14:59:55,754 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-30 14:59:55,754 INFO gave up: storagenode-updater entered FATAL state, too many start retries too quickly
2024-09-30 14:59:57,755 WARN received SIGQUIT indicating exit request
2024-09-30 14:59:57,756 INFO waiting for processes-exit-eventlistener to die
2024-09-30 14:59:59,758 WARN stopped: processes-exit-eventlistener (terminated by SIGTERM)
2024-09-30 15:00:08,086 INFO Set uid to user 0 succeeded
2024-09-30 15:00:08,109 INFO RPC interface ‘supervisor’ initialized
2024-09-30 15:00:08,110 INFO supervisord started with pid 1
2024-09-30 15:00:09,111 INFO spawned: ‘processes-exit-eventlistener’ with pid 9
2024-09-30 15:00:09,112 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 15:00:09,122 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable
2024-09-30 15:00:10,123 INFO success: processes-exit-eventlistener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-09-30 15:00:10,124 INFO spawnerr: command at ‘/app/config/bin/storagenode’ is not executable
2024-09-30 15:00:10,125 INFO spawnerr: command at ‘/app/config/bin/storagenode-updater’ is not executable

Even when I put back the original config.yaml and restarting the service does not work.
Con someone help me?

arrogantrabbit · September 30, 2024, 6:02pm

Sleinous · October 1, 2024, 7:23pm

Not working for me, node is now done for over 2 hours due to this issue popping up on one of my nodes.
Assuming i’ve messed up the syntax of adding the sudo chown +x command into my docker run command but not sure where i’ve gone wrong

arrogantrabbit · October 1, 2024, 7:49pm

chmod. mod, not own.

If you are confused about paths, open shell in your container:

docker exec -it <storagenode contaienr> /bin/sh

and run this there:

chmod -R +x /app/config/bin

Sleinous · October 1, 2024, 8:11pm

yes sorry, chmod +x, didn’t seem to do anything when I just ran it straight from command but after about 10 minutes the node is finally back up again without that error message.
From what I read im meant to be able to incorporate this into --mount of docker run, but docker fails to provide an example of syntax

arrogantrabbit · October 1, 2024, 8:25pm

Yes, it’s just a mount option, you specify it along with a bunch of other mount options

For example,

docker run --mount type=bind,source=/meow,target=/wuff,exec,noatime

Sleinous · October 1, 2024, 9:29pm

ah thanks for that, so I can just add the chmod +x command there it there, cool

arrogantrabbit · October 1, 2024, 9:54pm

No, you can’t. You can’t cram command into bind options. You can specify a mount option that allows exec. mount(8) - Linux manual page

Alexey · October 2, 2024, 5:59am

docker cannot do so unfortunately.

invalid argument "type=bind,source=./test-disk,destination=/app/config,exec,noatime" for "--mount" flag: invalid field 'exec' must be a key=value pair
See 'docker run --help'.

arrogantrabbit · October 2, 2024, 6:26am

Hmm. Their own documentation have similar flags, not passed as key-value pairs. e.g. here, see readonly flag:

https://docs.docker.com/engine/storage/bind-mounts/#use-a-read-only-bind-mount

docker run -d \
 -it \
 --name devtest \
 --mount type=bind,source="$(pwd)"/target,target=/app,readonly \
 nginx:latest

Let me confirm, brb.