Spawnerr: no permission to run command

[blascelle]

Hi,

My storagenode is offline with permissions errors and attempts to recover the node by recreating the docker container with root permissions have not been successful. The errors are:

INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode’
INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode-updater’
INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode’
INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode-updater’
INFO spawnerr: no permission to run command ‘/app/config/bin/storagenode’
INFO gave up: storagenode entered FATAL state, too many start retries too quickly

Has anyone encountered this before?

`

[cj667113]

Yeah if you are mounting your drive with fstab you might need to make adjustments. I ran into this issue as well and had to modify mine. For example my change went from:

/dev/vdb /storj/config auto rw,user,auto 0 0

to

/dev/vdb /storj/config ext4 rw,user,exec 0 0

The reason is because exec allows of the execution binaries that are on that mount. Oddly enough, I believe by default linux with default should mount as exec but for some reason when I checked it was noexec, so by explicitly stating it became exec on reboot/remount.

Edit - So I checked, by default it is exec but if you specify the user option like I did it defaults to noexec requiring explicit exec

[Alexey]

Hello @blascelle,
Welcome to the forum!

@cj667113 is correct, you need to check your /etc/fstab (and /etc/mtab) to know what permissions it actually used.
In our documentation (How do I setup static mount via /etc/fstab for Linux? - Storj Docs) it’s suggested to mount as:

UUID=<your HD UUID> /mnt/<directory name> <FSTYPE> defaults 0 2

accordingly man mount it should allow execution:

   defaults
          Use the default options: rw, suid, dev, exec, auto, nouser, and async.

          Note that the real set of all default mount options depends on kernel and filesystem type.  See the be‐
          ginning of this section for more details.

But also could help to apply chmod +x to the bin subfolder in the data location.

This is related to

Storagenode Docker Update: Changes To Binaries Location For Persistence Between Restarts

TL;DR: The Storj Node Docker image is being updated to store binaries on the host system. This requires executable permissions on the host path that is mounted to /app/config directory. If you have a restrictive filesystem, you may need to adjust permissions. You can set a custom path with BINARY_DIR environment variable. Default is /app/config/bin New docker image tag will be marked as latest in the next 24 hours. Introduction: We’re announcing an important update to the storagenode docker …

[blascelle]

Thanks guys. I made changes to chmod +x and /etc/fstab/. Then rebooted and restarted the container.

Unfortunately, the error still persists.

My original /etc/fstab/:

Here are the current permissions and /etc/fstab/:

What might I still be missing?

[Alexey]

Do you run docker with sudo?
Could you please show the result of the command

ls -l /mnt/storj/storj_data

[izhyrnyy]

The node worked fine
Stopped to increase the node size

docker stop -t 300 storagenode5
docker rm storagenode5
docker pull storjlabs/storagenode:latest
A new image has been uploaded here!

And then I launched my config with the new size.

For testing, I tried to run the node restart script, which had been working perfectly for 5 years - everything is the same, the node does not start with this error.

[Alexey]

Please allow the execution from the bin subfolder in the data location.

[izhyrnyy]

This did not solve the problem

[cj667113]

The placement of exec is important, try
UUID= ext4 rw,users,exec,nofail,noatime 0 0

The reason is because users inherently has noexec so by placing it before stating users it causes issues.

[izhyrnyy]

I don’t understand what you mean?
Is this a mount option fstab?

I don’t use уче4
I use ZFS

[cj667113]

Did you mount this volume via fstab?

If you did please give us the output from “mount | grep /storj_mount”

Let me know if this has noexec there and if it does please give me the output of your fstab.

[izhyrnyy]

done

zfs set exec=on spoolU/storj5

why did you put the binaries in the storage array?

[cj667113]

Think the storj team said something about recovering binaries from failed upgrades is easier there. Will let them chime in more on that. Did setting the ZFS property like that fix your issue?

Personally I am running 7 nodes with ext4 and hit this issue today and figured that some options can cause noexec to be enabled causing execution problems even if you run chmod +x on the binaries in the mounted File Systems.

[izhyrnyy]

Yes, that solved the problem.

[blascelle]

I was able to resolve my issue after some trial and error.

My fstab config needed updating AND my bin folder needed to be executable. I tried defaults 0 2, but without nofail my system failed to boot and it took some time to mount the drive to another system to update the fstab.

Final fstab settings:
defaults,noatime,nofail 0 2

Full fstab file and output of ls -l /mnt/storj/storj_data:

[Alexey]

Storagenode Docker Update: Changes To Binaries Location For Persistence Between Restarts

TL;DR: The Storj Node Docker image is being updated to store binaries on the host system. This requires executable permissions on the host path that is mounted to /app/config directory. If you have a restrictive filesystem, you may need to adjust permissions. You can set a custom path with BINARY_DIR environment variable. Default is /app/config/bin New docker image tag will be marked as latest in the next 24 hours. Introduction: We’re announcing an important update to the storagenode docker …

[Alexey]

Thank you for your feedback.
I would suggest to subscribe to announcements though.

It doesn’t have issues on a default setup, exec is enabled by default, if you didn’t change a mount options for your drive from recommended.

[Alexey]

We do not use your email for announcements, and trying to reduce the spam from us to a required minimum like notifications when your node goes offline and returns online.

I do not think that there something regarding changing of the communications channel in the roadmap. There is also a problem with mass email relays - most will add satellites or address which we could use as a spammer and we cannot communicate important things to our customers. So, I would not expect that something would change here.
So, I can only suggest to subscribe to the forum.

[Alexey]

We got them after that, thus - changed.

You can create a feature request here Storage Node feature requests - voting or on our GitHub.

However, I would recommend to subscribe at least to Node Operators and Announcements anyway, then you likely would be notified in time.

[Alexey]

Yes, 2 days is in time for me. I noticed a notification and tested, that my docker setup will works, I used the same image which now is latest. 3 minutes test, and I free to go.
I did notice it in the end of the day though, because I was busy, however, it was enough.
The expectation for this change that you shouldn’t do anything, it will just works.
However, if you have a custom setup, then you should spend more time to monitor changes and make sure that your custom setup is compatible too. This is price for customization and I do not personally see any problem here.
You also will be notified when your node would go down. And you will check the forum, or Google and find the forum.

I would recommend to configure the ability to manage your node remotely, it helped many times. For my RPI I also bought a smart plug to be able to reboot it if it would hang, it’s helped several times. But I would agree, that IPMI would be better and not depend on an SD card… but well. Other nodes are still remotely available for me, so I may fix issues.

How to remote access the web dashboard - Storj Docs

@Scriptiee

And you don’t have to give me the “courtesy” of a reply to this, since it appears all you try to do is deflect a very valid criticism of the way you handled this.

I welcomed your feedback. I can only offer to submit a feature request for the consideration by the team or fix the issue right now by subscribing to the forum.
I tried to explain, why is it not a high priority and why we stopped to use provided email addresses. We have had less operators that time and were below the threshold when spam filters may block us for a massive emails. Now it’s not the case.