Storagenode Docker Update: Changes To Binaries Location For Persistence Between Restarts

Node Operators
, ,
1

TL;DR:

  • The Storj Node Docker image is being updated to store binaries on the host system.
  • This requires executable permissions on the host path that is mounted to /app/config directory.
  • If you have a restrictive filesystem, you may need to adjust permissions.
  • You can set a custom path with BINARY_DIR environment variable. Default is /app/config/bin
  • New docker image tag will be marked as latest in the next 24 hours.

Introduction:

We’re announcing an important update to the storagenode docker image that will affect how binaries are stored within the container. This update aims to improve persistence between container restarts and address a known issue where updates could fail due to database version incompatibilities (See Race condition in the update process in docker (when the new version is incompatible with the old one) · Issue #23 · storj/storagenode-docker · GitHub).

Changes:

  • The new version of the Storj Node Docker image will store binaries in the mounted /app/config directory. This means binaries will be located on your host system’s filesystem instead of within the container itself.
  • To ensure proper functionality, the /app/config directory on your host system needs to allow executable permissions for the downloaded binaries.
  • This does not affect the auto-updater feature. The binaries will still be updated automatically.

What You Need to Do:

  1. Nothing if you don’t have a custom setup with restrictive filesystem for your config path

  2. Check Your Filesystem Permissions:

    • Verify that the /app/config directory on your host system allows executable permissions.
    • Test the new image (storjlabs/storagenode:2159415):
      docker run -d --restart unless-stopped --stop-timeout 300 \
   -p 28967:28967/tcp \
   -p 28967:28967/udp \
   -p 127.0.0.1:14002:14002 \
   -e WALLET="0xXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" \
   -e EMAIL="user@example.com" \
   -e ADDRESS="domain.ddns.net:28967" \
   -e STORAGE="2TB" \
   --user $(id -u):$(id -g) \
   --mount type=bind,source="<identity-dir>",destination=/app/identity \
   --mount type=bind,source="<storage-dir>",destination=/app/config \
   --name storagenode storjlabs/storagenode:2159415

  3. Potential Adjustments:

    • If your system restricts executable permissions, you might need to adjust permissions for the /app/config directory or the user running the container.

Benefits:

  • Improved persistence of downloaded binaries between container restarts.
  • Ensure database version compatibility with current binary

NOTE: The new docker image tag (2159415) will be marked as latest in the next 24 hours.

Additional Resources:

8 Likes
2

So should we expect to see /app/config store multiple versions of binaries? Like today if I have one node on 1.111.4 and one on 1.112.2… /app/config will have a file for both?

Edit: I may have misunderstood that there would be one OS-level directory for all possible versions of node binaries. But now I think you’re saying you’re still using per-container relative /app/config directories… so maybe they would only ever persist the most current binary version?

3

Do you mind stating what errors we will see if we need to make changes but did not change the settings?

4

Yes, it will persist the most current binary version for each node using per-container relative /app/config. So if the mount source paths for your nodes are /mnt/node1 and /mnt/node2, the current binaries will be stored at /mnt/node1/bin/ and /mnt/node2/bin/, respectively.

5

So… to translate for my monkey brain, previously on a docker restart it would download the binary each time, and the new version will preserve it so it just uses the saved version instead?

7 Likes
6

So to change permissions in a linux enviroment, for non-linux guys, the command would be:
sudo chmod 777 /path/storj
where /path/storj is the directory where the config is located.
This makes the directory readable-writable-executable by anyone, if you don’t mind the security risk, if any.

1 Like
7

Yes, that’s the idea. If the binary doesn’t exist, it will be downloaded, otherwise it would be used and then handled by storagenode-updater service inside the container as usual.

2 Likes
8

latest tag still pointing on old release.

9

The new image has been marked as latest now. You can try again

3 Likes
10

No need for 777, only owner needs execute permissions. If you didn’t play with umask a directory should have already execute permissions for the owner.

And unless you mounted the disk with noexec you should be able to execute binaries too.

As @clement wrote, most people need to do nothing.

11

Better approach is to account for this unusual circumstances in the container: on start, copy out the executable to /tmp/, set all the flags needed, and run from there. That way data outside of the container remains data, and can only be executed inside the container deliberately.

1 Like
12
13

Then it wouldn’t survive restarts. This is how it was before and we have had a race condition issue.

14

On restart it copies the same executable from the external volume again.

Its exact same concept as proposed, executables are persisted on the volume, but launched are their copies with attributes adjusted.

15

Then I guess you mean that we should download them to /tmp, then chmod +x /tmp/storagenode*, then move them to /app/config/bin.

Ok, I got it, you mean the reverse: copy from /app/config/bin to /tmp, apply +x and execute from /tmp instead of /app/config/bin.
Then we need to update also dashboard.sh too to use a different path.

1 Like
16

No, download them to /app/config/bin, but launch node like so:

cp /app/config/bin/storagenode /tmp/storagenode
chmod +x /tmp/storagenode 
exec /tmp/storagenode
1 Like
18

To be fair if people are running highly customized deployments they should not be tagging :latest and updating willy-nilly without reading the changes to see if they are going to be affected.

3 Likes
19

A change… that most people won’t even notice… because they have to do nothing? How much notice do you need before a change is applied where you don’t have to do anything? :wink:

It’s nice for Storj to make this post so people with non-standard configs can find it and know what’s going on if something breaks. But those same people knew the risks of messing with a bunch of custom docker mounts when they changed what normally runs out-of-the-box.

3 Likes
21

I’m failing to see how the timeline of notice matters for people doing any sort of custom setup, highly customizable or not because:

  1. The docker image is independent of the node software version, so you can go months if not a year without updating the docker image. In fact my images are tagged a0f1d1ec7754 from 7 months ago.
  2. You should check the changelog to ensure updating to the latest image doesn’t break your custom setup.
3 Likes
22

I don’t know anything about your setup, nor did I state assumptions about it. Most SNOs aren’t affected by this change, and they won’t have to do anything when it rolls out… no matter what node config you may use.

You said it was unacceptable that a change like this only had a couple days notice. You didn’t explain why… but if you’d like to tell us a bit more… that would certainly be constructive. If the vast majority of SNOs don’t have to do anything: why do you feel only more notice would be acceptable?

Do you think this change will affect you? You didn’t say, and I don’t want to assume…

1 Like