# Storagenode Docker Update: Changes To Binaries Location For Persistence Between Restarts **URL:** \ **Category:** Node Operators\ **Tags:** docker, storagenode, file-permissions\ **Created:** [September 18, 2024, 12:38pm UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937 "2024-09-18T12:38:59Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [September 21, 2024, 2:46am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/24 "2024-09-21T02:46:58Z")
He has a point though… a good email to SNOs and a month notice would be helpful for all with important changes. See this case: [https://forum.storj.io/t/spawnerr-no-permission-to-run-command/27962?u=snorkel](https://forum.storj.io/t/spawnerr-no-permission-to-run-command/27962) What if that guy was in a cruising trip or couldn’t access his nodes for a month? They would all get DQed by Storj’s mods. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 21, 2024, 6:53am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/25 "2024-09-21T06:53:43Z")
I think we would not change the way of communication. > [@Spawnerr: no permission to run command](https://forum.storj.io/t/spawnerr-no-permission-to-run-command/27962/20): > > I do not think that there something regarding changing of the communications channel in the roadmap. There is also a problem with mass email relays - most will add satellites or address which we could use as a spammer and we cannot communicate important things to our customers. So, I would not expect that something would change here. > So, I can only suggest to subscribe to the forum. Maybe only to add an active notifications to the storagenode dashboard, but it is not in a roadmap too. They are almost static or provided by the node itself (like a DQ message or a Suspension message). > [@Scriptiee](#): > > Does falling back to older version is a feasible workaround for the time being until one figures out how to resolve this on their setup? no, it’s not. But you may change the binary location in multiple ways: > [@clement](#): > > You can set a custom path with `BINARY_DIR` environment variable. Default is `/app/config/bin` You may also mount this location from a different drive or even a docker volume. The last one would also solve any permissions issues, I believe. ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [September 21, 2024, 8:16am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/26 "2024-09-21T08:16:26Z")
These changes that potentialy affect in a bad way the node functionality should be optional, not forced imposed on good old working nodes. Many don’t keep an eye on the nodes or their percormance, don’t reed the forum, and expect the nodes to work, once set up and tuned to their particular enviroment, and not be broken by Storj mods or updates. This is the second time a change turned on by default negatively impacted the nodes? 1. The lazzy file walker - a bad and untested addin, that many had to turn it off. 2. The location of the storagenode binary that already broke some nodes. Both were annunced by the same person, but I don’t want to point fingers. Why this trend continuus? When you make a change, announce it, make it optional, let people test it, and after some time, like months, if all is well and dandy, than you can turn it on by default. And this thread dosen’t even stay in Announcements category where it belongs? The title dosen’t draw attention to click on it and to read the thread, also. It shoud begin with “ATTENTION!..” It happens that I click on almost every thread, that I clicked on it too. Didn’t know it was something important. ---
**Author:** ![clement](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/clement/32/11821_2.png) [@clement](https://forum.storj.io/u/clement)\ **Post date:** [September 21, 2024, 9:14am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/27 "2024-09-21T09:14:47Z")
Thank you for your valuable feedback. I apologise for the inconvenience caused by the short notice. Before rolling out this change, there were several factors that were considered: - This change fixes a very critical issue - This is basically a change that majority of SNOs have to do **nothing**. We stopped supporting watchtower a long time ago after we rolled out the auto-updated docker images. So for most setups, unless the container is manually restarted, they will continue to run the old images if the they’re using the `:latest` tag. So this post was actually to make the community aware of the change and what to expect. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 22, 2024, 3:36am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/29 "2024-09-22T03:36:50Z")
> [@Scriptiee](#): > > and now I’m actually running it as pods in k8s Now it could explain to me, why you were need more time for testing. How do you mount a storage there? ---
**Author:** ![digitalfrank](https://storj.bcdn.literatehosting.com/letter_avatar/digitalfrank/32/5_5575768a8748004e209b776fc1b2916d.png) [@digitalfrank](https://forum.storj.io/u/digitalfrank)\ **Post date:** [September 22, 2024, 6:13am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/30 "2024-09-22T06:13:17Z")
Hi, i would notify to all that i have update the docker on UNRAID and it upgrade of the docker is normally and perfectly. I use the standard configuration for the path and the docker is updates automatically ---
**Author:** ![jammerdan](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jammerdan/32/1121_2.png) [@jammerdan](https://forum.storj.io/u/jammerdan)\ **Post date:** [September 22, 2024, 12:40pm UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/31 "2024-09-22T12:40:09Z")
Can it be the reason for this? All of a sudden I am getting the same error: > [@Storagenode stopped working on Pi4 - ERROR: The certificate of 'version.storj.io' is not trusted](https://forum.storj.io/t/storagenode-stopped-working-on-pi4-error-the-certificate-of-version-storj-io-is-not-trusted/27976): > > Hi, my storagenode is not working anymore. Probably it happened after auto-updating via watchtower. The logfile says something about a certificate, that is not trusted and the date in the log also appears very strange for me: --1969-12-31 23:59:59– I am using docker on raspberry pi os (buster). Any idea, what I could do to get it running again? I tried removing and pulling the docker file, but that didn’t help Thanks and kind regards, Hannes ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 23, 2024, 2:35am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/32 "2024-09-23T02:35:21Z")
It could be possible, because the image should also got updates for the underlying OS. However, I tested this new image for ARM64 and it didn’t have this issue. This is usually happen, if a `ca-certificates` package is not updated. The OS is updated by the way: [Dockerfile: upgrade distro buster to bookworm (#22) · storj/storagenode-docker@7784e2a · GitHub](https://github.com/storj/storagenode-docker/commit/7784e2a9d998da2f21199bfe39e8534da2012e54) And the actual problem in the linked thread is: > [@Storagenode stopped working on Pi4 - ERROR: The certificate of 'version.storj.io' is not trusted](https://forum.storj.io/t/storagenode-stopped-working-on-pi4-error-the-certificate-of-version-storj-io-is-not-trusted/27976/2): > > Cannot get REALTIME clock frequency: Operation not permitted thus a solution: > [@Storagenode stopped working on Pi4 - ERROR: The certificate of 'version.storj.io' is not trusted](https://forum.storj.io/t/storagenode-stopped-working-on-pi4-error-the-certificate-of-version-storj-io-is-not-trusted/27976/4): > > You can workaround it by adding --privileged to the docker run command Seems the docker engine may be changed? ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [September 23, 2024, 4:42pm UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/33 "2024-09-23T16:42:15Z")
> [@Scriptiee](#): > > Anyway, I appreciate you at least acknowledging that this is indeed an inconvenience, and not jump on the ship of “well it’s your fault” like some replies I received. I too believe it’s completely and entirely your fault, and the purpose of these comments is not to jump on the banwagon of victim blaming. **Any** software update can bring **unknown number** of **new bugs**. If you are blindly updating to the whatever is pushed to latest tag you are choosing to live by someone else’s schedule and accepting that things will sometimes break, regardless on whether there is any communication about it, including for reasons entirely outside of storj — like os changes. This was already addressed in the comment above [Storagenode Docker Update: Changes To Binaries Location For Persistence Between Restarts - #18 by Ambifacient](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/18) By the way, this is precisely why storj deploys storagenode updates in waves. And yet you assume that container updates are inconsequential and update the container itself asap. Does not seem wise to me. If you want more time — don’t update the containers, or any other software for that matter, automatically. Postphone it until you have time to read the release notes, update, and address all the possible fallout or revert. > [@Scriptiee](#): > > I’m not looking for solutions, I’ve been here long enough to be able to resolve this. And yet you still auto update software. I’m baffled we have to discuss this with someone that has been dealing with software “long enough” > [@Scriptiee](#): > > This is about communication, which was lacking in this case. That’s all. It was communicated in the release notes. That you had practically infinite amount of time to read before installing the update. But you chose to blindly auto-update. TLDR: there can be changes outside of what’s documented, bugs that sneak past QA, and therefore it’s on you to manage updates responsibly and not ingest any new piece of software just because it’s out. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 24, 2024, 6:54am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/34 "2024-09-24T06:54:20Z")
I think we could do a change to affect less non-default setups, if we would download to `/app/config/bin` if there are no binaries, but then copy them to some folder inside the container, which is not exposed to the host and where we can set proper permissions, like `/app/bin` and use it to run binaries from there. Right now there are three solutions: 1. Add `exec` to the mount options in `/etc/fstab`, if possible (`defaults` includes it). 2. Use a docker volume, however, this seems doesn’t work on QNAP for some reason: [Node stopped working as docker container on qnap-container-station - #20 by beli](https://forum.storj.io/t/node-stopped-working-as-docker-container-on-qnap-container-station/27973/20) 3. Use the variable `BINARY_DIR` to redirect the binary folder inside the container. ---
**Author:** ![Roberto](https://storj.bcdn.literatehosting.com/letter_avatar/roberto/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roberto](https://forum.storj.io/u/Roberto)\ **Post date:** [September 24, 2024, 9:03am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/35 "2024-09-24T09:03:25Z")
Just to let you know, I have a Qnap arm, the TS230, and the container upgrade went smoothly ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [September 26, 2024, 12:22am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/36 "2024-09-26T00:22:09Z")
> [@Alexey](#): > > I think we could do a change to affect less non-default setups, if we would download to `/app/config/bin` if there are no binaries, but then copy them to some folder inside the container, which is not exposed to the host and where we can set proper permissions, like `/app/bin` and use it to run binaries from there. Please do consider that approach, because this seems to have broken the very default setup on synology I use. I was able to add the permissions manually, but default synology shares seem to not work. The whole point of containerizing software is to make it able to run on a wide range of different environments. Don’t impose this kind of file rights management on SNOs when you don’t need to. Persisting the binaries is fine, but running them from an external file system is asking for issues. Copying them back into the container and managing rights there would fix all those issues. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 26, 2024, 3:29am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/37 "2024-09-26T03:29:25Z")
> > > Related to #23 > > How to test: > \* build the docker image: > > \`\`\` > docker buildx build …-f Dockerfile -t node:latest . > \`\`\` > > Create an image file and mount it with \`noexec\` to emulate a restricted mount: > > \`\`\` > truncate -s 100M temp.img > mkfs -t ext4 temp.img > mkdir temp-disk > sudo mount -o rw,noexec temp.img temp-disk > sudo chown $(id -i):$(id -g) temp-disk > \`\`\` > > \* Run the container, and wait till the binaries are downloaded: > > \`\`\` > docker run --rm --name test --mount type=bind,source="${PWD}/test-disk",destination=/app/config node:latest > \`\`\` > > \* Check that the \`test-disk\` contains both binaries in "test-disk/bin" > > Rerun the container; binaries should not be downloaded again. > > \* cleanup > > \`\`\` > sudo umount test-disk > rm -rf test-disk test.img > \`\`\` ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [September 26, 2024, 3:38am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/38 "2024-09-26T03:38:05Z")
What if updater updates the binary after the container has been started, outside of entry point? ---
**Author:** ![storaje](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/storaje/32/1129_2.png) [@storaje](https://forum.storj.io/u/storaje)\ **Post date:** [September 26, 2024, 4:25am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/39 "2024-09-26T04:25:05Z")
Unix systems executable will point to the file inode. Updating the file changes the inode. The running process will remain pointed to the inode it started with ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [September 26, 2024, 4:50am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/40 "2024-09-26T04:50:31Z")
Wow, barely 3 hours after my post. Thanks @Alexey , that’s awesome! ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [September 26, 2024, 4:52am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/41 "2024-09-26T04:52:07Z")
That’s not a concern, and is irrelevant in the context of handling the shadow executables location. Two possibilities here: - updater updates file in the app/bin location. Who copies it to app/config/bin? - Updater updates file in the app/config/bin location. Who copies it to app/bin? I.e. updater needs to be aware of these two different locations, or there shall be some wrapper service that wraps the updater and handles copying the executable. I don’t see it in the PR. I see updater downloading new image is being only handled in the entry point. But updater runs every 15 minutes. What happens when one of those invocations result in a new image downloaded? ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 26, 2024, 5:54am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/42 "2024-09-26T05:54:15Z")
Depends on which one. I introduced another one - `BINARY_STORE`, it’s basically a permanent storage on the data location by default. They would be copied when the container is restarted. However, if they are older than the current version, they will be replaced and copied back. I do not know what would happen, if you put there a more new versions, but it’s easy to check 😉 If you would mount other folder for the `BINARY_DIR` location inside the container then - depends on the OS. For Linux you may replace it, but likely it will try to run them only on restart, but before that they will be replaced by binaries from `BINARY_STORE`, which can be updated to the current version if they were outdated (see above). For Windows I think you cannot replace them, they would be locked, however, I do not know how the 9p protocol (WSL2) or SMB/CIFS (Hyper-V) would handle that. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [September 26, 2024, 6:00am UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/43 "2024-09-26T06:00:55Z")
> [@arrogantrabbit](#): > > I see updater downloading new image is being only handled in the entry point. But updater runs every 15 minutes. What happens when one of those invocations result in a new image downloaded? It’s running with these parameters: > So, `storagenode-updater` knows about the binary location `$BINARY_DIR` and there would be all binaries. But you are correct, it wouldn’t copy them back to `$BINARY_STORE`. So issue would be spawned again on the next container restart after the update. If there would be an exception, the container would be restarted and this will fix it. If there wouldn’t be an exception, it will self update again. What would you suggest? ---
**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\ **Post date:** [September 26, 2024, 5:33pm UTC](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937/44 "2024-09-26T17:33:45Z")
> [@Alexey](#): > > What would you suggest? This is an option: > [@arrogantrabbit](#): > > or there shall be some wrapper service that wraps the updater and handles copying the executable. Or, better yet, the change needs to be made in the updater itself — it would need to update the executable in both places, and then restart the node correctly. Alternatively, it could write the most recently used node version it ever downloaded to data store, and then on container start update to at least that version. (I.e if cursor and mask indicates lower version — it will update to the last known higher version) This will solve the original problem without storing executables in the data folder in the first place nor copy them around. [Previous page](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937.md?page=1) [Next page](https://forum.storj.io/t/storagenode-docker-update-changes-to-binaries-location-for-persistence-between-restarts/27937.md?page=3)