# \[Tech Preview\] Storagenode Updater inside Docker **URL:** \ **Category:** STORJLINGS\ **Created:** [March 16, 2022, 6:13pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706 "2022-03-16T18:13:46Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [March 16, 2022, 6:13pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/1 "2022-03-16T18:13:46Z")
`v1.50.3` is ready for the storage node rollout. This time we have the storage node updater inside docker to test. The risks are high. Worst case docker nodes will not startup or will get stuck on updating to the following release. We need your help to mitigate these risks on as many test machines as possible. Please help us. The new docker image is already published but not tagged as the latest. So your existing docker node will not install it yet. We would like to run 2 tests with your help. For the first test round, please set up a test node and join the test network: [Please join our public test network](https://forum.storj.io/t/please-join-our-public-test-network/15954) Later this week we are going to deploy v1.51.0-rc on the test network. Hopefully, your test node will update just fine. After the first test round, there is still the risk that we crash production. So in a second test round, we would like to update as many mainnet nodes as possible before the docker image gets tagged as the latest. The last step after we finished both tests rounds will be a cleanup. For both test rounds, you will have to specify a specific docker image `5f2777af9-v1.50.4-go1.17.5`. At the end please reconfigure your system to switch back to the latest docker image. ---
**Author:** ![clement](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/clement/32/11821_2.png) [@clement](https://forum.storj.io/u/clement)\ **Post date:** [March 16, 2022, 11:58pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/3 "2022-03-16T23:58:08Z")
This issue might give context on why we decided to this: [Support slow storage node version rollout for all existing Linux/Docker installations · Issue #4489 · storj/storj · GitHub](https://github.com/storj/storj/issues/4489#issue-1137497729) ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [March 17, 2022, 8:28am UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/4 "2022-03-17T08:28:14Z")
Not a great start on my test node. Swapped out the image and it got stuck in restarting with the following repeated logs. ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/e/e9c2da4ecc7e683cde03041d65e3ed14ae8e849b.png) I also noticed that it’s connecting to the production version server instead of test for the initial download of the updater. Should we be setting VERSION\_SERVER\_URL in the run command? It seems to default to production in the docker file atm. I’ll revert for now. Let me provide some context. This is on a Synology DS3617xs, which is amd64 architecture. The only thing I changed in my (working) run command was the image tag. ```auto docker run -d --restart unless-stopped --stop-timeout 300 -p 28968:28967/tcp -p 28968:28967/udp -p 14003:14002 \ -e WALLET="redacted" \ -e EMAIL="redacted" \ -e ADDRESS="redacted:28968" \ -e STORAGE="1TB" \ --mount type=bind,source="/volume1/storj/test/identity",destination=/app/identity \ --mount type=bind,source="/volume1/storj/test/data",destination=/app/config \ --name storagenodetest storjlabs/storagenode:92ab23202-v1.50.3-go1.17.5 \ --operator.wallet-features="zksync" \ --log.level="debug" \ --log.output="/app/config/node.log" \ --server.use-peer-ca-whitelist="false" \ --storage2.trust.sources="1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU@satellite.qa.storj.io:7777,12ZQbQ8WWFEfKNE9dP78B1frhJ8PmyYmr8occLEf1mQ1ovgVWy@testnet.satellite.stefan-benten.de:7777" \ --version.server-address="https://version.qa.storj.io" ``` ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [March 17, 2022, 11:04am UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/6 "2022-03-17T11:04:58Z")
> [@BrightSilence](#): > > Swapped out the image and it got stuck in restarting with the following repeated logs. Oh no. I did a mistake. Version control is currently pointing to a not existing version. Should be fixed now. Please retry. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [March 17, 2022, 11:15am UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/7 "2022-03-17T11:15:22Z")
Getting the same result. Should I be using a new image? ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/3/351b9a0fc76a1a6e8a4526d527ac6edda4deda69.png) Also, this seems like a pretty big flaw in the system. If for some reason the url isn’t reachable or the file doesn’t exist, recreating the container would simply make it impossible for the node to start again. Could a fallback be built in that keeps a copy of the last working binaries in the storage location and reverts to them in case something like that happens? Otherwise I’m really not feeling good about this approach. I also agree with @CutieePie that this isn’t really how docker should work. You already use a custom version of watchtower, wouldn’t it be possible to make watchtower aware of the latest version have it use the rollout system to determine when to trigger automated updates of the node? I’m aware that this wouldn’t block SNOs from updating earlier, but how many would really do that manually? And if they do, how likely is it that they will do that very quickly? I just really don’t like that I could end up with a container that is unable to even download the binary. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [March 17, 2022, 11:32am UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/8 "2022-03-17T11:32:56Z")
Ok great. I did the second mistake as well and forgot to publish the v1.50.3 release. Now that is also fixed and I see no more reason why it shouldn’t find the binaries now. ---
**Author:** ![clement](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/clement/32/11821_2.png) [@clement](https://forum.storj.io/u/clement)\ **Post date:** [March 17, 2022, 12:11pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/9 "2022-03-17T12:11:32Z")
> [@BrightSilence](#): > > Should we be setting VERSION\_SERVER\_URL in the run command? It seems to default to production in the docker file atm. Yes you should set `VERSION_SERVER_URL` in the run command to point to `https://version.qa.storj.io` ---
**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\ **Post date:** [March 17, 2022, 1:19pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/10 "2022-03-17T13:19:34Z")
Any way to opt out of this? It looks bad to me. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [March 17, 2022, 2:35pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/11 "2022-03-17T14:35:20Z")
> [@clement](#): > > Yes you should set `VERSION_SERVER_URL` in the run command to point to `https://version.qa.storj.io` Will do! This should probably be added here as well then: [Please join our public test network](https://forum.storj.io/t/please-join-our-public-test-network/15954) Tried again and also set the VERSION\_SERVER\_URL in the run command, but same result. ![image](https://storj-s3.bcdn.literatehosting.net/original/2X/e/ece07297f0b24d6f0c5a34af6a0645ffbd4c09be.png) Also tried without the VERSION\_SERVER\_URL set and get the same error (with different url of course). ---
**Author:** ![peem](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/peem/32/6323_2.png) [@peem](https://forum.storj.io/u/peem)\ **Post date:** [March 17, 2022, 2:59pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/12 "2022-03-17T14:59:36Z")
> [@BrightSilence](#): > > Tried again and also set the VERSION\_SERVER\_URL in the run command, but same result. Or maybe: ```auto --version.server-url="https://version.qa.storj.io" ``` ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [March 17, 2022, 3:40pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/13 "2022-03-17T15:40:41Z")
There is another option for that which I already set. But the entrypoint script uses the environment variable before the binaries ever run. So for now you have to set it twice. I would suggest using the environment variable for both. But as you can see from the log I posted, setting the environment variable made it use the QA server. But it still didn’t work. ---
**Author:** ![LrrrAc](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/lrrrac/32/8809_2.png) [@LrrrAc](https://forum.storj.io/u/LrrrAc)\ **Post date:** [March 17, 2022, 5:39pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/14 "2022-03-17T17:39:51Z")
The issue linked above implies that this fixes a problem of being unable to push a fix for bad code to docker containers. Does that mean this updater will only be used in rare cases of known bad code as opposed to a general updater? I would consider this a much better idea than having a self-updating docker container. That would at a minimum increase the difficulty of downgrading. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [March 17, 2022, 7:05pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/15 "2022-03-17T19:05:22Z")
downgrade is a bad idea. If there are migrations for database - it will not be reverted and your downgraded version would not work. ---
**Author:** ![clement](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/clement/32/11821_2.png) [@clement](https://forum.storj.io/u/clement)\ **Post date:** [March 17, 2022, 10:26pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/16 "2022-03-17T22:26:32Z")
I’ve been trying to reproduce the issue for hours but still unable to reproduce; at least on docker desktop for Mac. I’m still unable to reproduce this with both test and production version server URL: ```auto docker run -d --restart unless-stopped --stop-timeout 300 -p 14003:14002 -p 28968:28967/tcp -p 28968:28967/udp \ -e WALLET="0xXX" \ -e EMAIL="email" \ -e ADDRESS="dns:28968" \ -e STORAGE="1TB" \ -e VERSION_SERVER_URL="https://version.qa.storj.io" \ --mount type=bind,source="...",destination=/app/identity \ --mount type=bind,source="...",destination=/app/config \ --name nodetestnet \ storjlabs/storagenode:35efb6462-go1.17.5 ``` The output as expected: ```auto downloading storagenode-updater Connecting to version.qa.storj.io (35.188.169.133:443) writing to stdout - 100% | ******************************** | 92 0:00:00 ETA written to stdout Connecting to github.com (140.82.121.3:443) Connecting to objects.githubusercontent.com (185.199.111.133:443) saving to '/tmp/storagenode-updater.zip' storagenode-updater. 100% | ******************************** | 7910k 0:00:00 ETA ... ``` ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [March 17, 2022, 10:30pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/17 "2022-03-17T22:30:18Z")
Anything I can do on my end to help figure it out? ---
**Author:** ![clement](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/clement/32/11821_2.png) [@clement](https://forum.storj.io/u/clement)\ **Post date:** [March 17, 2022, 10:40pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/18 "2022-03-17T22:40:46Z")
What happens when you change the entrypoint to sh and connect to the QA version server with wget? This should open the terminal inside the container: ```auto docker run -i -t --name testnetwork --rm --entrypoint sh storjlabs/storagenode:35efb6462-go1.17.5 ``` Connect to the QA version server: ```auto wget -O- version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64 ``` If wget fails to connect, install GNU wget and try again: ```auto apk add wget wget -O- version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64 ``` ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [March 17, 2022, 10:44pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/19 "2022-03-17T22:44:48Z")
```auto $ docker run -i -t --name testnetwork --rm --entrypoint sh storjlabs/storagenode:35efb6462-go1.17.5 Unable to find image 'storjlabs/storagenode:35efb6462-go1.17.5' locally 35efb6462-go1.17.5: Pulling from storjlabs/storagenode Digest: sha256:57afdc03a5e31106bc929bd24c201150dcfffecaab68be65c9f45a04e5776940 Status: Downloaded newer image for storjlabs/storagenode:35efb6462-go1.17.5 /app # wget -O- version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64 /app # Connecting to version.storj.io (35.224.88.204:80) Connecting to version.storj.io (35.224.88.204:443) wget: error getting response: Connection reset by peer apk add wget fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz (1/3) Installing libunistring (0.9.10-r1) (2/3) Installing libidn2 (2.3.2-r0) (3/3) Installing wget (1.21.2-r2) Executing busybox-1.34.1-r3.trigger OK: 67 MiB in 38 packages [1]+ Done(1) wget -O- version.storj.io/processes/storagenode/minimum/url?os=linux /app # wget -O- version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64 /app # Redirecting output to 'wget-log'. /app # ``` I did also notice you were using a different image than in the top post. ---
**Author:** ![clement](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/clement/32/11821_2.png) [@clement](https://forum.storj.io/u/clement)\ **Post date:** [March 17, 2022, 10:48pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/20 "2022-03-17T22:48:56Z")
Uh, so my suspicion has been confirmed. It is an issue with BusyBox wget, so we have to switch to GNU wget in the storagenode-base image. ---
**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\ **Post date:** [March 17, 2022, 10:50pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/21 "2022-03-17T22:50:48Z")
I think you’d also have to encapsulate the url in quotes. It seems to push it to the background because of the & in there. ```auto $ docker run -i -t --name testnetwork --rm --entrypoint sh storjlabs/storagenode:35efb6462-go1.17.5 /app # wget -O- "version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64" Connecting to version.storj.io (35.224.88.204:80) Connecting to version.storj.io (35.224.88.204:443) wget: error getting response: Connection reset by peer /app # wget -O- "https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64" Connecting to version.storj.io (35.224.88.204:443) wget: error getting response: Connection reset by peer /app # apk add wget fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz (1/3) Installing libunistring (0.9.10-r1) (2/3) Installing libidn2 (2.3.2-r0) (3/3) Installing wget (1.21.2-r2) Executing busybox-1.34.1-r3.trigger OK: 67 MiB in 38 packages /app # wget -O- "https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64" --2022-03-17 22:49:47-- https://version.storj.io/processes/storagenode/minimum/url?os=linux&arch=amd64 Resolving version.storj.io (version.storj.io)... 35.224.88.204 Connecting to version.storj.io (version.storj.io)|35.224.88.204|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 84 [text/plain] Saving to: 'STDOUT' - 0%[] 0 --.-KB/s h- 100%[============================================================================================================================================================================>] 84 --.-KB/s in 0s 2022-03-17 22:49:47 (25.3 MB/s) - written to stdout [84/84] /app # ``` Looks like the entrypoint already does that though. > ---
**Author:** ![clement](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/clement/32/11821_2.png) [@clement](https://forum.storj.io/u/clement)\ **Post date:** [March 17, 2022, 10:53pm UTC](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706/22 "2022-03-17T22:53:08Z")
> [@BrightSilence](#): > > I think you’d also have to encapsulate the url in quotes. It seems to push it to the background because of the & in there. Yes that’s true, but it still fails with BusyBox wget [Next page](https://forum.storj.io/t/tech-preview-storagenode-updater-inside-docker/17706.md?page=2)