Update version 1.50 killing my nodes

jammerdan · March 25, 2022, 5:01am

My run command contains the line

-e RUN_PARAMS="--filestore.write-buffer-size 4096kiB --pieces.write-prealloc-size 4096kiB"\

It seems that the update does not like and refuses to run, killing my nodes:

2022-03-25 04:32:56,962 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-25 04:32:56,963 INFO exited: storagenode-updater (exit status 1; not expected)
2022-03-25T04:32:57.094Z	INFO	trust	Scheduling next refresh	{"after": "5h8m56.904777231s"}
2022-03-25T04:32:57.095Z	INFO	bandwidth	Performing bandwidth usage rollups
2022-03-25 04:32:57,097 INFO spawned: 'storagenode-updater' with pid 34
2022-03-25 04:32:57,098 WARN received SIGQUIT indicating exit request
2022-03-25 04:32:57,098 INFO waiting for processes, storagenode, storagenode-updater to die
Error: unknown flag: --pieces.write-prealloc-size

and also

2022-03-25 04:52:46,646 INFO spawned: 'storagenode-updater' with pid 44
Error: unknown flag: --filestore.write-buffer-size

I have to remove that line completely to make the node start again.
So at least I know how to get my nodes working again and it seems that I have to remove this line from all my nodes now. But the question remains, if I need to remove it permanently or if it is a one time issue because of this particular update.

Alexey · March 25, 2022, 5:44am

What happen, if you place these options in the command line instead of variable?

jammerdan · March 25, 2022, 5:54am

What do you mean by ‘command line’? Does it mean without the -e RUN_PARAMS=?

How can I test it? The update has completed already and the affected nodes are running again.

Alexey · March 25, 2022, 5:55am

exactly. Just put your options as the last options in the docker run command (after the image name specifically).
I have my options specified without hacks like you have used, I just have added to the end of the command - and these nodes are happily updated. However, my options was only a wallet feature.

jammerdan · March 25, 2022, 5:59am

After this: --name storagenode storjlabs/storagenode:latest?

Alexey · March 25, 2022, 6:00am

yes.

docker run .... \
--name storagenode storjlabs/storagenode:latest \
--filestore.write-buffer-size 4096kiB \
--pieces.write-prealloc-size 4096kiB

jammerdan · March 25, 2022, 6:04am

Ok, I will add that to one node that has not been updated yet and then we will see if the update will work or not when this nodes update is due.

SGC · March 25, 2022, 9:55am

tried this, seems to work…
sorry that the original solution was a crappy one, i had no idea how to actually do this, so somebody else suggested the variable solution and i just went with that.

haven’t verified that the command actually works… but got no errors and it does show up in the command /entrypoint … when doing docker ps -a

so i will assume it works.

BrightSilence · March 25, 2022, 10:07am

The reason this now causes issues is because the RUN_PARAMS are also passed to the storagenode-updater process now. And that one doesn’t recognize the parameters you set, which are for the storagenode process.

github.com

storj/storj/blob/92ab23202307c92074159c45c5c0ba6d2332b038/cmd/storagenode/docker/entrypoint#L84


      
          
          if [ -n "${WALLET:-}" ]; then
            SNO_RUN_PARAMS="${SNO_RUN_PARAMS} --operator.wallet=${WALLET}"
          fi
          
          if [ "${SETUP:-}" = "true" ]; then
            echo "Running ./storagenode setup $SNO_RUN_PARAMS"
            exec ./storagenode setup ${SNO_RUN_PARAMS} ${*}
          else
            sed -i \
            "s#^command=/app/storagenode-updater\$#command=/app/storagenode-updater run --binary-location /app/storagenode ${RUN_PARAMS} #" \
            /etc/supervisor/supervisord.conf
          
            sed -i \
            "s#^command=/app/storagenode\$#command=/app/storagenode run ${SNO_RUN_PARAMS} ${*}#" \
            /etc/supervisor/supervisord.conf
          
            exec /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
          fi

All parameters passed to the entrypoint (which is what you do if you put them at the end of the run command as suggested) only get passed to the storagenode process using ${*} here.

github.com

storj/storj/blob/92ab23202307c92074159c45c5c0ba6d2332b038/cmd/storagenode/docker/entrypoint#L88


      
          
          if [ "${SETUP:-}" = "true" ]; then
            echo "Running ./storagenode setup $SNO_RUN_PARAMS"
            exec ./storagenode setup ${SNO_RUN_PARAMS} ${*}
          else
            sed -i \
            "s#^command=/app/storagenode-updater\$#command=/app/storagenode-updater run --binary-location /app/storagenode ${RUN_PARAMS} #" \
            /etc/supervisor/supervisord.conf
          
            sed -i \
            "s#^command=/app/storagenode\$#command=/app/storagenode run ${SNO_RUN_PARAMS} ${*}#" \
            /etc/supervisor/supervisord.conf
          
            exec /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
          fi

I use that method for vital settings for my nodes, so I can confirm that definitely works or my nodes wouldn’t even start.

jammerdan · March 25, 2022, 1:34pm

The node where I have added the additional parameters at the end got updated now. So I can confirm it is working this way.

starov · March 25, 2022, 3:09pm

not working. Have this state:

and log:

{"log":"--2022-03-25 15:02:46--  https://github.com/storj/storj/releases/download/v1.50.4/storagenode-updater_linux_amd64.zip\n","stream":"stderr","time":"2022-03-25T15:02:46.071070256Z"}
{"log":"Resolving github.com (github.com)... 140.82.121.4\n","stream":"stderr","time":"2022-03-25T15:02:46.165140482Z"}
{"log":"Connecting to github.com (github.com)|140.82.121.4|:443... connected.\n","stream":"stderr","time":"2022-03-25T15:02:46.224703161Z"}
{"log":"HTTP request sent, awaiting response... 302 Found\n","stream":"stderr","time":"2022-03-25T15:02:46.408860946Z"}
{"log":"Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20220325T150205Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=bbb9afa6affd55022d8da82dbcf8643dd923f0d61b788895c1c3db2ec9982822\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026key_id=0\u0026repo_id=128089774\u0026response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip\u0026response-content-type=application%2Foctet-stream [following]\n","stream":"stderr","time":"2022-03-25T15:02:46.408896635Z"}
{"log":"--2022-03-25 15:02:46--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20220325T150205Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=bbb9afa6affd55022d8da82dbcf8643dd923f0d61b788895c1c3db2ec9982822\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026key_id=0\u0026repo_id=128089774\u0026response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip\u0026response-content-type=application%2Foctet-stream\n","stream":"stderr","time":"2022-03-25T15:02:46.408922343Z"}
{"log":"Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.111.133, ...\n","stream":"stderr","time":"2022-03-25T15:02:46.450841635Z"}

starov · March 25, 2022, 3:32pm

Disable autoupdate and run previous version v1.49.5 (f1a22a725dd5)

Alexey · March 25, 2022, 4:30pm

Could you please show your docker run?

starov · March 25, 2022, 4:33pm

docker run -d --network ovs --ip=10.0.8.222 --dns 10.0.8.254 -p 14002:14002 --restart unless-stopped --stop-timeout 300 -e WALLET=“0xXXXXXXXXXXXXXXXXXXXXXXXXXXX” -e EMAIL="example@example.com" -e ADDRESS=“255.255.255.255:28967” -e STORAGE=“1.9TB” --mount type=bind,source="/home/storj",destination=/app/identity --mount type=bind,source="/mnt/mirror2/storj",destination=/app/config --name storagenode storjlabs/storagenode:latest --filestore.write-buffer-size 40960kiB --pieces.write-prealloc-size 40960kiB

addreess, wallet, email was changed.

Alexey · March 25, 2022, 4:36pm

Accordingly your logs you killed the container before it downloaded a new version.

Could you please run the latest again and wait until it will be up and running?

starov · March 25, 2022, 4:38pm

I have waiting over than 30 minutes and processlist not changed. Traffic in vlan equal 0. Container running and do nothing. Screenshot attached…

Alexey · March 25, 2022, 4:49pm

Please, show the logs instead of pictures. The process list and your picture shows that wget is trying to download the binary and the last message, that it trying to resolve GitHub hostnames. There is no evidence that it’s trying to actually download something.

starov · March 25, 2022, 4:49pm

I try exec command:

docker exec storagenode wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.50.4/storagenode-updater_linux_amd64.zip
–2022-03-25 16:47:06-- https://github.com/storj/storj/releases/download/v1.50.4/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)… 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220325T164514Z&X-Amz-Expires=300&X-Amz-Signature=370d974a464c67cba07f3a6ea2c4cef9ac046db50d2221361ae83065a827ec99&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
–2022-03-25 16:47:06-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220325T164514Z&X-Amz-Expires=300&X-Amz-Signature=370d974a464c67cba07f3a6ea2c4cef9ac046db50d2221361ae83065a827ec99&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)… 185.199.108.133, 185.199.111.133, 185.199.110.133, …
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443…

and nothing happens after the redirect. I think this problem github…

Alexey · March 25, 2022, 4:51pm

Hm. It’s working for my location. Maybe you have some firewall which is blocking traffic from GitHub?

starov · March 25, 2022, 4:55pm

I found problem. Network 185.199.108.0/22 in my router have special static routing in openvpn tunnel. This rule did not allow the update to download in docker vlan. Thanks, v1.50.4 started and doing update

Anyway thanks.