Update version 1.50 killing my nodes

My run command contains the line

-e RUN_PARAMS="--filestore.write-buffer-size 4096kiB --pieces.write-prealloc-size 4096kiB"\

It seems that the update does not like and refuses to run, killing my nodes:

2022-03-25 04:32:56,962 INFO success: storagenode-updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-25 04:32:56,963 INFO exited: storagenode-updater (exit status 1; not expected)
2022-03-25T04:32:57.094Z	INFO	trust	Scheduling next refresh	{"after": "5h8m56.904777231s"}
2022-03-25T04:32:57.095Z	INFO	bandwidth	Performing bandwidth usage rollups
2022-03-25 04:32:57,097 INFO spawned: 'storagenode-updater' with pid 34
2022-03-25 04:32:57,098 WARN received SIGQUIT indicating exit request
2022-03-25 04:32:57,098 INFO waiting for processes, storagenode, storagenode-updater to die
Error: unknown flag: --pieces.write-prealloc-size

and also

2022-03-25 04:52:46,646 INFO spawned: 'storagenode-updater' with pid 44
Error: unknown flag: --filestore.write-buffer-size

I have to remove that line completely to make the node start again.
So at least I know how to get my nodes working again and it seems that I have to remove this line from all my nodes now. But the question remains, if I need to remove it permanently or if it is a one time issue because of this particular update.

What happen, if you place these options in the command line instead of variable?

What do you mean by ‘command line’? Does it mean without the -e RUN_PARAMS=?

How can I test it? The update has completed already and the affected nodes are running again.

exactly. Just put your options as the last options in the docker run command (after the image name specifically).
I have my options specified without hacks like you have used, I just have added to the end of the command - and these nodes are happily updated. However, my options was only a wallet feature.

After this: --name storagenode storjlabs/storagenode:latest?

yes.

docker run .... \
--name storagenode storjlabs/storagenode:latest \
--filestore.write-buffer-size 4096kiB \
--pieces.write-prealloc-size 4096kiB
1 Like

Ok, I will add that to one node that has not been updated yet and then we will see if the update will work or not when this nodes update is due.

1 Like

tried this, seems to work…
sorry that the original solution was a crappy one, i had no idea how to actually do this, so somebody else suggested the variable solution and i just went with that.

haven’t verified that the command actually works… but got no errors and it does show up in the command /entrypoint … when doing docker ps -a

so i will assume it works.

The reason this now causes issues is because the RUN_PARAMS are also passed to the storagenode-updater process now. And that one doesn’t recognize the parameters you set, which are for the storagenode process.

All parameters passed to the entrypoint (which is what you do if you put them at the end of the run command as suggested) only get passed to the storagenode process using ${*} here.

I use that method for vital settings for my nodes, so I can confirm that definitely works or my nodes wouldn’t even start.

4 Likes

The node where I have added the additional parameters at the end got updated now. So I can confirm it is working this way.

3 Likes

not working. Have this state:

and log:

{"log":"--2022-03-25 15:02:46--  https://github.com/storj/storj/releases/download/v1.50.4/storagenode-updater_linux_amd64.zip\n","stream":"stderr","time":"2022-03-25T15:02:46.071070256Z"}
{"log":"Resolving github.com (github.com)... 140.82.121.4\n","stream":"stderr","time":"2022-03-25T15:02:46.165140482Z"}
{"log":"Connecting to github.com (github.com)|140.82.121.4|:443... connected.\n","stream":"stderr","time":"2022-03-25T15:02:46.224703161Z"}
{"log":"HTTP request sent, awaiting response... 302 Found\n","stream":"stderr","time":"2022-03-25T15:02:46.408860946Z"}
{"log":"Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20220325T150205Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=bbb9afa6affd55022d8da82dbcf8643dd923f0d61b788895c1c3db2ec9982822\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026key_id=0\u0026repo_id=128089774\u0026response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip\u0026response-content-type=application%2Foctet-stream [following]\n","stream":"stderr","time":"2022-03-25T15:02:46.408896635Z"}
{"log":"--2022-03-25 15:02:46--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256\u0026X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request\u0026X-Amz-Date=20220325T150205Z\u0026X-Amz-Expires=300\u0026X-Amz-Signature=bbb9afa6affd55022d8da82dbcf8643dd923f0d61b788895c1c3db2ec9982822\u0026X-Amz-SignedHeaders=host\u0026actor_id=0\u0026key_id=0\u0026repo_id=128089774\u0026response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip\u0026response-content-type=application%2Foctet-stream\n","stream":"stderr","time":"2022-03-25T15:02:46.408922343Z"}
{"log":"Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.111.133, ...\n","stream":"stderr","time":"2022-03-25T15:02:46.450841635Z"}

Disable autoupdate and run previous version v1.49.5 (f1a22a725dd5)

Could you please show your docker run?

docker run -d --network ovs --ip=10.0.8.222 --dns 10.0.8.254 -p 14002:14002 --restart unless-stopped --stop-timeout 300 -e WALLET=“0xXXXXXXXXXXXXXXXXXXXXXXXXXXX” -e EMAIL="example@example.com" -e ADDRESS=“255.255.255.255:28967” -e STORAGE=“1.9TB” --mount type=bind,source="/home/storj",destination=/app/identity --mount type=bind,source="/mnt/mirror2/storj",destination=/app/config --name storagenode storjlabs/storagenode:latest --filestore.write-buffer-size 40960kiB --pieces.write-prealloc-size 40960kiB

addreess, wallet, email was changed.

Accordingly your logs you killed the container before it downloaded a new version.

Could you please run the latest again and wait until it will be up and running?

I have waiting over than 30 minutes and processlist not changed. Traffic in vlan equal 0. Container running and do nothing. Screenshot attached…

Please, show the logs instead of pictures. The process list and your picture shows that wget is trying to download the binary and the last message, that it trying to resolve GitHub hostnames. There is no evidence that it’s trying to actually download something.

I try exec command:

docker exec storagenode wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.50.4/storagenode-updater_linux_amd64.zip
–2022-03-25 16:47:06-- https://github.com/storj/storj/releases/download/v1.50.4/storagenode-updater_linux_amd64.zip
Resolving github.com (github.com)… 140.82.121.4
Connecting to github.com (github.com)|140.82.121.4|:443… connected.
HTTP request sent, awaiting response… 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220325T164514Z&X-Amz-Expires=300&X-Amz-Signature=370d974a464c67cba07f3a6ea2c4cef9ac046db50d2221361ae83065a827ec99&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [following]
–2022-03-25 16:47:06-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/dfa144f5-ef82-4c12-85ce-c92acb9f202d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220325T164514Z&X-Amz-Expires=300&X-Amz-Signature=370d974a464c67cba07f3a6ea2c4cef9ac046db50d2221361ae83065a827ec99&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)… 185.199.108.133, 185.199.111.133, 185.199.110.133, …
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443…

and nothing happens after the redirect. I think this problem github…

Hm. It’s working for my location. Maybe you have some firewall which is blocking traffic from GitHub?

I found problem. Network 185.199.108.0/22 in my router have special static routing in openvpn tunnel. This rule did not allow the update to download in docker vlan. Thanks, v1.50.4 started and doing update

Anyway thanks.

3 Likes