Watchtower update from 19.0 to 19.5 failed and resulted in both my nodes entering a docker restart loop due to the above error. I have now resolved by stopping and recreating the containers. I am unsure why this problem occurred as previous watchtower updates have been OK.
If you check the watchtower logs, do you see anything strange there?
The watchtower logs look normal (see below). Its almost as if watchtower recreated the containers and didnāt apply the same startup flags\variables to the container.
time="2019-08-29T16:39:25Z" level=info msg="First run: 2019-08-29 16:44:25 +0000 UTC"
time="2019-08-30T11:39:37Z" level=info msg="Found new storjlabs/storagenode:alpha image (sha256:90b60ab071cba4aeb3c20c7ec4501cc0cfe22ed8306b4b388a24a18a213bf438)"
time="2019-08-30T11:39:39Z" level=info msg="Found new storjlabs/storagenode:alpha image (sha256:90b60ab071cba4aeb3c20c7ec4501cc0cfe22ed8306b4b388a24a18a213bf438)"
time="2019-08-30T11:39:40Z" level=info msg="Stopping /storagenode2 (e765949dfac5e46c4e67b3199721a66699882bf6924ebc206cbd5bce810d9e5d) with SIGTERM"
time="2019-08-30T11:39:47Z" level=info msg="Stopping /storagenode (aca7700641ad5f60cd0275ec0def7336d08def37c968944debd40152223eb024) with SIGTERM"
time="2019-08-30T11:39:51Z" level=info msg="Creating /storagenode"
time="2019-08-30T11:39:53Z" level=info msg="Creating /storagenode2"I had a similar issue when updating to 0.19.0. This time I updated manually so Iām not sure my issue is resolved or still there. I ended up just manually recreating the container.
Hit this with the update from 0.19.5 to 0.19.6 this morning.
Watchtower log seems normal.
time="2019-09-02T16:32:21Z" level=info msg="Found new storjlabs/storagenode:alpha image (sha256:3c12df329941bf2301472e00c925b0474f759ab4fd748896f57e59ce2f2c36c2)"
time="2019-09-02T16:32:21Z" level=info msg="Stopping /storagenode (890a1425d0122492724158ba96ef81547a9e5ec0119d68d568790d9aaf7b07aa) with SIGTERM"
time="2019-09-02T16:32:28Z" level=info msg="Creating /storagenode"
Rebuilding the container itself worked fine. @julez and @BrightSilence would you guys be on a Synology NAS by any chance?
I had this issue again with the 0.19.6 update as well. Yes, Iām on Synology NAS.
Maybe have a read of this:
Thereās a link in there for the old SPKs and you should be able to just stop your containers, uninstall Docker while preserving the config, manually install the SPK and restart everything. Iāve not tried this in my own Storj environment yet however.
This issue caused all kinds of weirdness with other systems I had running a variety of containers. I canāt say for sure, and Iāll need to do the downgrade and test it myself when Storj next push an update, but I really wouldnāt be surprised if the issue with variable persistence that seems wrapped up in this was causing issues with Watchtower as well.
If itās just us people using Synology NASs facing this issue Iād put good money on the Docker build being kind of broken being the issue.
1 Like
Yeah, that sounds like it could be the problem. Synology mentioned they would fix it with the next update, so for now I have stopped watchtower. Iāll update manually. Havenāt run into issues with other containers yet and I donāt feel like downgrading manually. Thanks for the link though!
Thanks @billstanden . Yes, Iām on Synology too. This definitely looks like it could be the root cause. I will try the rollback and see if this improves the situation. Looks like Synology have acknowledged the issue and working on a fix too. Is there any way to simulate a watchtower update, so we can check it is working OK?
Just saw that your watchtower log sais youāre still using :alpha, I think you should update this to :beta if not already done.
btw. I had the same problem twice with my Synology NAS. First time on august 30 (node went offline 3 days) and second time some hours ago (node was ājustā 3 hours offline).
It currently runs:
- DSM Version: DSM 6.2.2-24922 Update 3
- Installed Docker Version: 18.09.0-0505 (while it sais newest version is 17.05.0-0401, kind of weird)
Does anybody knows if I can set an notification in case that my container goes offline?
Found the notification option under āControl Panel ā Notification ā Push Serviceā
I hope like this I not just get a notification next time the container stops unexpectedly but also once they released an update for Docker.
I just went back to 17.05 by uninstalling and reinstalling the docker package in the package center since newest version is 17.05 again. (stopped containers first and did NOT select the delete-all-data checkbox during the uninstall process)
So far everything looks fine, container running for 20 minutes without unexpected errors.
I have two nodes, both of which are constantly rebooting when they try and update. Running on Synology boxes in Docker. Hereās what I get when I try and run the dashboard:
Error response from daemon: Container BLAHBLAHBLAHBLAHBLAH is restarting, wait until the container is running
Hereās what the logs say:
JSHFLKJSHD@*** : ~ $ sudo docker logs --tail 20 storagenode
2019-09-03T22:22:02.947Z INFO Configuration loaded from: /app/config/config.yaml
2019-09-03T22:22:02.968Z WARN Operator email address isnāt specified.
2019-09-03T22:22:02.968Z ERROR Invalid configuration: operator wallet address isnāt specified
2019-09-03T22:22:02.968Z FATAL Unrecoverable error {āerrorā: āoperator wallet address isnāt specifiedā}
2019-09-03T22:23:04.484Z INFO Configuration loaded from: /app/config/config.yaml
2019-09-03T22:23:04.506Z WARN Operator email address isnāt specified.
2019-09-03T22:23:04.506Z ERROR Invalid configuration: operator wallet address isnāt specified
2019-09-03T22:23:04.506Z FATAL Unrecoverable error {āerrorā: āoperator wallet address isnāt specifiedā}
2019-09-03T22:24:06.071Z INFO Configuration loaded from: /app/config/config.yaml
2019-09-03T22:24:06.092Z WARN Operator email address isnāt specified.
2019-09-03T22:24:06.092Z ERROR Invalid configuration: operator wallet address isnāt specified
2019-09-03T22:24:06.092Z FATAL Unrecoverable error {āerrorā: āoperator wallet address isnāt specifiedā}
2019-09-03T22:25:07.809Z INFO Configuration loaded from: /app/config/config.yaml
2019-09-03T22:25:07.830Z WARN Operator email address isnāt specified.
2019-09-03T22:25:07.830Z ERROR Invalid configuration: operator wallet address isnāt specified
2019-09-03T22:25:07.830Z FATAL Unrecoverable error {āerrorā: āoperator wallet address isnāt specifiedā}
2019-09-03T22:26:09.269Z INFO Configuration loaded from: /app/config/config.yaml
2019-09-03T22:26:09.291Z WARN Operator email address isnāt specified.
2019-09-03T22:26:09.291Z ERROR Invalid configuration: operator wallet address isnāt specified
2019-09-03T22:26:09.291Z FATAL Unrecoverable error {āerrorā: āoperator wallet address isnāt specifiedā}
But they obviously are as if I restart it manually it works fine. Hereās the command I use in watchtower to update:
sudo docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode watchtower --stop-timeout 300s --interval 21600
Thoughts? I have one node still down intentionally so we can troubleshoot this but the other I corrected manually.
OK Iām a big dummy and somehow missed this other thread.
Have same issue. But on xpenology. Docker version 18.09.6, build 8cdf373. Switch off watchtower and update manually.
The best part of this glitch is that uptime robot blows up my mailbox that my node is offline so I know thereās an update to apply manually.
Disabled watchtower for nowā¦ But would be great if I had some kind of alert on node to get info when it is unavailableā¦
Hello @BeardedTinker,
Welcome to the forum!
You have them:
I updated to :beta and now Iām running v0.20.1
Looks to be working for now. Iāll see tomorrow if it is wtill running.
No issues for me with the v0.20.1 upgrade via Watchtower after downgrading the Docker SPK.
1 Like