Watchtower updates fail on Synology NAS with Docker v18.09.6 > FATAL Unrecoverable error {"error": "operator wallet address isn't specified"}

Watchtower update from 19.0 to 19.5 failed and resulted in both my nodes entering a docker restart loop due to the above error. I have now resolved by stopping and recreating the containers. I am unsure why this problem occurred as previous watchtower updates have been OK.

1 Like

If you check the watchtower logs, do you see anything strange there?

The watchtower logs look normal (see below). Its almost as if watchtower recreated the containers and didnā€™t apply the same startup flags\variables to the container.

time="2019-08-29T16:39:25Z" level=info msg="First run: 2019-08-29 16:44:25 +0000 UTC" 
time="2019-08-30T11:39:37Z" level=info msg="Found new storjlabs/storagenode:alpha image (sha256:90b60ab071cba4aeb3c20c7ec4501cc0cfe22ed8306b4b388a24a18a213bf438)" 
time="2019-08-30T11:39:39Z" level=info msg="Found new storjlabs/storagenode:alpha image (sha256:90b60ab071cba4aeb3c20c7ec4501cc0cfe22ed8306b4b388a24a18a213bf438)" 
time="2019-08-30T11:39:40Z" level=info msg="Stopping /storagenode2 (e765949dfac5e46c4e67b3199721a66699882bf6924ebc206cbd5bce810d9e5d) with SIGTERM" 
time="2019-08-30T11:39:47Z" level=info msg="Stopping /storagenode (aca7700641ad5f60cd0275ec0def7336d08def37c968944debd40152223eb024) with SIGTERM" 
time="2019-08-30T11:39:51Z" level=info msg="Creating /storagenode" 
time="2019-08-30T11:39:53Z" level=info msg="Creating /storagenode2"

I had a similar issue when updating to 0.19.0. This time I updated manually so Iā€™m not sure my issue is resolved or still there. I ended up just manually recreating the container.

Hit this with the update from 0.19.5 to 0.19.6 this morning.

Watchtower log seems normal.

time="2019-09-02T16:32:21Z" level=info msg="Found new storjlabs/storagenode:alpha image (sha256:3c12df329941bf2301472e00c925b0474f759ab4fd748896f57e59ce2f2c36c2)"
time="2019-09-02T16:32:21Z" level=info msg="Stopping /storagenode (890a1425d0122492724158ba96ef81547a9e5ec0119d68d568790d9aaf7b07aa) with SIGTERM"
time="2019-09-02T16:32:28Z" level=info msg="Creating /storagenode"

Rebuilding the container itself worked fine. @julez and @BrightSilence would you guys be on a Synology NAS by any chance?

I had this issue again with the 0.19.6 update as well. Yes, Iā€™m on Synology NAS.

Maybe have a read of this:

Thereā€™s a link in there for the old SPKs and you should be able to just stop your containers, uninstall Docker while preserving the config, manually install the SPK and restart everything. Iā€™ve not tried this in my own Storj environment yet however.

This issue caused all kinds of weirdness with other systems I had running a variety of containers. I canā€™t say for sure, and Iā€™ll need to do the downgrade and test it myself when Storj next push an update, but I really wouldnā€™t be surprised if the issue with variable persistence that seems wrapped up in this was causing issues with Watchtower as well.

If itā€™s just us people using Synology NASs facing this issue Iā€™d put good money on the Docker build being kind of broken being the issue.

1 Like

Yeah, that sounds like it could be the problem. Synology mentioned they would fix it with the next update, so for now I have stopped watchtower. Iā€™ll update manually. Havenā€™t run into issues with other containers yet and I donā€™t feel like downgrading manually. Thanks for the link though!

Thanks @billstanden . Yes, Iā€™m on Synology too. This definitely looks like it could be the root cause. I will try the rollback and see if this improves the situation. Looks like Synology have acknowledged the issue and working on a fix too. Is there any way to simulate a watchtower update, so we can check it is working OK?

Just saw that your watchtower log sais youā€™re still using :alpha, I think you should update this to :beta if not already done.

btw. I had the same problem twice with my Synology NAS. First time on august 30 (node went offline 3 days) and second time some hours ago (node was ā€˜justā€™ 3 hours offline).
It currently runs:

  • DSM Version: DSM 6.2.2-24922 Update 3
  • Installed Docker Version: 18.09.0-0505 (while it sais newest version is 17.05.0-0401, kind of weird)

Does anybody knows if I can set an notification in case that my container goes offline?

Found the notification option under ā€œControl Panel ā†’ Notification ā†’ Push Serviceā€
I hope like this I not just get a notification next time the container stops unexpectedly but also once they released an update for Docker.

I just went back to 17.05 by uninstalling and reinstalling the docker package in the package center since newest version is 17.05 again. (stopped containers first and did NOT select the delete-all-data checkbox during the uninstall process)
So far everything looks fine, container running for 20 minutes without unexpected errors.

I have two nodes, both of which are constantly rebooting when they try and update. Running on Synology boxes in Docker. Hereā€™s what I get when I try and run the dashboard:

Error response from daemon: Container BLAHBLAHBLAHBLAHBLAH is restarting, wait until the container is running

Hereā€™s what the logs say:

JSHFLKJSHD@*** : ~ $ sudo docker logs --tail 20 storagenode

2019-09-03T22:22:02.947Z INFO Configuration loaded from: /app/config/config.yaml

2019-09-03T22:22:02.968Z WARN Operator email address isnā€™t specified.

2019-09-03T22:22:02.968Z ERROR Invalid configuration: operator wallet address isnā€™t specified

2019-09-03T22:22:02.968Z FATAL Unrecoverable error {ā€œerrorā€: ā€œoperator wallet address isnā€™t specifiedā€}

2019-09-03T22:23:04.484Z INFO Configuration loaded from: /app/config/config.yaml

2019-09-03T22:23:04.506Z WARN Operator email address isnā€™t specified.

2019-09-03T22:23:04.506Z ERROR Invalid configuration: operator wallet address isnā€™t specified

2019-09-03T22:23:04.506Z FATAL Unrecoverable error {ā€œerrorā€: ā€œoperator wallet address isnā€™t specifiedā€}

2019-09-03T22:24:06.071Z INFO Configuration loaded from: /app/config/config.yaml

2019-09-03T22:24:06.092Z WARN Operator email address isnā€™t specified.

2019-09-03T22:24:06.092Z ERROR Invalid configuration: operator wallet address isnā€™t specified

2019-09-03T22:24:06.092Z FATAL Unrecoverable error {ā€œerrorā€: ā€œoperator wallet address isnā€™t specifiedā€}

2019-09-03T22:25:07.809Z INFO Configuration loaded from: /app/config/config.yaml

2019-09-03T22:25:07.830Z WARN Operator email address isnā€™t specified.

2019-09-03T22:25:07.830Z ERROR Invalid configuration: operator wallet address isnā€™t specified

2019-09-03T22:25:07.830Z FATAL Unrecoverable error {ā€œerrorā€: ā€œoperator wallet address isnā€™t specifiedā€}

2019-09-03T22:26:09.269Z INFO Configuration loaded from: /app/config/config.yaml

2019-09-03T22:26:09.291Z WARN Operator email address isnā€™t specified.

2019-09-03T22:26:09.291Z ERROR Invalid configuration: operator wallet address isnā€™t specified

2019-09-03T22:26:09.291Z FATAL Unrecoverable error {ā€œerrorā€: ā€œoperator wallet address isnā€™t specifiedā€}

But they obviously are as if I restart it manually it works fine. Hereā€™s the command I use in watchtower to update:

sudo docker run -d --restart=always --name watchtower -v /var/run/docker.sock:/var/run/docker.sock storjlabs/watchtower storagenode watchtower --stop-timeout 300s --interval 21600

Thoughts? I have one node still down intentionally so we can troubleshoot this but the other I corrected manually.

OK Iā€™m a big dummy and somehow missed this other thread.

Have same issue. But on xpenology. Docker version 18.09.6, build 8cdf373. Switch off watchtower and update manually.

The best part of this glitch is that uptime robot blows up my mailbox that my node is offline so I know thereā€™s an update to apply manually.

Disabled watchtower for nowā€¦ But would be great if I had some kind of alert on node to get info when it is unavailableā€¦

Hello @BeardedTinker,
Welcome to the forum!

You have them:

I updated to :beta and now Iā€™m running v0.20.1

Looks to be working for now. Iā€™ll see tomorrow if it is wtill running.

No issues for me with the v0.20.1 upgrade via Watchtower after downgrading the Docker SPK.

1 Like