Watchtower errors

Node Operators troubleshooting
, ,
1

Hi!

Looks like got some problem with watchtower update:

‘’’
time=“2022-11-04T10:04:03Z” level=info msg=“Unable to update container /watchtower. Proceeding to next.”
time=“2022-11-05T08:12:27Z” level=info msg=“Unable to update container /storagenode2. Proceeding to next.”
time=“2022-11-05T08:12:27Z” level=info msg=“Unable to update container /storagenode3. Proceeding to next.”
time=“2022-11-05T08:12:27Z” level=info msg=“Unable to update container /storagenode5. Proceeding to next.”
time=“2022-11-05T08:12:27Z” level=info msg=“Unable to update container /storagenode1. Proceeding to next.”
time=“2022-11-05T08:12:27Z” level=info msg=“Unable to update container /watchtower. Proceeding to next.”
time=“2022-11-05T10:27:08Z” level=info msg=“Waiting for running update to be finished…”
time=“2022-11-06T05:29:51Z” level=info msg=“Waiting for running update to be finished…”
‘’’

And the following in node:

2022-11-06T05:03:30.249Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}

2022-11-06T05:03:30.470Z ERROR Error retrieving version info. {“Process”: “storagenode-updater”, “error”: “version checker client: Get "https://version.storj.io": EOF”, “errorVerbose”: “version checker client: Get "https://version.storj.io": EOF\n\tstorj.io/storj/private/version/checker.
(*Client).All:68\n\tmain.loopFunc:21\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:378\n\tstorj.io/private/process.cleanup.func1:396\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:20\n\truntime.main:255”}

2022-11-06T05:18:30.263Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}

2022-11-06T05:18:30.482Z ERROR Error retrieving version info. {“Process”: “storagenode-updater”, “error”: “version checker client: Get "https://version.storj.io": EOF”, “errorVerbose”: “version checker client: Get "https://version.storj.io": EOF\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tmain.loopFunc:21\n\tstorj.io/common/sync2.(*Cycle).Run:160\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:378\n\tstorj.io/private/process.cleanup.func1:396\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:20\n\truntime.main:255”}

2022-11-06 05:29:51,565 WARN received SIGTERM indicating exit request

2022-11-06 05:29:51,566 INFO waiting for storagenode, processes-exit-eventlistener, storagenode-updater to die

2022-11-06T05:29:51.566Z INFO Got a signal from the OS: “terminated” {“Process”: “storagenode-updater”}

2022-11-06 05:29:51,568 INFO stopped: storagenode-updater (exit status 0)

2022-11-06 05:29:51,964 INFO stopped: storagenode (exit status 0)

2022-11-06 05:29:51,965 INFO stopped: processes-exit-eventlistener (terminated by SIGTERM)

2022-11-06 05:30:42,338 INFO Set uid to user 0 succeeded

2022-11-06 05:30:42,348 INFO RPC interface ‘supervisor’ initialized

2022-11-06 05:30:42,348 INFO supervisord started with pid 1

2022-11-06 05:30:43,350 INFO spawned: ‘processes-exit-eventlistener’ with pid 11

2022-11-06 05:30:43,351 INFO spawned: ‘storagenode’ with pid 12

2022-11-06 05:30:43,352 INFO spawned: ‘storagenode-updater’ with pid 13

2022-11-06T05:30:43.456Z INFO Configuration loaded {“Process”: “storagenode-updater”, “Location”: “/app/config/config.yaml”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.address”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.wallet-features”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “contact.external-address”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “operator.email”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “server.private-address”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-bandwidth”}

2022-11-06T05:30:43.456Z INFO Invalid configuration file key {“Process”: “storagenode-updater”, “Key”: “storage.allocated-disk-space”}

2022-11-06T05:30:43.491Z INFO Anonymized tracing enabled {“Process”: “storagenode-updater”}

2022-11-06T05:30:43.605Z INFO Running on version {“Process”: “storagenode-updater”, “Service”: “storagenode-updater”, “Version”: “v1.63.1”}

2022-11-06T05:30:43.605Z INFO Downloading versions. {“Process”: “storagenode-updater”, “Server Address”: “https://version.storj.io”}

2022-11-06T05:30:43.823Z ERROR Error retrieving version info. {“Process”: “storagenode-updater”, “error”: “version checker client: Get "https://version.storj.io": EOF”, “errorVerbose”: “version checker client: Get "https://version.storj.io": EOF\n\tstorj.io/storj/private/version/checker.(*Client).All:68\n\tmain.loopFunc:21\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tmain.cmdRun:136\n\tstorj.io/private/process.cleanup.func1.4:378\n\tstorj.io/private/process.cleanup.func1:396\n\tgithub.com/spf13/cobra.(*Command).execute:852\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:960\n\tgithub.com/spf13/cobra.(*Command).Execute:897\n\tstorj.io/private/process.ExecWithCustomConfigAndLogger:93\n\tmain.main:20\n\truntime.main:255”}

2

Delete the watchtower container in Docker, then in PowerShell:

docker pull storjlabs/watchtower

and got the following message:

Using default tag: latest
Error response from daemon: Get "https://registry-1.docker.io/v2/": EOF```
3

You have problems with your network configuration on this server, it cannot retrieve even a small file.
Try to wsl --shutdown from PowerShell, the Docker Desktop should suggest to restart the docker subsystem - agree with it, then try to pull the image.
If it doesn’t help - try to reboot.
If you do not use wsl2 in the Docker desktop, then try to restart the docker subsystem from the Docker desktop application (it’s in the debug section).

4

I’ve tried:

del the watchtower image->wsl --shutdown->Docker restart->pull the image by docker pull storjlabs/watchtower->get the same error;

del the watchtower image->reboot pc->->wsl --shutdown->Docker restart->pull the image by docker pull storjlabs/watchtower->get the same error.

the Nodes itself in online status in Node Dashboard

5

unfortunately it’s a network error

  1. Try to restart the router
  2. try to reset the network
    in the elevated PowerShell
wsl --shutdown
netsh winsock reset
netsh int ip reset all
netsh winhttp reset proxy
ipconfig /flushdns
netsh winsock reset
shutdown /r
  1. Try to reinstall the docker desktop and related VMs:
    Uninstall the docker desktop. If you used wsl2, uninstall the wsl2 distro, install wsl2 subsystem again, install the distro back. And install the docker desktop back.
1 Like
6

it’s just usual normal node activity:
piecestore download started
piecestore downloaded

The problem in watchtower itself or in dependencies of watchtower container.
It appeared after WIN 22H2 update that I’ve made yesterday.

7

The problem is much worse - it cannot download from docker hub and a small file from version.storj.io
so, nodes will be affected too, because something closing connections before finishing.

8

I have passed all 3 steps with trying to elevate the nodes after each step.
The 3rd one worked for me.
Thank you Alexey!

1 Like