Docker update v1.71.2 - auto stopped docker and didnt restart

Node Operators troubleshooting
, ,
1

I run Storj in a Docker on Unraid, all works fine

I just had a uptime monitor alert to say the node has gone offline, I checked docker and sure enough the docker was off, I manually restarted it an noticed it has updated to v1.71.2.

What normally happens to other dockers is it states there is an update so i can manually do this, but for Storj this isn’t the case, it seems to auto update on its own accord but shuts the docker down and doesn’t restart it.

This not ideal as if it happened at say 12am when i am sleeping i would not know it was offline until the morning which would mean 7 hrs of downtime :confused:

How do i make sure the Storj docker doesn’t auto update like this? or fix it so it auto restarts?

Thanks

2

Make sure you have --restart unless-stopped in your run command. You can’t turn off auto update (and shouldn’t) as that runs automatically inside the docker container.

3 Likes
3

That’s not the cause. I had happening the same on one node after it got updated to v1.71.2.
Others restarted fine after update, this one did not.

1 Like
4

Right now, in Unraid, i have in the Extra Paremeters bit: “–mount type=bind,source=”/mnt/disks/Storj_Node_1/storj-certs/identity/storagenode/“,destination=/app/identity --mount type=bind,source=”/mnt/disks/Storj_Node_1/storj/“,destination=/app/config”

shall I add --restart unless-stopped at the end of this?

EDIT:
I also may add this happened on the previous update too for me, i thought it was a one off, until this one didnt auto start as well.

Is there anyway we can get a heads up on updates? so we can be ready for it?

5

Hi Antec1234,

Well, LittleSkunk will post up the release notes just before release. However, releases are staggered so you might not get it right away when it first goes out, and may get it sometime later in the week depending on where you fall in the release schedule.

You may want to investigate a way to monitor whether your nodes are online or not and receive notification when they go offline.

6

I had this happen last night, two nodes updated to 1.71.2 at 4:30am and didnt restart. Just noticed and restarted them now. Not sure how monitoring helps in the middle of the night. This doesnt happen all the time so it cant be a docker config issue, right? All my nodes are restart unless-stopped.

1 Like
7

What platform do you use docker in?

8

Ubuntu 22.04.1 LTS. I can provide other info if that helps. Its happened before, its not all nodes, not just these nodes, and not every update.

1 Like
9

hopefully we get to the bottom of it before the next update!

do you have Uptime robot setup for your nodes? it will email you if the node goes down

10

I have pushbullet notifications, but i dont think the solution is wake up at 4:30am to restart a docker container.

3 Likes
11

SNOs never sleep! :smile: :grin:

12

That may be true, but i do take a nap from 4:20-4:47. Gotta got my 27 mins.

2 Likes
13

Today, it happened on my node, too.

I always had this in my command:

sudo docker run -d --restart unless-stopped …

But it still died around 4am. I restarted it in the morning.

Before restart, I looked at the log of died container and it had this line:

ERROR Error updating service. {“Process”: “storagenode-updater”, 
“Service”: “storagenode-updater”, 
“error”: “context canceled”, “errorVerbose”: “context canceled
	main.downloadBinary:58
	main.update:39
	main.loopFunc:32
	storj.io/common/sync2.(*Cycle).Run:160
	main.cmdRun:136
	storj.io/private/process.cleanup.func1.4:377
	storj.io/private/process.cleanup.func1:395
	github.com/spf13/cobra.(*Command).execute:852
	github.com/spf13/cobra.(*Command).ExecuteC:960
	github.com/spf13/cobra.(*Command).Execute:897
	storj.io/private/process.ExecWithCustomConfigAndLogger:92
	main.main:20
	runtime.main:250”}

Perhaps, node can be configured not to upgrade during the night, so in case if anything goes wrong - operator has higher chance to be awake and fix.

2 Likes
14

To those effected, does the container actually stop or just the storage node service inside the container?

15

For me in Unraid the container stopped, i panicked and didnt want downtime so didnt check the logs before restarting it, maybe if it happens again, i can check the logs for support.

all that i needed to was start the docker container for Storj and it started up ok, think i lost about 1 hour :confused:

16

So just a docker start command? You didn’t have to stop it first?

17

nope, it was already stopped, i just had to start it. is there anyway i can check logs? where are they stored?

18

If you did not redirect them to the file, they are lost when the container is recreated.
However, if you did not remove the container and just started, then they probably should be still there

docker logs storagenode

you may use option --since to provide a start date and time, see

docker logs --help

And stopped container means that --restart policy is not specified, so you need to add --restart unless-stopped to your docker run command. I can guess, but you likely uses the Community storagenode application for UnRaid and looks like it doesn’t include this parameter.

1 Like
19

It was stopped, but not removed.

Because of that I was able to look at logs.

But before I was able to start, I needed to remove.

20

It happened few times also on my nodes, first time I saw that on version ~1.68. Not sure what triggers this. I am now trying “–restart always” policy, but I am not sure about disadvantages, so I don’t recommend.
From docker docs it looks like, restart policy is reset, when you do manual “docker stop container” and then manual docker start.