Infinite restart loop after update

Node Operators troubleshooting
,
1

downloading storagenode-updater
–2024-07-30 20:03:07-- https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
Resolving … 34.173.164.90
Connecting to 34.173.164.90|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 93 [text/plain]
Saving to: ‘STDOUT’

 0K                                                       100% 81.6M=0s

2024-07-30 20:03:08 (81.6 MB/s) - written to stdout [93/93]

/tmp/storagenode-updater.zip: No such file or directory
downloading storagenode-update

2

So, what’s the exact question here?
Is it in docker?
What did you try yourself thus far?

3

The question is the loop is run forever, both conteiner is always restarting. Really no such file in /tmp

4

Did you already so the container, remove it and restart? And did you try to update the image?

5

Yes I removed it, and pull, and rerun
docker top storagenode
Error response from daemon: Container d28c03860a10751f9089aa849efb973efdaa15c9fc7abea49b56688ba99fffed is restarting, wait until the container is running

6

storj3@Microserver:~$ docker -v
Docker version 24.0.7, build 24.0.7-0ubuntu2~22.04.1

7

docker exec -it storagenode wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
Error response from daemon: Container 385a763be40f32160aef2e3651af5a0275df5a7cda39053d615ca5fd4ccc70ba is restarting, wait until the container is running

8

wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
–2024-07-30 23:43:53-- https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
Распознаётся github.com (github.com)… 140.82.121.3
Подключение к github.com (github.com)|140.82.121.3|:443… соединение установлено.
HTTP-запрос отправлен. Ожидание ответа… 302 Found
Адрес: https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240730%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240730T204353Z&X-Amz-Expires=300&X-Amz-Signature=55125fceac1aa09954790a19d23bb34b488879cd2417f02a9bf3543519e9c545&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream [переход]
–2024-07-30 23:43:53-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/128089774/7cc525c9-8b78-4199-98e6-861239e6b4f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240730%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240730T204353Z&X-Amz-Expires=300&X-Amz-Signature=55125fceac1aa09954790a19d23bb34b488879cd2417f02a9bf3543519e9c545&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=128089774&response-content-disposition=attachment%3B%20filename%3Dstoragenode-updater_linux_amd64.zip&response-content-type=application%2Foctet-stream
Распознаётся objects.githubusercontent.com (objects.githubusercontent.com)… 185.199.110.133, 185.199.109.133, 185.199.108.133, …
Подключение к objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443… соединение установлено.
HTTP-запрос отправлен. Ожидание ответа… 200 OK
Длина: 8106329 (7,7M) [application/octet-stream]
Сохранение в: ‘/tmp/storagenode-updater.zip’

/tmp/storagenode-updater.zip 100%[====================================================================>] 7,73M 19,2MB/s за 0,4s

2024-07-30 23:43:54 (19,2 MB/s) - ‘/tmp/storagenode-updater.zip’ сохранён [8106329/8106329]

storj3@Microserver:~$ unzip -t /tmp/storagenode-updater.zip
Archive: /tmp/storagenode-updater.zip
testing: storagenode-updater OK
No errors detected in compressed data of /tmp/storagenode-updater.zip.

9

How can I downgrade storj node version?
Looks like watchtower works better

10

Minimum version is 1.108.3: https://version.storj.io/

11

What tag your can advice https://hub.docker.com/r/storjlabs/storagenode/tags?

12

On github I dont found docker image. My node works fine from version 2. But ugly update will blow me out of storj

13

So more investigation while conterner does not go to restart loop:
no /tmp inside!
storj3@Microserver:~$ docker exec -it storagenode wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
Error response from daemon: Container f1b64ca3a5433a90621b3830c7158d7c35e0b8afa3a6e36237ab2cac337cb17f is restarting, wait until the container is running
storj3@Microserver:~$ docker stop storagenode
storagenode
storj3@Microserver:~$ docker start storagenode
storagenode
storj3@Microserver:~$ docker exec -it storagenode wget -O /tmp/storagenode-updater.zip https://github.com/storj/storj/releases/download/v1.52.2/storagenode-updater_linux_amd64.zip
/tmp/storagenode-updater.zip: No such file or directory
storj3@Microserver:~$ docker exec -it storagenode ls /
Error response from daemon: Container f1b64ca3a5433a90621b3830c7158d7c35e0b8afa3a6e36237ab2cac337cb17f is restarting, wait until the container is running
storj3@Microserver:~$ docker stop storagenode
storagenode
storj3@Microserver:~$ docker start storagenode
storagenode
storj3@Microserver:~$ docker exec -it storagenode ls /
app bin boot dev entrypoint etc home lib lib64 opt proc root sbin srv sys usr var

14

docker exec -it storagenode mkdir /tmp
Error response from daemon: Container f1b64ca3a5433a90621b3830c7158d7c35e0b8afa3a6e36237ab2cac337cb17f is restarting, wait until the container is running
storj3@Microserver:~$ docker stop storagenode
storagenode
storj3@Microserver:~$ docker start storagenode
storagenode
storj3@Microserver:~$ docker exec -it storagenode mkdir /tmp
mkdir: cannot create directory ‘/tmp’: Permission denied
storj3@Microserver:~$ docker exec -it storagenode sudo mkdir /tmp
OCI runtime exec failed: exec failed: unable to start container process: exec: “sudo”: executable file not found in $PATH: unknown

15

Hello @Darkt.Weider,
Welcome to the forum!

Please post a result of the command:

df --si -T

Also try to prune all docker stuff:

docker stop storagenode
docker rm storagenode
docker system prune

After that please try to run the container again.

16

and to add, whenever you use ‘sudo’ it has to go at the front of the command, never inbetween.

1 Like
17

storj3@Microserver:~$ df --si -T
Файл.система Тип Размер Использовано Дост Использовано% Cмонтировано в
tmpfs tmpfs 1,1G 3,1M 1,1G 1% /run
/dev/sda4 ext4 53G 13G 38G 25% /
tmpfs tmpfs 5,2G 0 5,2G 0% /dev/shm
tmpfs tmpfs 5,3M 0 5,3M 0% /run/lock
/dev/sdd1 ext4 2,0T 1,6T 481G 76% /data/klk15
/dev/sdc1 ext4 4,0T 3,5T 449G 89% /data/wd2
/dev/sde1 ext4 4,0T 3,8T 217G 95% /data/wd4
/dev/sdf1 ext4 4,0T 3,3T 666G 84% /data/wd42
/dev/sda2 ext4 474M 269M 170M 62% /boot
/dev/sda6 ext4 154G 15G 131G 11% /home
/dev/sda5 ext4 21G 5,2G 15G 27% /var
tmpfs tmpfs 1,1G 0 1,1G 0% /run/user/1000

After prune of 3,3U, of old conteiners, I pull and run latest
But:
downloading storagenode-updater
–2024-07-31 08:40:08-- https://version.storj.io/processes/storagenode-updater/minimum/url?os=linux&arch=amd64
Resolving version.storj.io (version.storj.io)… 34.173.164.90
Connecting to version.storj.io (version.storj.io)|34.173.164.90|:443… connected.
HTTP request sent, awaiting response… 200 OK
Length: 93 [text/plain]
Saving to: ‘STDOUT’

 0K                                                       100% 86.5M=0s

2024-07-31 08:40:09 (86.5 MB/s) - written to stdout [93/93]

/tmp/storagenode-updater.zip: No such file or directory
downloading storagenode-updater

And no /tmp directory inside image:
storj3@Microserver:~$ docker stop storagenode
storagenode
storj3@Microserver:~$ docker start storagenode
storagenode
storj3@Microserver:~$ docker exec -it storagenode ls /
app bin boot dev entrypoint etc home lib lib64 opt proc root sbin srv sys usr var

но /tmp here

18

This is very weird, I cannot reproduce that on my arm64.
Please try to add --mount type=tmpfs,destination=/tmp to your docker run command before the image name.

Or map your host’s /tmp there: --mount type=bind,source=/tmp,destination=/tmp, or you may also use a docker volume for that: -v tmp:/tmp

1 Like
19

A lot of thanks! This helps.

1 Like
20

@Darkt.Weider

Can you try to create a container from another official image (e.g. docker run --rm -ti debian:bookworm /bin/bash)?

And then execute ls -l / and df -h and paste the outputs here?

1 Like