Storagenode restarting every minute

Node Operators troubleshooting
1

Setting up a new node on an Ubuntu VM with lots of resources. However, my storagenode is constantly restarting every minute or so. Did clean up and reinstall the docker with no luck.

The docker is sending out IPv6 broadcasts to DST=ff02:0000:0000:0000:0000:0000:0000:0016 and ff02:0000:0000:0000:0000:0000:0000:00fb port 5353. These are currently being blocked in the firewall. Is this block causing this issue?

Ideas?

$ docker exec -it storagenode /app/dashboard.sh
Error response from daemon: Container is restarting, wait until the container is running

2

Try to disable IPv6, if you using Ubuntu 18.04:

sudo vi /etc/default/grub

Modify the GRUB_CMDLINEs to look like:

GRUB_CMDLINE_LINUX_DEFAULT=“ipv6.disable=1”
GRUB_CMDLINE_LINUX=“ipv6.disable=1”

Then execute:

sudo update-grub
sudo reboot

Enjoy ipv4.

3

Please, copy the output of the docker logs --tail 20 storagenode command

4
2019-06-28T05:23:08.251Z INFO Configuration loaded from: /app/config/config.yaml
2019-06-28T05:23:08.253Z FATAL file or directory not found error: open identity/identity.key: no such file or directory

These entries seem to be a few hours old.

5

Which OS you have used?
Please, copy your docker run command here (you can redact the private info)

6

lsb_release -a on my VM (running the storage node):
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

docker run -d --restart unless-stopped -p 28967:28967
-e WALLET=“0x…”
-e EMAIL=“x@x.com”
-e ADDRESS=“my-public-static-IP:28967"
-e BANDWIDTH=“20TB”
-e STORAGE=“3TB”
–mount type=bind,source="/home/storj/.local/share/storj/identity/",destination=/app/identity
–mount type=bind,source="/home/storj/storj-data/",destination=/app/config
–name storagenode storjlabs/storagenode:alpha

7

Running Ubuntu 16.04.6. Just disabled IPv6 without luck.

8

For 16.04 I have another steps:

Add this lines to the end of /etc/sysctl.conf

#Disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

then run sysctl -p

and check that ipv6 was gone with ifconfig

9

Tried that one as well. Just checked ifconfig and confirmed that no IPv6 is in there.

10

[UFW BLOCK] IN= OUT=docker0 SRC=172.17.0.1 DST=224.0.0.251 LEN=92 TOS=0x00 PREC=0x00 TTL=255 ID=56140 DF PROTO=UDP SPT=5353 DPT=5353 LEN=72

11

Please, copy the output of the command: df -h
Please, stop and remove the container:
docker stop storagenode
docker rm storagenode
Please, change the path to your identity from /home/storj/.local/share/storj/identity to /home/storj/.local/share/storj/identity/storagenode and run the container with all your parameters, include changed one

12

This is not needed. The problem on the surface, and it’s not in the IPv6

13

I see the that root cause alredy found, but this is best practice, and can help understand it:

before:

The docker is sending out IPv6 broadcasts to DST=ff02:0000:0000:0000:0000:0000:0000:0016 and ff02:0000:0000:0000:0000:0000:0000:00fb port 5353.

after:

[UFW BLOCK] IN= OUT= docker0 SRC=172.17.0.1 DST=224.0.0.251 LEN=92 TOS=0x00 PREC=0x00 TTL=255 ID=56140 DF PROTO=UDP SPT=5353 DPT=5353 LEN=72

now it clear.

14

Works like a charm and with IPv6 enabled. “storagenode” was missing from the identity path.
Thank you both.

4 Likes