Failed to load identity - node keeps restarting

Node Operators troubleshooting
,
1

Hey everybody, my holiday nightmare finally happened. Just got back to check how it was the node and found out it was loop-restarting again.

  1. editing the config.yaml file info TO debug
  2. running a docker logs storagenode 2>&1 | grep -B 3 -i fatal
  3. It gives me this repeated message (actually i got it also from a simple docker logs)
  • failed to load identity "identity/identity.cert", "identity/identity.key": cert chain length error: identity chain does not contain a CA certificate

(My identity folder is located as usual into .local/share/storj/identity/storagenode )

  1. Using a docker ps -a
    it gives out two running services as following:
    c16214a8e348 storjlabs/storagenode:arm "/entrypoint" 9 minutes ago Restarting (1) 9 seconds ago storagenode
    8c30e1e5340a storjlabs/watchtower:latest-arm32v6 "/watchtower storage…" 2 months ago Up 27 hours watchtower

  2. If I try to run the dashboard.sh it says
    Error response from daemon: Container c16214a8e3484c41998eeaeda58c339b04746369aa1eb2d45c2b194dc5cabff7 is restarting, wait until the container is running
    [UPDATE]Also if I try to execute dashboard right after docker run it gives me this very unknown:
    OCI runtime exec failed: exec failed: container_linux.go:338: creating new parent process caused "container_linux.go:1897: running lstat on namespace path \"/proc/2600/ns/ipc\" caused \"lstat /proc/2600/ns/ipc: no such file or directory\"": unknown

my identity folder is all good, nothing different from before. I don’t really want to restart again, it has been 3 months now and July was a very good month.
I can get we are in beta, I can accept everything but it’s very odd that a SNO update every single time put my node in this situation…
Please help!

2

Please, post your docker run command

Also, please, check your autoupdate command, it has been changed: https://documentation.storj.io/setup/software-updates#automatic-updates

3

sudo docker run -d --restart unless-stopped -p 28967:28967
-e WALLET=“MYWALLETADDRESS”
-e EMAIL="aaa@mail.com"
-e ADDRESS="(MYIPNUMBER):28967"
-e BANDWIDTH=“100TB”
-e STORAGE=“1.8TB”
–mount type=bind,source="/home/pi/.local/share/storj/identity/storagenode",destination=/app/identity
–mount type=bind,source="/mnt/storagenode",destination=/app/config
–name storagenode storjlabs/storagenode:arm

Also I checked the watchtower and resubmitted it too… still same errors when running storagenode

4

Please, copy the output of the commands:

df -h
ls -l /home/pi/.local/share/storj/identity/storagenode
5

Hey there,
i had the same problem so i can submit how i solved my Problem:
My Identity was on the “/home/USERNAME/.local/…” Dir and my node keep restarting and restarting …
I copied my ID-files into “/storj/id/” and set the “” param. to “/storj/id”.
After that i do “docker stop storagenode” then “docker rm storagenode” and after all i re-runed the install script with the new ID-Dir.
After That it worked just fine

To create a dir on / use your terminal and type “mkdir /YOUR-DIR-Name” and then “chown YOUR_USERNAME YOUR-DIR-Name”
after that press F5 and wolla

Tip:

Look at permissions of your Files and set also the Dir-Perm.

1 Like
6

Hello @abc,
Welcome to the forum!

Thanks for the tip!

7

Hi

how are you?

1.- my error is: docker logs storagenode

2020-05-09T19:25:37.594Z INFO Configuration loaded {“Location”: “/app/config/config.yaml”}
2020-05-09T19:25:37.594Z INFO tracing disabled
2020-05-09T19:25:37.595Z FATAL Failed to load identity. {“error”: “file or directory not found error: open identity/identity.cert: not a directory”, “errorVerbose”: “file or directory not found error: open identity/identity.cert: not a directory\n\tstorj.io/common/identity.Config.Load:329\n\tmain.cmdRun:146\n\tstorj.io/private/process.cleanup.func1.4:344\n\tstorj.io/private/process.cleanup.func1:362\n\tgithub.com/spf13/cobra.(*Command).execute:840\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:945\n\tgithub.com/spf13/cobra.(*Command).Execute:885\n\tstorj.io/private/process.ExecWithCustomConfig:86\n\tstorj.io/private/process.ExecCustomDebug:68\n\tmain.main:329\n\truntime.main:203”}

2.- my output show always restarting

root@docker-s-1vcpu-1gb-nyc1-01:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7db1fa01f124 storjlabs/storagenode:beta “/entrypoint” 6 minutes ago Restarting (1) 46 seconds ago storagenode
87f88c6e1b01 hello-world “/hello” About an hour ago Exited (0) About an hour ago laughing_hawking
9400c9a10ff6 storjlabs/watchtower “/watchtower storage…” About an hour ago Up About an hour watchtower
3ccc5fec7c85 ubuntu “bash” 2 hours ago Exited (127) 2 hours ago hardcore_noyce
570ca9b043aa hello-world “/hello” 2 hours ago Exited (0) 2 hours ago cranky_minsky

3.- i was try delete with out:

docker stop -t 300 storagenode
docker rm storagenode
rm /mnt/volume_nyc1_03/config.yaml

4.- after remove o was try run again the following:

docker run -d --restart unless-stopped --stop-timeout 300
-p 28967:28967
-p 127.0.0.1:14002:14002
-e WALLET=“0x630a8F9EDA472C55b9057e706F7A8f427e9ab48B”
-e EMAIL="mmyname@mydomain.com"
-e ADDRESS=“162.243.163.153:28967”
-e STORAGE=“550GB”
–mount type=bind,source="/usr/local/bin/identity",destination=/app/identity
–mount type=bind,source="/mnt/volume_nyc1_03",destination=/app/config
–name storagenode storjlabs/storagenode:beta

5.- i was verify my identity with:

root@docker-s-1vcpu-1gb-nyc1-01:~# grep -c BEGIN ~/.local/share/storj/identity/storagenode/ca.cert
2
root@docker-s-1vcpu-1gb-nyc1-01:~# grep -c BEGIN ~/.local/share/storj/identity/storagenode/identity.cert
3

6.- my dashboard show

root@docker-s-1vcpu-1gb-nyc1-01:~# docker exec -it storagenode /app/dashboard.sh
Error response from daemon: Container 7db1fa01f124dc9a87da7598614fdc00169f5e35a1c48db529085c1a5588698c is restarting, wait until the container is running

7.- my dh-f and ls /usr/local/bin/identity

root@docker-s-1vcpu-1gb-nyc1-01:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 463M 0 463M 0% /dev
tmpfs 99M 748K 98M 1% /run
/dev/vda1 25G 2.8G 22G 12% /
tmpfs 493M 0 493M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 493M 0 493M 0% /sys/fs/cgroup
/dev/vda15 105M 3.6M 101M 4% /boot/efi
/dev/sda 546G 68M 524G 1% /mnt/volume_nyc1_03
tmpfs 99M 0 99M 0% /run/user/0
overlay 25G 2.8G 22G 12% /var/lib/docker/overlay2/983da027c4c33d00fa9efdb0fe67de214c83c56c8d55c563bdcc163d6593c0c4/merged

root@docker-s-1vcpu-1gb-nyc1-01:~# ls /usr/local/bin/identity
/usr/local/bin/identity

8.- i was try copy identity to /storj my output:

root@docker-s-1vcpu-1gb-nyc1-01:~# ls /storj/
identity

i was remove:

docker stop -t 300 storagenode
docker rm storagenode
rm /mnt/volume_nyc1_03/config.yaml

and my scrip:

docker run -d --restart unless-stopped --stop-timeout 300
-p 28967:28967
-p 127.0.0.1:14002:14002
-e WALLET=“0x630a8F9EDA472C55b9057e706F7A8f427e9ab48B”
-e EMAIL=“mcastro@securebyte.com.ve”
-e ADDRESS=“162.243.163.153:28967”
-e STORAGE=“550GB”
–mount type=bind,source="/storj",destination=/app/identity
–mount type=bind,source="/mnt/volume_nyc1_03",destination=/app/config
–name storagenode storjlabs/storagenode:beta

my logs is the same

8

The identity is not the “identity” binary you used for creating and signing the identity. It is the folder ~/.local/share/storj/identity/storagenode which contains 6 files.

9

Hi

thanks for your support, yes, the problem solvent when i put in my scrip:

-mount type=bind,source=“/root/.local/share/storj/identity/storagenode/”,destination=/app/identity \

regards,

10

Hi guys! My terminal windows in docker keeps spitting out this same sequence of code over and over again in an attempt to restart my node. I was wondering if anyone has a solution to this issue. Again, this is the only chunk of code docker is giving me repeatedly. Thank you all for your time and suggestions!
Here is the code:

(*Command).execute:840\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:945\n\tgithub.com/spf13/cobra.(*Command).Execute:885\n\tstorj.io/private/process.ExecWithCustomConfig:86\n\tstorj.io/private/process.ExecCustomDebug:68\n\tmain.main:329\n\truntime.main:203"}

2020-05-10T21:41:09.198Z INFO Configuration loaded {"Location": "/app/config/config.yaml"}

2020-05-10T21:41:09.199Z INFO tracing disabled

2020-05-10T21:41:09.200Z FATAL Failed to load identity. {"error": "file or directory not found error: open identity/identity.cert: no such file or directory", "errorVerbose": "file or directory not found error: open identity/identity.cert: no such file or directory\n\tstorj.io/common/identity.Config.Load:329\n\tmain.cmdRun:146\n\tstorj.io/private/process.cleanup.func1.4:344\n\tstorj.io/private/process.cleanup.func1:362\n\tgithub.com/spf13/cobra.

11
2 Likes
12

Thanks so much, this worked!