New Node Crashing

Node Operators troubleshooting
,
1

Hey,

I recently did setup a storage node, based on your guide.
When I start the node with docker, it keeps restarting, according to the log, it keeps crashing:

https://pastebin.com/raw/xUBq5Jzi

Apparently this bug is know since over 1 Year.
Anyone an idea on this and why is it not fixed?

storj-v3-storage-node-service-not-starting/7803?page=2
tech-preview-storage-node-multinode-dashboard/11749?page=4

I could not find any solution on this.
Would be nice if someone could help.

Thanks!

edit: I can’t really but any links in here because the forum is blocking me.

2

Hello @Ne00n ,
Welcome to the forum!

You can edit your post to answer on these questions:

  1. OS?
  2. Architecture
    for Linux
uname -a

for Windows

systeminfo.exe
  1. How many RAM?
    for Linux
free

for Windows (from cmd.exe)

wmic MEMORYCHIP get BankLabel, DeviceLocator, Capacity, Speed

or systeminfo.exe

  1. Type of the storagenode - binary or docker?
  2. Did you compile it yourself or have used a released version?
1 Like
3
  1. Debian 11 64bit
  2. x86
  3. 2GB
  4. Docker I followed your guide: Storage Node - Node Operator
  5. Docker, so already compiled

I allocated the System Disk 2.7TB, I could not find any detailed info, so I gave the system 2GB of Memory. However, according to htop/top at no point the system did hit the limit.

4

Please, give me result of the command:

docker info

Please restart the docker service, remove the image and pull it again, then try to start:

sudo service docker restart

docker rmi storjlabs/storagenode:latest

docker pull storjlabs/storagenode:latest

And run the container: Storage Node - Node Operator

5

Well, I wiped it, since I found something oddly wired.
I installed docker and create a user to run storj with.

Gave the user permission to the docker socket, right.
However, the storj docker file, creates files, especially the config files with root privileges.

I only noticed it when I wanted to recreate the container, that the user could not access the files.
So I checked the permission on these and they where owned by root.

So I will try it again without any sub user.
I suspect something is broken with the storj docker file.

6

I did setup everything again, under root.
Same issue: https://pastebin.com/raw/MU7nWw0S

@Alexey
docker info:
https://pastebin.com/raw/QTNLAn74

The other stuff:
https://pastebin.com/raw/8i38857M

Still same issue, lemme know if you need more.

7

Can you replace Debian to Ubuntu?
I have different troubles with the latest Debian, and on my other project we migrated all our k8s clusters to Ubuntu because of kernel bug, no problems so far.
The problems started since Debian 10. We did not have any issues before.

8

I get the same error on Ubuntu 20.04.

9

Then it’s likely related to the hardware.
Please try to use a binary: [Tech Preview] Linux Storage Node & Updater or [Tech Preview] Linux Installer Part 2

10

I am using a KVM here, unlikely.
Maybe some dependencies missing?

11

The docker should just work. Everything is working in the container can have an own OS and all dependencies usually included into the container. The only thing it uses - the kernel from the host OS.
So, very likely the problem is in the kernel.
Since you switched to Ubuntu - it has an own kernel and it’s likely more fresh than on Debian.
If you have the same issue even on Ubuntu - I suspect the hardware issues.
You could try to use a VM though.

12

With KVM I mean a VM, the entire time I am using a VM.
But I did enable Host passthrough.

13

Please, disable it or likely switch to default. The docker doesn’t use a virtualization, so the cpu features should not be too important.
If it’s a VM anyway, then you can use a binary version.

By the way, please post result of docker info

14

Disabling CPU Passthrough is not fixing it.
Something else is broken in the Image.

Docker:
https://pastebin.com/raw/4ETzDcJ0

15

Then please remove all images and pull the latest one.
The image is working on thousands nodes across the globe, so it must be something local.

16

It keeps crashing, makes no difference.

17

Can you copy the first 10 lines from the log after start?

18

Yea sure.
https://pastebin.com/raw/e6eguW3R

I don’t use docker, so No idea how docker gives out the log file but still.

19

-e STORAGE="primary" must be -e STORAGE="2TB" or similar

20

You must be kidding me.
Please do input validation and prevent these issues.

That shit cost me an arm and a leg, but the node seems to be online now.
Let see what it does return.