Container restarting

Node Operators troubleshooting
1

My container is restarting constantly. I am getting errors in the logs about disk space. ( xxxxx-d out some personal data.) This ran fine for months. I did have a power issue and then internet outage for a few days. Any help would be appreciated. Getting an error when trying to post the logs and details about 2 links for a new user.

2

My startup command

sudo docker run -d --restart unless-stopped -p 28967:28967
-p 127.0.0.1:14002:14002
-e WALLET=“xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”
-e EMAIL=“xxxxxxxxxxxxx.com”
-e ADDRESS=“xxxxx,xxxxxx.com:28967”
-e BANDWIDTH=“100TB”
-e STORAGE=“1.35TB”
–mount type=bind,source="/home/xxxxxxxx/.local/share/storj/identity/storagenode",destination=/app/identity
–mount type=bind,source="/storj/Node",destination=/app/config
–name storagenode storjlabs/storagenode:beta

3

errors in logs

2020-01-17T03:26:05.535Z WARN piecestore:monitor Disk space is less than requested. Allocating space {“bytes”: 466939491328}
2020-01-17T03:26:05.535Z ERROR piecestore:monitor Total disk space less than required minimum {“bytes”: 500000000000}

4

I would make sure you drive is mounted correctly, Which os are you running?

5

sudo docker inspect storagenode -f ‘{{ .Mounts }}’
[{volume 048f20bf188377d7e58640e9cee43d81cb09e79c28f3aeb3f79ab8d566eb56f2 /var/lib/docker/volumes/048f20bf188377d7e58640e9cee43d81cb09e79c28f3aeb3f79ab8d566eb56f2/_data /root/.local/share/storj/storagenode local true } {bind /home/xxxxxxxxx/.local/share/storj/identity/storagenode /app/identity true rprivate} {bind /storj/Node /app/config true rprivate}]

6

$ df -h /storj/Node
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/storj2–vg-storj 1.4T 1.1T 209G 84% /storj/Node

7

I can view the data on the drive ok. I am running Ubuntu 18.04

$ ls -la /storj/Node
total 312
drwxr-xr-x 4 root root 4096 Jan 16 23:04 .
drwxr-xr-x 5 xxxxxxxx xxxxxxxx 4096 Mar 4 2019 …
-rw-r–r-- 1 root root 636 Mar 4 2019 config.yaml
-rw------- 1 root root 262144 Oct 3 15:23 kademlia
drwx------ 2 root root 16384 Mar 4 2019 lost+found
-rw------- 1 root root 32768 Jan 16 23:04 revocations.db
drwx------ 7 root root 69632 Jan 16 23:04 storage
-rw------- 1 root root 981 Jan 16 23:04 trust-cache.json

8

How much free space does it have?

9

209GB free, the only data on this entire drive is for my storagenode

10

Ok what you can try to do is edit the config
If it was just a warn you would be fine but your also looks like your also getting an error.

#how much disk space a node at minimum has to advertise
storage2.monitor.minimum-disk-space: 500.0 GB

Which version are you running 29.3?

11

I had this problem once. My piece_spaced_used.db file was blank and was telling my node that it had zero stored bytes of data on my node which was messing up its internal math, resulting in that error message. you can use the sqlite3 program to check your database to see if yours is doing the same thing. I’m posting a screen shot of the commands I used. This database query should return total space used by all pieces stored on disk. in my example it returned 488982176000 (bytes). If it returns zero, that could be the problem. I had a recent backup copy of that DB file that I was able to replace the bad one with. If you don’t have a backup, I’m wondering if it would be possible to manually input the number into the database yourself, assuming your issue was similar to mine. if you try this, I’d backup your DB files first, in case things get worse.

spaceused

1 Like
12

Looks ok to me

root@xxxxxxx:/storj/Node/storage# sqlite3 piece_spaced_used.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter “.help” for usage hints.
sqlite> select total from piece_space_used where satellite_id is NULL;
242754307072
sqlite>

13

This is the fatal error, still related to fee space. I think you are on the right track. I have lots of used and some free space, but for some reason it is not calculating correctly something.

2020-01-17T14:14:30.271Z FATAL Unrecoverable error {“error”: “piecestore monitor: disk space requirement not met”, “errorVerbose”: "piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.

14

I would hate to wipe my data and start over, but I am getting close to that point as I have been down for a while now and unable to find a fix.

15

Have you tried to edit the config file?

16

Edit how? The config file does not match my startup commands, Is this the right file you are referring to?

root@xxxxxx:/storj/Node# cat config.yaml

path to the certificate chain for this identity

identity.cert-path: “identity/identity.cert”

path to the private key for this identity

identity.key-path: “identity/identity.key”

the public address of the Kademlia node, useful for nodes behind NAT

kademlia.external-address: “”

operator email address

kademlia.operator.email: “”

operator wallet adress

kademlia.operator.wallet: “”

the minimum log level to log

log.level: info

address to listen on

server.address: “:28967”

total allocated bandwidth in bytes

storage.allocated-bandwidth: 500.0 GiB

total allocated disk space in bytes

storage.allocated-disk-space: 1.0 TB

17

Yes that is the one, Find

storage2.monitor.minimum-disk-space: 500

18

How much space on your drive is your storage node folder actually using? Also how much free space is left on your drive?

19

That is not in my config file. I added it, and the container started and stayed up, but it looks like it is not seeing any of my existing data that was stored.

20

I am thinking it is best to just wipe and start over unless anyone has some ideas to fix me.