After storenode reboot my displayed space is decrese from 1.2 TB to 16GB why?

Node Operators troubleshooting
1

Hello,

today I have reboot my storenode in this way:

docker stop storenode
docker rm storenode
reboot the machine

after the reboot I have start again the storenode correctly but checking the dashboard with:

docker exec -it storagenode /app/dashboard.sh

the used space is change from 1.16 TB to 16.7 GB:

Storage Node Dashboard ( Node Version: v0.26.2 )

======================

ID 12hpvUSVRDAhujRZAq1fSjq2woz2sh3Gns9vYP7f56ukypCVYAN
Last Contact ONLINE
Uptime 15m11s

                          Available           Used       Egress      Ingress
 Bandwidth      359.6 TB     398.0 GB     172.6 GB     225.4 GB (since Dec 1)
      Disk                 4.0 TB      16.7 GB

Internal 127.0.0.1:7778
External bstorj.ddns.net:28967

looking the disk usage it seems that all the data are still present :slight_smile:

fabrizio@ubuntu:/mnt/storage$ df -h /mnt/
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 4.0T 1.2T 2.6T 31% /mnt

what’s happen ?

what I can do ?

thx

2

Hi @brizio71
Please double check that your storage is mounted with mount or lsblk command

3

this is the output:

fabrizio@ubuntu:/mnt/storage$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 4T 0 disk
└─sda1 8:1 0 4T 0 part /mnt
sr0 11:0 1 1024M 0 rom
vda 252:0 0 30G 0 disk
└─vda1 252:1 0 30G 0 part /

4

how much data you have on /mnt ?

5

What’s your docker run command?
Stop your node with 16GB as it might result in a lot of failed audits.

6

/mnt have 1.2TB used

7

docker run -d --restart unless-stopped -p 28967:28967 -e WALLET=“0xB8f8a2A65916b1442B824833Cfd6101910f8Ab7b” -e EMAIL="mymail@gmail.com" -e ADDRESS=“bstorj.ddns.net:28967” -e BANDWIDTH=“360TB” -e STORAGE=“4TB” --mount type=bind,source="/mnt/certs",destination=/app/identity --mount type=bind,source="/mnt",destination=/app/config --name storagenode storjlabs/storagenode:beta

8

that actually looks correct. I don’t know why it would start with showing 16GB used.

9

Are those correct paths?

10

yes are rights, storenode can not start with wrong path

11

Check your logs docker logs --tail 20 storagenode

12

it seems that all is correct:

fabrizio@ubuntu : /mnt/storage $ docker logs --tail 20 storagenode

2019-12-09T10:55:39.690Z INFO piecestore uploaded {“Piece ID”: “UON6KTE4OCPQZO5WECRWOM4Y22FHNBL52ORJEIYHA6EZT33T5SYA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “PUT”}

2019-12-09T10:55:39.760Z INFO piecestore upload started {“Piece ID”: “5TSNGPMIFYJ7MTG65PDNXKK45Y3CQM64AOHTD32GZTTDP4JGH2OQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}

2019-12-09T10:55:40.248Z INFO piecestore uploaded {“Piece ID”: “5TSNGPMIFYJ7MTG65PDNXKK45Y3CQM64AOHTD32GZTTDP4JGH2OQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}

2019-12-09T10:55:41.389Z INFO piecestore uploaded {“Piece ID”: “WTGDD35UPO4RY2YDTBCSHJZ7CIUNJ3RGUHGYCJ5UGL765R6GLOZA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “PUT”}

2019-12-09T10:55:42.247Z INFO piecestore download started {“Piece ID”: “VJRR4IQQ77ZWVKQNAIKHJOAOASU4PJSXI52VFQHD5TGPXX57AMTA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}

2019-12-09T10:55:42.285Z INFO piecestore downloaded {“Piece ID”: “LFSNNSYKKREVRK7QSARJEFRGXTKZEGBV4Y7U6762MOFOQEIT6YKQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}

2019-12-09T10:55:43.228Z INFO piecestore download started {“Piece ID”: “VRO5FHOW4QMHP2FDHMIBQ5IQNWS7BRD7WYD3PKMTQE53JKKNUJDA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}

2019-12-09T10:55:43.970Z INFO piecestore uploaded {“Piece ID”: “LISYGGMVWAWE5CNT5Y76I6F24542S23J7KLDYKFDIE2ATLM7EXDQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “PUT”}

2019-12-09T10:55:46.204Z INFO piecestore upload started {“Piece ID”: “UPG4KXSCALVMX5SLIZVWTF54PO6KSTHA5T2DRNUBFDENUJNKANXQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}

2019-12-09T10:55:47.022Z INFO piecestore downloaded {“Piece ID”: “VJRR4IQQ77ZWVKQNAIKHJOAOASU4PJSXI52VFQHD5TGPXX57AMTA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}

2019-12-09T10:55:48.271Z INFO piecestore downloaded {“Piece ID”: “VRO5FHOW4QMHP2FDHMIBQ5IQNWS7BRD7WYD3PKMTQE53JKKNUJDA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}

2019-12-09T10:55:49.468Z INFO piecestore download started {“Piece ID”: “RTHAPYRGLVLFFI6O6ONPMR5YBCV2QI3CPSL4SKBJ6SCA7GB7WWCQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET”}

2019-12-09T10:55:49.863Z INFO piecestore uploaded {“Piece ID”: “UPG4KXSCALVMX5SLIZVWTF54PO6KSTHA5T2DRNUBFDENUJNKANXQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}

2019-12-09T10:55:51.406Z INFO piecestore uploaded {“Piece ID”: “H3LPXA2M25CF7N6ATDZJQ3HCYRHU5W4CYROFFX6XRTDYLGLQ5T2Q”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “PUT”}

2019-12-09T10:55:51.920Z INFO piecestore upload started {“Piece ID”: “Z3X7IGPZRIKOXMC2VIM5PZNSRIJMGF6MGRJJSUXYJJXJ72R2U5EQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}

2019-12-09T10:55:51.931Z INFO piecestore uploaded {“Piece ID”: “Z3X7IGPZRIKOXMC2VIM5PZNSRIJMGF6MGRJJSUXYJJXJ72R2U5EQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}

2019-12-09T10:55:52.179Z INFO piecestore upload started {“Piece ID”: “2KMA4VZY6YPJVBG2SVTWH7KZ7BMED4CFEENTGSZDLS7FDOFVRR2A”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}

2019-12-09T10:55:52.210Z INFO piecestore uploaded {“Piece ID”: “2KMA4VZY6YPJVBG2SVTWH7KZ7BMED4CFEENTGSZDLS7FDOFVRR2A”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”}

2019-12-09T10:55:53.563Z INFO piecestore upload started {“Piece ID”: “YSOQWHUQBGQYRWLDZYPKCR7ZZ7RTFTXXJ24KN7KSWQQRYGVFHHAA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}

2019-12-09T10:55:54.084Z INFO piecestore upload started {“Piece ID”: “7TIUF2IBZY2D4MNWK74MGYE4HS7RTFMKMNGUFJQQJF7BKBOC7EUA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “PUT”}

13

I see you have the identity directory inside the data directory. I am not sure if that causes issues but I’d check just to be sure

14

I didn’t change any config since beginning, all was working fine until the reboot

15

please check that your 6 identity files are really in the identity folder you specified, and not in a subfolder or somewhere else

Also make sure you did not actually use curly quotes anywhere

16

Yes the 6 identity files are there:

fabrizio@ubuntu : ~/certs $ ll

total 32

drwxrwxr-x 2 fabrizio fabrizio 4096 Jun 20 17:26 . /

drwxr-xr-x 10 fabrizio fabrizio 4096 Dec 8 10:45 … /

-rw-rw-r-- 1 fabrizio fabrizio 546 Jun 20 17:26 ca.1561026062.cert

-rw-rw-r-- 1 fabrizio fabrizio 1076 Jun 20 17:26 ca.cert

-rw-rw-r-- 1 fabrizio fabrizio 241 Jun 20 17:26 ca.key

-rw-rw-r-- 1 fabrizio fabrizio 1084 Jun 20 17:26 identity.1561026062.cert

-rw-rw-r-- 1 fabrizio fabrizio 1614 Jun 20 17:26 identity.cert

-rw-rw-r-- 1 fabrizio fabrizio 241 Jun 20 17:26 identity.key

17

I suspect the hdd got mounted after the storagenode started. Can you post the line from /etc/fstab where you mount the hdd?

Can you stop the node, unmount the hdd and post the output of du -hd 1 /mnt here?

I also think it is better to have the data of the storagenode in a subfolder of the mountpoint, not directly at /mnt. Then the container wouldn’t start if the hdd is not mounted and therefore the subfolder does not exist.

Edit: Or even better, post the output of du -hd 1 /mnt before and after unmounting the hdd. I also noticed further up you wrote that a folder /mnt/storage exists. Is it possible that the data resides inside this folder? Then the folder in your docker command is wrong. Edit2: I was wrong with this statement.

1 Like
18

~/certs is not the same folder as /mnt/certs unless you changed the homefolder of the user from /home/user to /mnt manually.

19

This is the output with /mnt mounted:

fabrizio@ubuntu : ~ $ sudo du -hd 1 /mnt

du: cannot access ‘/mnt/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/aw/gyadxtnoydtyrk4lbxq36x3izlx3xzhs5sgk5ro3b4vv6zsb4a.sj1’: Bad message

du: cannot access ‘/mnt/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/xc/fqky5lcppymfsoduirwc7ll7bxbydvix35wz4a7exkfai5xktq.sj1’: Structure needs cleaning

1.2T /mnt/storage

28K /mnt/certs

8.0K /mnt/config

56K /mnt/identity

16K /mnt/lost+found

1.2T /mnt

This is the output with /mnt umount:

fabrizio@ubuntu : ~ $ sudo umount /mnt

fabrizio@ubuntu : ~ $ sudo du -hd 1 /mnt

4.0K /mnt/tmp

4.0K /mnt/QNAP

2.5M /mnt/FREENAS

2.5M /mnt

20

Ok, it looks like it got mounted correctly, but everything is kind of a mess.

Firstly you seem to have some filesystem corruption. Google “du bad message” and “du structure needs cleaning”.
Secondly you mount a subfolder inside the container of an already inside the container mounted folder.
According to your docker run command your identity should be in /mnt/certs. What does ll /mnt/certs say?

To be honest I would restructure the whole thing a bit. It looks like you’re using /mnt for other mounts (freenas, qnap, …) too. With the hdd unmounted I would create a folder /mnt/storagenode and mount the hdd there (don’t forget to change it in fstab too). Then with the hdd mounted I would create a folder “/mnt/storagenode/node” and move the folder "storage " in there. Your docker run command should then look like
...--mount type=bind,source="/mnt/storagenode/certs",destination=/app/identity --mount type=bind,source="/mnt/storagenode/node",destination=/app/config...

But first you should fix the filesystem corruption.