node1
July 3, 2020, 1:45pm
1
I’ve noticed strange activity on one of the nodes. Zabbix show that used space of drive is going down all the time, but network activity does not show download.
After examination i see this:
Storage Node Dashboard ( Node Version: v1.6.4 )
======================
ID xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Available Used Egress Ingress
Bandwidth N/A 3.3 GB 3.3 GB 0 B (since Jul 1)
Disk -549.2 GB 1.3 TB
(please note, that this is 1Tb drive only, but dashboard show 1.3TB in use)
$ sudo df -h
Graphical Dashbord shows:
Docker command:
Trash folder however is only 174M on disk.
The garbage collector copy the data to the trash, then removes it from the original place. It doesn’t use the move command.
node1
July 3, 2020, 6:16pm
3
Should i do something about it ?
Is it a Linux docker node?
journalctl | grep shim
Also, it’s worth to check all databases:
docker run --rm -it --mount type=bind,source=$PWD,destination=/data sstc/sqlite3 find . -iname *.db -maxdepth 1 -print -exec sqlite3 '{}' 'PRAGMA INTEGRITY_CHECK;' ';'
replace $PWD with the absolute path to the databases location.
The storagenode must be stopped.
node1
July 3, 2020, 6:28pm
5
Jul 03 19:18:26 node containerd[575]: time=“2020-07-03T19:18:26.479397156+01:00” level=info msg=“shim reaped” id=584a9dd077e6a0853a559ccfbf54805962675276e12251d7b3c42fdfb2813370
node1:
shim reaped
The node was killed with OOM.
Also, how is your HDD connected?
node1
July 3, 2020, 6:41pm
7
HDD connected. in the first post, i’ve posted usage of the space. so there is no way that command would give me answer without disk. will try that second comand.
Should i provide path to revocations.db ?
Ok, got the answer:
docker run --rm -it --mount type=bind,source=/home/storj_data/,destination=/data sstc/sqlite3 find . -iname *.db -maxdepth 1 -print -exec sqlite3 ‘{}’ ‘PRAGMA INTEGRITY_CHECK;’ ‘;’
standard_init_linux.go:211: exec user process caused “exec format error”
failed to resize tty, using default size
I mean - is it a USB-connected drive or internal?
node1
July 3, 2020, 6:52pm
9
Yes its rpi, with usb drive.
While now i get very strange error
$ docker run --rm -it --mount type=bind,source=/home/storj_data/storage,destination=/data sstc/sqlite3 find . -iname *.db -maxdepth 1 -print0 -exec sqlite3 ‘{}’ ‘PRAGMA INTEGRITY_CHECK;’ ';'
Says docker command not found, but if i type just “docker” i get the list of the commands.
Alexey
July 3, 2020, 6:54pm
10
Then you should install sqlite3 locally.
sudo apt update && sudo apt install sqlite3
sqlite3 --version
if it’s not lower than v3.25.2, you can do like this
stop the storagenode
docker stop -t 300 storagenode
execute in the shell
find /home/storj_data/storage -iname *.db -maxdepth 1 -print0 -exec sqlite3 '{}' 'PRAGMA INTEGRITY_CHECK;' ';'
node1
July 3, 2020, 7:02pm
11
sqlite version
v 3.27.2 2019-02-25
$ docker run --rm -it --mount type=bind,source=/home/storj_data/storage,destination=/data sstc/sqlite3 find . -iname *.db -maxdepth 1 -print0 -exec sqlite3 ‘{}’ ‘PRAGMA INTEGRITY_CHECK;’ ';'
-bash: docker: command not found
$ find /home/storj_data/storage -iname *.db -maxdepth 1 -print0 -exec sqlite3 ‘{}’ ‘PRAGMA INTEGRITY_CHECK;’ ';'
-bash: find: command not found
Alexey
July 3, 2020, 7:21pm
12
Please, do not use a docker at this point
find /home/storj_data/storage -iname *.db -maxdepth 1 -print0 -exec sqlite3 '{}' 'PRAGMA INTEGRITY_CHECK;' ';'
The storagenode must be stopped.
node1
July 3, 2020, 7:28pm
13
Ok, this time worked a bit:
$ sudo find /home/storj_data/storage -iname *.db -maxdepth 1 -print0 -exec sqlite3 ‘{}’ ‘PRAGMA INTEGRITY_CHECK;’ ‘;’
[sudo] password for node:
find: warning: you have specified the -maxdepth option after a non-option argument -iname, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it). Please specify options before other arguments.
/home/storj_data/storage/storage_usage.dbok
/home/storj_data/storage/satellites.dbok
/home/storj_data/storage/info.dbok
/home/storj_data/storage/reputation.dbok
/home/storj_data/storage/piece_spaced_used.dbok
/home/storj_data/storage/pieceinfo.dbok
/home/storj_data/storage/notifications.dbok
/home/storj_data/storage/orders.dbok
/home/storj_data/storage/piece_expiration.dbok
/home/storj_data/storage/heldamount.dbok
/home/storj_data/storage/pricing.dbok
/home/storj_data/storage/used_serial.dbok
/home/storj_data/storage/bandwidth.dbok
node@node:~ $
Alexey
July 3, 2020, 7:32pm
14
Looks good. Do you have a --memory option in your docker run command as suggested there:
node1
July 3, 2020, 7:35pm
15
I did had before, but added now again. But it always gives me this error:
--name storagenode storjlabs/storagenode:latest
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
node1
July 3, 2020, 7:40pm
16
but even with memory command it keeps restarting, and shows used 1.3TB while the drive is only 1Tb
Alexey
July 3, 2020, 7:54pm
17
Could you please show the 20 last lines from the log?
docker logs --tail 20 storagenode
You can ignore that.
node1
July 3, 2020, 8:02pm
18
in logs everytime looks normal, it starting to get pieces “Action”: GET_REPAIR’ and GET_AUDIT" and etc.
…but this time between these GET_ i found also one bad line in the log. It says"
ERROR db Unable to read the disk, please verify the disk is not corrupt
Alexey
July 3, 2020, 8:04pm
19
Please, do this, it doesn’t joking.
node1
July 3, 2020, 8:04pm
20
this is not smile, its “expressionless”
