All statistic was set to zero

Node Operators troubleshooting
,
1

Hi! My storj node works 24/7. And few minutes ago I decided to check statistic and found strange thing. All statistic was set to zero. I tried to choose satellites, but nothing happened. I pushed “refresh button” in a header of the page and still got zero statistic. I tried to refresh page, but no results (still zero). But status was “ONLINE” (look at screenshot below). I checked “docker ps -a” - everything works. Then I decided to reboot whole node and now it works fine.

zero
zero953×1810 83.6 KB

Could you add LOG output to web GUI ?

May be it would be better to add “force” checking of status. I mean web GUI should show alert if all stat set to zero with 200GB data in storage.

2

Check HDD, looks like it cant get DB, Be fast, sattelite can DQ you forever.
be offline is better than sattelite will think you lost data.

1 Like
3

Please, check all your databases:

4

Started checking HDD for bad blocks. But it is new HDD. I don’t think it will be a lot of problems.

Now node is working. I can see no alerts in “docker logs storagenode”

Begun to read “How to fix a “database disk image is malformed””.
But now I don’t have enough time :frowning: I have to go. I’ll do it later. Is it possible to check integrity using script?

5

linux is not my best, but check mounting folder, that you HDD mount not braked, then it will see a just folter of your OS, not mounted HDD.

6

HDD mounted correctly (using fstab). It works.
I’ll check integrity of all db files. But there are so many steps :frowning: Why there are no script that does it automatically? It is strange. I mean like

“docker exec -it **** /app/integrity.sh”

It’ll be very useful to create this script!

7 
docker run --rm -it --mount type=bind,source=$PWD,destination=/data sstc/sqlite3 find . -iname *.db -maxdepth 1 -print -exec sqlite3 '{}' 'PRAGMA INTEGRITY_CHECK;' ';'

replace $PWD with the absolute path to the databases location.
The storagenode must be stopped.

2 Likes
8

Just fixed a problem: bad power supply of HDD :frowning:

HDD said: “Structure Needs Cleaning”. After that I used “fsck” to restore it. HDD works fine. I copied and checked all *.db files and there are a lot of mistakes:

root@alex-pc:~/storj# find -name *.db -exec echo -n '{} - ’ ; -exec sqlite3 ‘{}’ ‘PRAGMA INTEGRITY_CHECK;’ ‘;’
bandwidth.db - *** in database main ***
Page 494: btreeInitPage() returns error code 11
Page 492: btreeInitPage() returns error code 11
… ~20 same problems …
Page 472: btreeInitPage() returns error code 11
Page 468: btreeInitPage() returns error code 11
Error: database disk image is malformed
heldamount.db - ok
info.db - ok
notifications.db - ok
orders.db - ok
piece_expiration.db - ok
pieceinfo.db - ok
piece_spaced_used.db - ok
pricing.db - ok
reputation.db - ok
revocations.db - Error: file is not a database
satellites.db - ok
storage_usage.db - ok
used_serial.db - ok

Now I begin to restore “bandwidth.db”. But I have a question about “revocations.db”. Should I try to restore “revocations.db” or it is not a true DB file?

9

I followed instructions from article: “How to fix a “database disk image is malformed””

And now almost all DB files OK. Except “revocations.db”.

root@alex-pc:~/storj# find -name *.db -exec echo -n '{} - ’ ; -exec sqlite3 ‘{}’ ‘PRAGMA INTEGRITY_CHECK;’ ‘;’
./storage/revocations.db - Error: file is not a database
./storage/storage/piece_spaced_used.db - ok
./storage/storage/reputation.db - ok
./storage/storage/piece_expiration.db - ok
./storage/storage/orders.db - ok
./storage/storage/satellites.db - ok
./storage/storage/heldamount.db - ok
./storage/storage/bandwidth.db - ok
./storage/storage/info.db - ok
./storage/storage/notifications.db - ok
./storage/storage/pieceinfo.db - ok
./storage/storage/used_serial.db - ok
./storage/storage/pricing.db - ok
./storage/storage/storage_usage.db - ok

I’ll try to start node and check if it works without recovering “revocations.db”.

10

Everything is fixed. Node has status “ONLINE”. I don’t see any failed audits (should I?) when I execute:

docker logs storagenode 2>&1 | grep GET_AUDIT | grep failed | grep open -c

And I don’t see any disqualification notice in web GUI. I hope that problem is solved. And there is no reason to restore “revocations.db” (may be it is not even a true DB file).

11

The revocations.db in a different format, so you can ignore it.