After storenode reboot my displayed space is decrese from 1.2 TB to 16GB why?

Node Operators troubleshooting
21

Is it local or remote drive?

22

I have run sudo e2fsck /dev/sda1 and fix the errors, now after the restart I get this error:

docker exec -it storagenode /app/dashboard.sh

OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused “read init-p: connection reset by peer”: unknown

and looking my log I have this errors:

docker logs --tail 20 storagenode

2019-12-09T23:36:27.038Z ERROR orders cleaning archive {“error”: “ordersdb error: database disk image is malformed”, “errorVerbose”: “ordersdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).CleanArchive:320\n\tstorj.io/storj/storagenode/orders.(*Service).cleanArchive:136\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/private/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2019-12-09T23:36:27.057Z FATAL Unrecoverable error {“error”: “bandwidthdb error: database disk image is malformed”, “errorVerbose”: “bandwidthdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:120\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).MonthSummary:73\n\tstorj.io/storj/storagenode/monitor.(*Service).usedBandwidth:174\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:83\n\tstorj.io/storj/storagenode.(*Peer).Run.func2:423\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2019-12-09T23:37:30.157Z INFO Configuration loaded from: /app/config/config.yaml

2019-12-09T23:37:30.236Z INFO Operator email: brizio71@gmail.com

2019-12-09T23:37:30.239Z INFO operator wallet: 0xB8f8a2A65916b1442B824833Cfd6101910f8Ab7b

2019-12-09T23:37:30.872Z INFO version running on version v0.26.2

2019-12-09T23:37:30.904Z INFO db.migration Database Version {“version”: 25}

2019-12-09T23:37:30.924Z INFO bandwidth Performing bandwidth usage rollups

2019-12-09T23:37:30.929Z INFO Node 12hpvUSVRDAhujRZAq1fSjq2woz2sh3Gns9vYP7f56ukypCVYAN started

2019-12-09T23:37:30.929Z INFO Public server started on [::]:28967

2019-12-09T23:37:30.929Z INFO Private server started on 127.0.0.1:7778

2019-12-09T23:37:30.935Z INFO contact:chore Storagenode contact chore starting up

2019-12-09T23:37:30.943Z ERROR orders cleaning archive {“error”: “ordersdb error: database disk image is malformed”, “errorVerbose”: “ordersdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).CleanArchive:320\n\tstorj.io/storj/storagenode/orders.(*Service).cleanArchive:136\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/private/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2019-12-09T23:37:30.952Z WARN orders DB contains invalid marshalled orders {“error”: “ordersdb error: database disk image is malformed”, “errorVerbose”: “ordersdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).ListUnsentBySatellite:169\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders:152\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/private/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2019-12-09T23:37:30.957Z ERROR piecestore:cacheUpdate error getting current space used calculation: {“error”: “context canceled”}

2019-12-09T23:37:30.958Z ERROR piecestore:cacheUpdate error persisting cache totals to the database: {“error”: “piece space used error: context canceled”, “errorVerbose”: “piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdateTotal:115\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:82\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:68\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:63\n\tstorj.io/storj/storagenode.(*Peer).Run.func6:435\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2019-12-09T23:37:30.961Z ERROR version Failed to do periodic version check: version control client error: Get https://version.storj.io: context canceled

2019-12-09T23:37:30.963Z ERROR bandwidth Could not rollup bandwidth usage {“error”: “bandwidthdb error: database disk image is malformed”, “errorVerbose”: “bandwidthdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Rollup:253\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Rollup:53\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Run:45\n\tstorj.io/storj/storagenode.(*Peer).Run.func8:441\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2019-12-09T23:37:30.964Z ERROR collector error during collecting pieces: {“error”: “v0pieceinfodb error: context canceled”, “errorVerbose”: “v0pieceinfodb error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*v0PieceInfoDB).GetExpired:195\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:410\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:85\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:52\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/collector.(*Service).Run:51\n\tstorj.io/storj/storagenode.(*Peer).Run.func4:429\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2019-12-09T23:37:31.036Z FATAL Unrecoverable error {“error”: “bandwidthdb error: database disk image is malformed”, “errorVerbose”: “bandwidthdb error: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:120\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).MonthSummary:73\n\tstorj.io/storj/storagenode/monitor.(*Service).usedBandwidth:174\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:83\n\tstorj.io/storj/storagenode.(*Peer).Run.func2:423\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

23
24

It works the db fix and now 1.3TB is shown correctly. thanks

25

You still haven’t answered the question, is it an internal HDD?

26

You mean about ll /mnt/certs output ?

fabrizio@ubuntu : ~ $ ll /mnt/certs/

total 32

drwxrwxrwx 2 root root 4096 Sep 25 14:34 ./

drwxr-xr-x 7 root root 4096 Sep 27 09:11 … /

-rw-r–r-- 1 root root 546 Jun 20 17:26 ca.1561026062.cert

-rw-r–r-- 1 root root 1076 Jun 20 17:26 ca.cert

-rw-r–r-- 1 root root 241 Jun 20 17:26 ca.key

-rw-r–r-- 1 root root 1084 Jun 20 17:26 identity.1561026062.cert

-rw-r–r-- 1 root root 1614 Jun 20 17:26 identity.cert

-rw-r–r-- 1 root root 241 Jun 20 17:26 identity.key

27

No, is it connected to the SATA port of your computer or is it on the network inside a NAS or similar?

28

It’s an ISCSI connection on a QNAP NAS

29

Try to search something unusual in syslog:
cat /var/log/syslog|grep "mount"

30

the last messages was before the db fix:

Dec 10 09:22:00 ubuntu dockerd[897]: time=“2019-12-10T09:22:00.924443963+01:00” level=warning msg=“3e6e721efbd67b30846807104bfa86c9dda27237c5d7d621a95cf2990808a7e8 cleanup: failed to un mount IPC: u mount /var/lib/docker/containers/3e6e721efbd67b30846807104bfa86c9dda27237c5d7d621a95cf2990808a7e8/ mount s/shm, flags: 0x2: no such file or directory”

Dec 10 09:26:23 ubuntu dockerd[897]: time=“2019-12-10T09:26:23.102251376+01:00” level=warning msg=“7799b65e8ec04192bbbac085ebbbb64f00e0fc4dcdcdb812f57ea32112c07052 cleanup: failed to un mount IPC: u mount /var/lib/docker/containers/7799b65e8ec04192bbbac085ebbbb64f00e0fc4dcdcdb812f57ea32112c07052/ mount s/shm, flags: 0x2: no such file or directory”

31

I think your system drive also was corrupted, please try to remove docker container and recreate it with your docker run command.

docker stop storagenode -t 300
docker rm storagenode 
docker image rm storjlabs/storagenode:beta
docker run ......