Cannot start node, "error": "bandwidthdb error: no such table: bandwidth_usage_rollups" occurs

Node Operators troubleshooting
1

My node has suddenly crashed today.
I have run storagenode.exe diag and got following result:
2020-01-23T22:00:37.182+0100 FATAL Unrecoverable error {"error": "bandwidthdb error: no such table: bandwidth_usage_rollups", "errorVerbose": "bandwidthdb error: no such table: bandwidth_usage_rollups\n\tstorj.io/storj/storagenode/storagenodedb.
Both Storj V3 Storage Node and Updates services cannot be started.
I use Windows GUI service and haven’t touched anything in config

I have found this in storagenode-updater-recovery.log:
A system error has occurred.

System error 1067 has occurred.

The process terminated unexpectedly.

Thu 01/23/2020 22:11:27.54 recovery triggered to execute in 3 minutes 
Thu 01/23/2020 22:11:27.54 recovery command executed: 
    1 file(s) copied.
The Storj V3 Storage Node Updater service is starting.
The Storj V3 Storage Node Updater service could not be started.

A system error has occurred.

System error 1067 has occurred.

The process terminated unexpectedly.
2

can you go to HDD that serve this storage, can you open it, files there?

3

Yes, everything seems normal. I can list files and have ~0.5TB free space

4

tried windows restart? some times update make bad things, and after restart working again

5

Hello @thev,
Welcome to the forum!

Please, show the last 20 lines from the log: https://documentation.storj.io/resources/faq/check-logs

6

Sorry, I thought it wasn’t relevant as I have already freed some space before and but still cannot start node.

2020-01-23T21:24:46.151+0100    INFO    preflight:localtime     start checking local system clock with trusted satellites' system clock.
2020-01-23T21:24:47.080+0100    INFO    preflight:localtime     local system clock is in sync with trusted satellites' system clock.
2020-01-23T21:24:47.080+0100    INFO    Node 12geU8Jg6xB1uFoxYSrXe4V4S8PXjAWm6no4varMAbzGubQG3uH started
2020-01-23T21:24:47.080+0100    INFO    Public server started on [::]:28967
2020-01-23T21:24:47.080+0100    INFO    Private server started on 127.0.0.1:7778
2020-01-23T21:24:47.080+0100    INFO    bandwidth       Performing bandwidth usage rollups
2020-01-23T21:24:47.080+0100    INFO    trust   Scheduling next refresh {"after": "4h31m2.621403602s"}
2020-01-23T21:24:47.086+0100    INFO    piecestore:monitor      Remaining Bandwidth     {"bytes": 29630089388288}
2020-01-23T21:24:47.086+0100    WARN    piecestore:monitor      Disk space is less than requested. Allocating space     {"bytes": 428090948864}
2020-01-23T21:24:47.086+0100    ERROR   piecestore:monitor      Total disk space less than required minimum     {"bytes": 500000000000}
2020-01-23T21:24:47.086+0100    ERROR   piecestore:cache        error getting current space used calculation:   {"error": "context canceled"}
2020-01-23T21:24:47.086+0100    ERROR   version Failed to do periodic version check: version control client error: Get https://version.storj.io: context canceled
2020-01-23T21:24:47.086+0100    ERROR   piecestore:cache        error during init space usage db:       {"error": "piece space used error: context canceled", "errorVerbose": "piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).Init:71\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:64\n\tstorj.io/storj/storagenode.(*Peer).Run.func6:477\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-23T21:24:47.086+0100    ERROR   piecestore:cache        error persisting cache totals to the database:  {"error": "piece space used error: context canceled", "errorVerbose": "piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdatePieceTotal:172\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:73\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:68\n\tstorj.io/storj/storagenode.(*Peer).Run.func6:477\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-23T21:24:47.086+0100    ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:253\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:323\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-23T21:24:47.086+0100    ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:253\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:323\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-23T21:24:47.087+0100    ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:253\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:323\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-23T21:24:47.087+0100    ERROR   pieces:trash    emptying trash failed   {"error": "pieces error: filestore error: context canceled", "errorVerbose": "pieces error: filestore error: context canceled\n\tstorj.io/storj/storage/filestore.(*blobStore).EmptyTrash:127\n\tstorj.io/storj/storagenode/pieces.(*BlobsUsageCache).EmptyTrash:253\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:323\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1:51\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/common/sync2.(*Cycle).Start.func1:68\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-23T21:24:47.100+0100    ERROR   collector       error during collecting pieces:         {"error": "piece expiration error: context canceled", "errorVerbose": "piece expiration error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).GetExpired:39\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:477\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:88\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:55\n\tstorj.io/common/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/collector.(*Service).Run:51\n\tstorj.io/storj/storagenode.(*Peer).Run.func4:471\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-01-23T21:24:47.107+0100    FATAL   Unrecoverable error     {"error": "piecestore monitor: disk space requirement not met", "errorVerbose": "piecestore monitor: disk space requirement not met\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:118\n\tstorj.io/storj/storagenode.(*Peer).Run.func2:465\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
7

My storj partition is 686GB, 516GB of it is free at this moment. My allocated disk space is
storage.allocated-disk-space: 0.5 TB

8

Thanks, but already tried with restart, no success

9

what version of soragenode you have? 0.30.5
how did you cleaned space there? what files you deleted?

10 
PS C:\Program Files\Storj\Storage Node> .\storagenode.exe version
2020-01-23T22:33:24.522+0100    INFO    Configuration loaded from: C:\Users\thev\AppData\Roaming\Storj\Storagenode\config.yaml
Release build
Version: v0.30.5
Build timestamp: 21 Jan 20 16:53 CET
Git commit: 72805d55ca6dcbc5dd32883e64538d77fcb35b8a
11

how did you cleaned space there? what files you deleted?

12

I had some backups on the same volume, not related with storj. Just removed them

13

Before that I had ~200GB free space

14

try to uncoment # storage2.monitor.minimum-disk-space: 500.0 GB
and make it 300
also alocated space 0.3TB

15

looks like it see that you have less space.

16

Still no results, no new entries in storagenode-updater.log nor storagenode.log. Only storagenode-updater-recovery.log gets updates with same errors

17

looks like your updater stucked. after you make change in config you shold restart your node in services

18

Are there any additional logs I can gather for updater?

19

At this moment both updater and node services are stoppeed. Cannot start node nor updater service

20

After tried start node, check storagenode.log
new messages?