FATAL Unrecoverable error {"error": "bandwidthdb error: disk I/O error"

claudiosi · December 26, 2019, 3:39pm

Hello @Alexey , i receive the same error after 25 days of no problems.

I was checking my node almost everyday and was working, but now, after 23-25 days suddenly is not working.

My router was not restarted, i was not doing any changes on IP or something like this.

Here is my storagenode.log . Please check it and advise me how to proceed

PS C:\WINDOWS\system32> cat "$env:ProgramFiles/Storj/Storage Node/storagenode.log" -Tail 20
2019-12-26T16:00:04.845+0200    INFO    db.migration    Database Version        {"version": 26}
2019-12-26T16:00:04.867+0200    INFO    Node 12rWuyoZQRdCZGFXUupW7MbxxugMeTK98ot7rx2ZDZMbMjqocqk started
2019-12-26T16:00:04.867+0200    INFO    Public server started on [::]:28967
2019-12-26T16:00:04.867+0200    INFO    Private server started on 127.0.0.1:7778
2019-12-26T16:00:04.868+0200    INFO    pieces:trashchore       Storagenode TrashChore starting up
2019-12-26T16:00:04.868+0200    INFO    contact:chore   Storagenode contact chore starting up
2019-12-26T16:00:04.867+0200    INFO    bandwidth       Performing bandwidth usage rollups
2019-12-26T16:00:05.007+0200    INFO    version running on version v0.27.1
2019-12-26T16:00:05.040+0200    INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      sending {"count": 2}
2019-12-26T16:00:05.040+0200    INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      sending {"count": 4}
2019-12-26T16:00:05.525+0200    INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      finished
2019-12-26T16:00:06.044+0200    INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      finished
2019-12-26T16:00:08.567+0200    INFO    Interrogate request received.
2019-12-26T16:00:08.683+0200    INFO    Stop/Shutdown request received.
2019-12-26T16:00:15.450+0200    ERROR   piecestore:cacheUpdate  error getting current space used calculation:   {"error": "context canceled"}
2019-12-26T16:00:15.450+0200    ERROR   piecestore:cacheUpdate  error persisting cache totals to the database:  {"error": "piece space used error: context canceled", "errorVerbose": "piece space used error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceSpaceUsedDB).UpdateTotal:115\n\tstorj.io/storj/storagenode/pieces.(*CacheService).PersistCacheTotals:82\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run.func1:68\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:63\n\tstorj.io/storj/storagenode.(*Peer).Run.func6:445\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-12-26T16:00:16.687+0200    ERROR   orders  archiving orders        {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).archiveOne:238\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Archive:202\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches.func2:213\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches:237\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func1:164\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-12-26T16:01:21.336+0200    ERROR   bandwidth       Could not rollup bandwidth usage        {"error": "bandwidthdb error: database is locked", "errorVerbose": "bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Rollup:300\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Rollup:53\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/bandwidth.(*Service).Run:45\n\tstorj.io/storj/storagenode.(*Peer).Run.func9:454\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-12-26T16:02:13.641+0200    ERROR   collector       error during collecting pieces:         {"error": "piece expiration error: context canceled", "errorVerbose": "piece expiration error: context canceled\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).GetExpired:39\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:473\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:88\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:55\n\tstorj.io/storj/private/sync2.(*Cycle).Run:87\n\tstorj.io/storj/storagenode/collector.(*Service).Run:51\n\tstorj.io/storj/storagenode.(*Peer).Run.func4:439\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-12-26T16:02:17.481+0200    FATAL   Unrecoverable error     {"error": "bandwidthdb error: disk I/O error", "errorVerbose": "bandwidthdb error: disk I/O error\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:153\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:111\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).MonthSummary:73\n\tstorj.io/storj/storagenode/monitor.(*Service).usedBandwidth:174\n\tstorj.io/storj/storagenode/monitor.(*Service).Run:83\n\tstorj.io/storj/storagenode.(*Peer).Run.func2:433\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
PS C:\WINDOWS\system32>

Alexey · December 26, 2019, 8:36pm

What happened with your disk with data?
How is it connected?

Please, try to change the owner of all files in the data folder on your drive to the SYSTEM, grant full rights to it and for your user too, then apply them recursively.

claudiosi · December 26, 2019, 9:10pm

Hello @Alexey , thank you very much for your replay

All storj files are stored into a HDD external Seagate Expansion 4TB, 3.5", via his own USB 3.0 cable directly with my laptop.

I don’t change any files into storj folder.

How can i resolve this situation ?

Alexey · December 26, 2019, 9:52pm

First - clear the “Read only” checkbox and click “Apply”.
Then check is the problem gone.
If not - replace the owner to the SYSTEM, grant full rights to it and for your user recursively.
There’s a plenty information in the internet, how to change permissions on files in the Windows, for example https://www.windowscentral.com/how-take-ownership-files-and-folders-windows-10

claudiosi · December 27, 2019, 4:18am

Dear @Alexey , i perform all the steps, i clear read only and apply, but same issue again.
I change the security owner from my own account to SYSTEM and try to access http://localhost:14002/ but same issue

After restarting the laptop the flag from read only (storj folder) was back on

Any other advise friend ?

Thank you for the support

Alexey · December 27, 2019, 5:31am

Try to start the storagenode service either from the Services applet, or from the elevated Powershell:

Start-Service storagenode

Check what error on that time in the log: https://documentation.storj.io/resources/faq/check-logs

Show the content of the folder (Powershell):

ls "D:\storj files 1"

Show permissions:

Get-Acl "D:\storj files 1\bandwidth.db"

claudiosi · December 27, 2019, 7:32am

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6

PS C:\WINDOWS\system32> Start-Service storagenode
PS C:\WINDOWS\system32> ls "D:\storj files 1"


    Directory: D:\storj files 1


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        12/2/2019   7:12 PM                blobs
d-----       12/21/2019  12:51 AM                garbage
d-----       12/20/2019  11:42 AM                temp
d-----       11/30/2019   8:52 PM                trash
-a----       12/21/2019  12:43 AM         827392 bandwidth.db
-a----       12/27/2019   9:27 AM          32768 bandwidth.db-shm
-a----       12/21/2019  12:43 AM        4214792 bandwidth.db-wal
-a----       11/30/2019   8:52 PM           8192 info.db
-a----       12/27/2019   9:27 AM          32768 info.db-shm
-a----       12/27/2019   9:27 AM              0 info.db-wal
-a----       12/27/2019   6:10 AM          16384 notifications.db
-a----       12/27/2019   9:27 AM          32768 notifications.db-shm
-a----       12/27/2019   9:27 AM              0 notifications.db-wal
-a----       12/27/2019   6:10 AM       24981504 orders.db
-a----       12/27/2019   9:27 AM          32768 orders.db-shm
-a----       12/27/2019   9:27 AM             32 orders.db-wal
-a----       11/30/2019   8:52 PM          16384 pieceinfo.db
-a----       12/27/2019   9:27 AM          32768 pieceinfo.db-shm
-a----       12/27/2019   9:27 AM              0 pieceinfo.db-wal
-a----       12/11/2019   9:50 PM          28672 piece_expiration.db
-a----       12/27/2019   9:27 AM          32768 piece_expiration.db-shm
-a----       12/27/2019   9:27 AM              0 piece_expiration.db-wal
-a----       12/26/2019   4:02 PM          12288 piece_spaced_used.db
-a----       12/27/2019   9:27 AM          32768 piece_spaced_used.db-shm
-a----       12/27/2019   9:27 AM              0 piece_spaced_used.db-wal
-a----       12/27/2019   6:10 AM          12288 reputation.db
-a----       11/30/2019   9:28 PM          32768 satellites.db
-a----       12/27/2019   9:27 AM          32768 satellites.db-shm
-a----       12/27/2019   9:27 AM              0 satellites.db-wal
-a----       12/27/2019   6:10 AM          36864 storage_usage.db
-a----       12/26/2019   4:02 PM        5677056 used_serial.db
-a----       12/27/2019   9:27 AM          32768 used_serial.db-shm
-a----       12/27/2019   9:27 AM              0 used_serial.db-wal


PS C:\WINDOWS\system32> Get-Acl "D:\storj files 1\bandwidth.db"


    Directory: D:\storj files 1


Path         Owner               Access
----         -----               ------
bandwidth.db NT AUTHORITY\SYSTEM BUILTIN\Administrators Allow  FullControl...


PS C:\WINDOWS\system32>

Alexey · December 27, 2019, 7:45am

What is in the storagenode logs?

claudiosi · December 27, 2019, 7:57am

here is the logs @Alexey

sorry but the forum says Sorry, new users can only put 2 links in a post.

i will add screen

Alexey · December 27, 2019, 8:14am

Please, try to check it with this article:

Alexey · August 27, 2021, 8:51am

A post was merged into an existing topic: I got a message local host refused to connect