Node total out of work

Node Operators troubleshooting
1

Strange things ongoing. Now one of my node totally desapeared

image
image903×498 99.5 KB

I have restarted service, computer. Checked config - its fine.
I tryed to open error log file, but it is huge - over 12Gb! No one redactor can’t open it.

Shall I wait couple of days and it will fix itself or shall I do something?

Regards,
Alexander

2

Hi @Aka985
You need to view the log file - How do I check my logs? - Storj Node Operator Docs

3

PS C:\Users\Storj2> Get-Content “$env:ProgramFiles/Storj/Storage Node/storagenode.log” -Tail 20 -Wait
2022-11-22T18:39:24.536+0300 INFO piecestore downloaded {“Piece ID”: “2WKYGKKZKILUE7BQSWK2B47GC637UTOCVSZFGKQAC4M6UXCCZNEA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2022-11-22T18:39:24.840+0300 INFO piecestore download started {“Piece ID”: “XZJH46FEWGZ5O6YDGIESRLJDPN364KSVTCA4ZRORSH2IBBUIC5RA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-11-22T18:39:26.629+0300 INFO piecedeleter delete piece sent to trash {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Piece ID”: “XZJH46FEWGZ5O6YDGIESRLJDPN364KSVTCA4ZRORSH2IBBUIC5RA”}
2022-11-22T18:39:26.878+0300 INFO piecestore upload started {“Piece ID”: “ASULHK4FVZRCTPXCAIMPYXC4XUB3SFGQNYEVLAXTT2ZGDWJZWUTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 3674065629952}
2022-11-22T18:39:26.886+0300 INFO piecestore upload started {“Piece ID”: “HIHIEW4SGVSSKGLAD4HFKUHV2GSD3JPG3TMRU5NZGMZBQUOCQUBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 3674065629952}
2022-11-22T18:39:26.952+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:670\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:694\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:26.952+0300 INFO piecestore downloaded {“Piece ID”: “XZJH46FEWGZ5O6YDGIESRLJDPN364KSVTCA4ZRORSH2IBBUIC5RA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-11-22T18:39:27.131+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:445\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.131+0300 INFO piecestore uploaded {“Piece ID”: “HIHIEW4SGVSSKGLAD4HFKUHV2GSD3JPG3TMRU5NZGMZBQUOCQUBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 2816}
2022-11-22T18:39:27.222+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:445\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.222+0300 INFO piecestore uploaded {“Piece ID”: “ASULHK4FVZRCTPXCAIMPYXC4XUB3SFGQNYEVLAXTT2ZGDWJZWUTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 9472}
2022-11-22T18:39:27.222+0300 ERROR piecestore:cache error getting current used space: {“error”: “FindNextFile F:\Storj2\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\5g: The file or directory is corrupted and unreadable.”}
2022-11-22T18:39:27.223+0300 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “FindNextFile F:\Storj2\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\5g: The file or directory is corrupted and unreadable.”}
2022-11-22T18:39:27.226+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:351\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.228+0300 INFO piecestore upload canceled {“Piece ID”: “Q7NNDGBZCXW6WCZ7YT3ZV72MQIY4VSZOOF65KBLHUOEDXMXFUIMA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 311296}
2022-11-22T18:39:27.228+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:351\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.230+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:673\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:694\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.230+0300 INFO piecestore downloaded {“Piece ID”: “SE6FCPUWJCPIDG3IYF2PVP2RQ3H5M4G2ISUHYEQXABHQA4US57CQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-11-22T18:39:27.231+0300 INFO piecestore upload canceled {“Piece ID”: “ILPROK7UZ4LD2EKZ75AQUCMZCF2UZARRU7F3GR5DR233EVJ4OJ3A”,

Yep. Here it is.

“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 1318912}
2022-11-22T18:39:27.406+0300 FATAL Unrecoverable error {“error”: “FindNextFile F:\Storj2\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\5g: The file or directory is corrupted and unreadable.”}

4

Yes, run chkdsk with fix on F:

This is also an issue. You’ll need to repair or replace bandwidth.db -

5

I did chkdsk, it found errors and fix it. But node still is offline. What yet can be done?

6

Please fix databases too.
Then please provide the last 20 lines from your logs between two new lines with three backticks like this:

```
log lines here
```
7

Hi, could you help me with spelling, please

image
image860×439 92.9 KB

How command shall looks like for CMD?
I used this:
image
image665×667 15.5 KB

Regards,
Alexander

8

Before databases check please stop the storagenode service either from the Services applet or from the elevated PowerShell:

Stop-Service storagenode

if it’s a Windows native version of sqlite3, then you need to use normal Windows paths.
In the command prompt you can either provide a path to the database as an argument, or open it after sqlite3 tool is running.
From the Command Prompt:

c:\sqlite>sqlite3 F:\Storj2\bandwidth.db "PRAGMA integrity_check;"
OK

from the sqlite3 prompt:

c:\sqlite>sqlite3
sqlite> .open F:\Storj2\bandwidth.db
sqlite> PRAGMA integrity_check;
OK
sqlite> .exit

However, you actually need to check all databases, so please open a PowerShell instead and check all databases:

Get-ChildItem F:\Storj2\*.db -File | %{$_.Name + " " + $(C:\sqlite\sqlite3.exe $_.FullName "PRAGMA integrity_check;")}
9

I did, looks there some mistake with bandwidth.db

image
image1338×624 76.9 KB

Shall I now do this?
image

Regards,
Alexander

10

The error is “database disk image is malformed” so the fix is this link -

If you don’t care about the information in the current bandwidth.db (i.e. dashboard bandwidth stats) then you can skip to deletion and recreation as in the step you highlighted.

11

Looks something is wrong with spelling, Could you help

image
image1458×765 113 KB

DB location is:
image

Regards,
Alexander

12

Looks like you need to stop the node first, as the database is still in use.

Then if you are at step 9…

image
image700×142 4 KB 

cp F:/Storj2/bandwidth.db F:/Storj2/bandwidth.db.bak
sqlite3.exe F:/Storj2/bandwidth.db
13

Storj service is off

Something is not OK:

image
image1174×272 70.2 KB

It said, what “sqlite.exe3 is not recognized as a command, check location and spelling”
I also tryed from С:\ and from C:\sqlite\
image
image1183×566 75.1 KB

And than it said “sqlite3.exe not found, but exist. PowerShall do not download commands from this location. If you are sure enter .sqlite3.exe”
And now it stay like this:
image
image1175×153 37 KB

If I understand right I shall see this:
image
image720×165 2.75 KB

But not

What can be wrong?

Regards,
Alexander

14

do yo have some sqlite3 instaled?
I use DB Brouser app, it is for sqlite it hase user interface and it easier to use for normal people

15

Ah ok. This is correct
image
The path needs to be C:\sqlite and then to run an executable you need .\sqlite3.exe, but then you should have the path\databasename after:

PS C:\sqlite> .\sqlite3.exe F:\Storj2\bandwidth.db

Then at the prompt ‘within’ sqlite you type the next commands:

sqlite> .mode insert
sqlite> .output F:\Storj2\dump_all.sql
sqlite> .dump
sqlite> .exit

The above can take time to run.

16

Okay, looks whis I did. But no visual respond. Is it OK?

image
image1200×523 63.7 KB

Next - from which directory I shall start next command from p.11.2?
image
image784×285 6.85 KB

Get-Content dump_all.sql | Select-String -NotMatch TRANSACTION | Select-String -NotMatch ROLLBACK | Select-String -NotMatch COMMIT | Set-Content -Encoding utf8 dump_all_notrans.sql

Regards,
Alexander

17

Like this?

image
image1190×145 45.2 KB

No…Something is wrong(

image
image1187×207 62.3 KB

image
image757×291 45.9 KB

Regards,
Alexander

18

No.

Check in C:\sqlite and in F:\Storj2 for a file called ‘dump_all.sql’ then run the Get-Content command from that folder.

19

Looks I have done this step:

image
image708×305 90.7 KB

And p.12 too:
image
image592×649 69.5 KB

But stuck with 13:
image
image1192×485 78.9 KB

What is wrong?

Regards,
Alexander

20

I found and corrected path mistake, but still a problemm

image
image1187×430 135 KB

And even like this:
image
image1177×156 93.3 KB

Please help
Regards,
Alexander