Aka985
November 22, 2022, 5:00pm
1
Strange things ongoing. Now one of my node totally desapeared
I have restarted service, computer. Checked config - its fine.
I tryed to open error log file, but it is huge - over 12Gb! No one redactor can’t open it.
Shall I wait couple of days and it will fix itself or shall I do something?
Regards,
Alexander
Stob
November 22, 2022, 5:08pm
2
Aka985
November 22, 2022, 5:22pm
3
PS C:\Users\Storj2> Get-Content “$env:ProgramFiles/Storj/Storage Node/storagenode.log” -Tail 20 -Wait
2022-11-22T18:39:24.536+0300 INFO piecestore downloaded {“Piece ID”: “2WKYGKKZKILUE7BQSWK2B47GC637UTOCVSZFGKQAC4M6UXCCZNEA”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET”}
2022-11-22T18:39:24.840+0300 INFO piecestore download started {“Piece ID”: “XZJH46FEWGZ5O6YDGIESRLJDPN364KSVTCA4ZRORSH2IBBUIC5RA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-11-22T18:39:26.629+0300 INFO piecedeleter delete piece sent to trash {“Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Piece ID”: “XZJH46FEWGZ5O6YDGIESRLJDPN364KSVTCA4ZRORSH2IBBUIC5RA”}
2022-11-22T18:39:26.878+0300 INFO piecestore upload started {“Piece ID”: “ASULHK4FVZRCTPXCAIMPYXC4XUB3SFGQNYEVLAXTT2ZGDWJZWUTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 3674065629952}
2022-11-22T18:39:26.886+0300 INFO piecestore upload started {“Piece ID”: “HIHIEW4SGVSSKGLAD4HFKUHV2GSD3JPG3TMRU5NZGMZBQUOCQUBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Available Space”: 3674065629952}
2022-11-22T18:39:26.952+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:670\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:694\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:26.952+0300 INFO piecestore downloaded {“Piece ID”: “XZJH46FEWGZ5O6YDGIESRLJDPN364KSVTCA4ZRORSH2IBBUIC5RA”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “GET”}
2022-11-22T18:39:27.131+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:445\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.131+0300 INFO piecestore uploaded {“Piece ID”: “HIHIEW4SGVSSKGLAD4HFKUHV2GSD3JPG3TMRU5NZGMZBQUOCQUBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 2816}
2022-11-22T18:39:27.222+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:445\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.222+0300 INFO piecestore uploaded {“Piece ID”: “ASULHK4FVZRCTPXCAIMPYXC4XUB3SFGQNYEVLAXTT2ZGDWJZWUTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 9472}
2022-11-22T18:39:27.222+0300 ERROR piecestore:cache error getting current used space: {“error”: “FindNextFile F:\Storj2\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\5g: The file or directory is corrupted and unreadable.”}
2022-11-22T18:39:27.223+0300 ERROR services unexpected shutdown of a runner {“name”: “piecestore:cache”, “error”: “FindNextFile F:\Storj2\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\5g: The file or directory is corrupted and unreadable.”}
2022-11-22T18:39:27.226+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:351\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.228+0300 INFO piecestore upload canceled {“Piece ID”: “Q7NNDGBZCXW6WCZ7YT3ZV72MQIY4VSZOOF65KBLHUOEDXMXFUIMA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 311296}
2022-11-22T18:39:27.228+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:351\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.230+0300 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “bandwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).getSummary:171\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Summary:113\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:731\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:673\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:694\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}
2022-11-22T18:39:27.230+0300 INFO piecestore downloaded {“Piece ID”: “SE6FCPUWJCPIDG3IYF2PVP2RQ3H5M4G2ISUHYEQXABHQA4US57CQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET”}
2022-11-22T18:39:27.231+0300 INFO piecestore upload canceled {“Piece ID”: “ILPROK7UZ4LD2EKZ75AQUCMZCF2UZARRU7F3GR5DR233EVJ4OJ3A”,
Yep. Here it is.
“Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “Size”: 1318912}
2022-11-22T18:39:27.406+0300 FATAL Unrecoverable error {“error”: “FindNextFile F:\Storj2\blobs\ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa\5g: The file or directory is corrupted and unreadable.”}
Stob
November 22, 2022, 5:45pm
4
Yes, run chkdsk with fix on F:
This is also an issue. You’ll need to repair or replace bandwidth.db -
If your node was abruptly terminated due to power failure, the database file could be irreversible corrupted, you can see an error Error starting master database on storagenode: database: file is n...
1 Like
Aka985
November 22, 2022, 6:49pm
5
I did chkdsk, it found errors and fix it. But node still is offline. What yet can be done?
1 Like
Alexey
November 23, 2022, 4:25am
6
Please fix databases too.
Then please provide the last 20 lines from your logs between two new lines with three backticks like this:
```
log lines here
```
1 Like
Aka985
November 27, 2022, 8:25am
7
Hi, could you help me with spelling, please
How command shall looks like for CMD?
I used this:
Regards,
Alexander
Alexey
November 27, 2022, 8:52am
8
Before databases check please stop the storagenode service either from the Services applet or from the elevated PowerShell:
Stop-Service storagenode
if it’s a Windows native version of sqlite3, then you need to use normal Windows paths.
In the command prompt you can either provide a path to the database as an argument, or open it after sqlite3 tool is running.
From the Command Prompt:
c:\sqlite>sqlite3 F:\Storj2\bandwidth.db "PRAGMA integrity_check;"
OK
from the sqlite3 prompt:
c:\sqlite>sqlite3
sqlite> .open F:\Storj2\bandwidth.db
sqlite> PRAGMA integrity_check;
OK
sqlite> .exit
However, you actually need to check all databases, so please open a PowerShell instead and check all databases:
Get-ChildItem F:\Storj2\*.db -File | %{$_.Name + " " + $(C:\sqlite\sqlite3.exe $_.FullName "PRAGMA integrity_check;")}
Aka985
November 28, 2022, 9:21am
9
I did, looks there some mistake with bandwidth.db
Shall I now do this?
Regards,
Alexander
Stob
November 28, 2022, 9:28am
10
The error is “database disk image is malformed” so the fix is this link -
If you don’t care about the information in the current bandwidth.db (i.e. dashboard bandwidth stats) then you can skip to deletion and recreation as in the step you highlighted.
1 Like
Aka985
November 28, 2022, 10:22am
11
Looks something is wrong with spelling, Could you help
DB location is:
Regards,
Alexander
Stob
November 28, 2022, 10:53am
12
Looks like you need to stop the node first, as the database is still in use.
Then if you are at step 9…
cp F:/Storj2/bandwidth.db F:/Storj2/bandwidth.db.bak
sqlite3.exe F:/Storj2/bandwidth.db
Aka985
November 28, 2022, 12:14pm
13
Storj service is off
Something is not OK:
It said, what “sqlite.exe3 is not recognized as a command, check location and spelling”
I also tryed from С:\ and from C:\sqlite\
And than it said “sqlite3.exe not found, but exist. PowerShall do not download commands from this location. If you are sure enter .sqlite3.exe”
And now it stay like this:
If I understand right I shall see this:
But not
What can be wrong?
Regards,
Alexander
Vadim
November 28, 2022, 12:15pm
14
do yo have some sqlite3 instaled?
I use DB Brouser app, it is for sqlite it hase user interface and it easier to use for normal people
Stob
November 28, 2022, 12:32pm
15
Ah ok. This is correct
The path needs to be C:\sqlite and then to run an executable you need .\sqlite3.exe, but then you should have the path\databasename after:
PS C:\sqlite> .\sqlite3.exe F:\Storj2\bandwidth.db
Then at the prompt ‘within’ sqlite you type the next commands:
sqlite> .mode insert
sqlite> .output F:\Storj2\dump_all.sql
sqlite> .dump
sqlite> .exit
The above can take time to run.
Aka985
November 28, 2022, 1:32pm
16
Okay, looks whis I did. But no visual respond. Is it OK?
Next - from which directory I shall start next command from p.11.2?
Get-Content dump_all.sql | Select-String -NotMatch TRANSACTION | Select-String -NotMatch ROLLBACK | Select-String -NotMatch COMMIT | Set-Content -Encoding utf8 dump_all_notrans.sql
Regards,
Alexander
Stob
November 28, 2022, 2:05pm
18
No.
Check in C:\sqlite and in F:\Storj2 for a file called ‘dump_all.sql’ then run the Get-Content command from that folder.
Aka985
November 28, 2022, 2:22pm
19
Looks I have done this step:
And p.12 too:
But stuck with 13:
What is wrong?
Regards,
Alexander
Aka985
November 28, 2022, 2:31pm
20
I found and corrected path mistake, but still a problemm
And even like this:
Please help
Regards,
Alexander