ERROR after adding another node with the same identity

Node Operators troubleshooting
1

I added a 2nd storage node to my network today. Unfortunately I missed the warning text about needing a unique identity for each node underneath 5 paragraphs about RAID, and misunderstood what the identity was for, so I used the same identity as my 1st node that’s been running since v3 started.

2nd node was running for only a few minutes when I noticed the Suspension & Audit falling like a rock on my 1st node, quickly stopped the 2nd node… then re-read everything and set it up with a unique identity.

My 1st node is now suspended… and displaying MANY errors in the log like this:
2022-03-24T23:10:05.925-0500 ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb: database disk image is malformed”, “errorVerbose”: “ban
dwidthdb: database disk image is malformed\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).begin
SaveOrder.func1:722\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:661\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:685\n\tstorj.
io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj
.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/dr
pcctx.(*Tracker).track:52”}

Can anyone help? Is there a way to repair the disk image? My heart is sinking with the idea that it might just be gone after such a dumb mistake.

Thanks in advance.

2

But regarding audit - you can try to copy the blobs folder from the cloned node over the original one without replace (to allow only new files to be added), this could save your node from disqualification.
If you have removed the cloned node and its data, then you could only hope that the first one could survive.

2 Likes
3

Thanks Alexey! Suspension and Audit went back to 100% on their own. The errors continued, but I was able to use the article you wrote/linked to correct the problem. Very nice. Thank you.

In case it helps anyone else. I did get stuck on this line for a while, until I used an absolute path to the .sql file.

sqlite3 /storage/bandwidth.db “.read /storage/dump_all_notrans.sql”

It took a few hours to run. It would have taken days, but I copied it to a faster PC/storage and ran the rebuild there, then copied the new .db file back.

Thanks so much!

4

You could use a RAM disk, see