Used_serial.db malformed

Hi All,

I was hoping someone could help?

I’ve recently had an issue with on of my db’s after my cache disk filled up to 100%, it seemed to have caused a malformation of the used_serial.db database.

I’ve followed the link within the forums to used sqlite to try and correct the db but the process is taking forever. Its been running for about 14 hours and has only complete 40Mb of the 2000Mb db. Is there any way of speeding this up as I think my node will become completely disqualified before the process has finished.

Also, if the db cannot be repaired what are my options as this node has been running for 9 months and has nearly $300 in held back money, will i be able to get this money or at least some of it should the node not be able to be recovered?

Thanks

Ian.

Did you see this tip on that page as well?

Tip . You can use tmpfs to restore your databases. It uses memory instead of disk and should take a lot less time than on HDD (you can read more about usage of tmpfs with Docker in the Use tmpfs mounts guide).

You could also move the db file to an SSD and do the procedure there. But tmpfs is by far the fastest.

Hey,

Thaks for that, i’d didn’t see it no. I’ve taken a look through and i think its a little out of my depth. I was able to follow the instructions for the other method as they were very easy to follow.

Ian.

It’s much simpler than you thought:

docker run -it --rm --mount type=tmpfs,destination=/ramdisk,tmpfs-size=1.5G --mount type=bind,src=Y:\storagenode2\storage\,dst=/data sstc/sqlite3 sh
/data # cp used_serial.db /ramdisk/
/data # sqlite3 /ramdisk/used_serial.db
sqlite> .mode insert
sqlite> .output /ramdisk/dump_all.sql
sqlite> .dump
sqlite> .exit
/data # rm /ramdisk/used_serial.db
/data # cat /ramdisk/dump_all.sql | grep -v TRANSACTION | grep -v ROLLBACK | grep -v COMMIT >/ramdisk/dump_all_notrans.sql
/data # rm /ramdisk/dump_all.sql
/data # sqlite3 /ramdisk/used_serial.db ".read /ramdisk/dump_all_notrans.sql"
/data # mv used_serial.db used_serial.db.bak
/data # cp /ramdisk/used_serial.db used_serial.db
/data # exit
3 Likes

BrightSilence, Alexey,

So your advice was spot on i’ve been trying to sort this for 36hours, and with your suggestion and list of commands it was completed in an hour!!!

It looks like my node is back up and running, eccept i’m getting alot of errors that i’m not sure what they mean i’ve copied it below.

2020-06-12T18:32:47.547Z ERROR piecestore download failed {“Piece ID”: “RIBXI6UXEQN4U3I32A4QBBQVYXSNDBLBK6FPH7W6KL5HPBUXWUKA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

If I look at the dashboard when ‘all satelites’ is selected there is no information in the window for ingress egress and with held amount is $0. If i select individual satlites then all the information for ingress, held amount etc is displayed for the satelite.

I think the error is still something to do with the used_serials db saying its locked but i’m not sure where to go next?

Do you have any other advice?

Thank you so much for your help so far.

Ian

I think you have other malformed databases as well. Please, stop the storagenode and check all databases.

Hi Alexey,

So i did what you said and checked every db in the node. I found that bandwidth.db was malformed also and have repaired this. Also revocations.db when i integrity check it it says it was unable to open database.

Is revocations.db able to be fixed?

I have reloaded all the databases back into the node excpet revocations and they are all ok. The node dashboard is back up and all information is in it but i am still getting usedserialdb database locked. I check the used_serials.db again but it passes ok.

Is there anything else I can check?

Ian

This is the error within the log that i keep getting.

2020-06-12T22:21:24.340Z ERROR piecestore download failed {“Piece ID”: “GOBJJCJPNQYUYYXVJLCBAT7ZDM4CZMBDFQ7BQPKFTWZF35ZMBRKQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

Its strange because everything looks good everwhere else.

Ian

Please stop storagenode, rename this database and start the storagenode back.


and it’s fixed in the next version. At the moment can help only vacuuming:

Morning Alexey,

I ran the VACUUM script and everything reported OK, i’ve started the node backup and everything seems to be working but i’m getting a lot of database locked errors, i’m hoping like you say it’ll be fixed in the next version.

I’ll leave the node up and running now and keep my fingers crossed that it doesn’t get disqualified and that the suspensions get lifted.

Unless there’s anything else you can think of that my help?

I just wanted to say thank you for helping me with my problems it was very much appriciated.

Ian.

What do you mean by this? Is it a data backup too?
If so - is your node running at least once since the backup? If so - it can be disqualified because of lost data since backup

Sorry bad English. I didn’t mean backup, i meant i’ve turned it on to start normal operation.