Used_serial.db malformed

[ianllew600]

Hi All,

I was hoping someone could help?

I’ve recently had an issue with on of my db’s after my cache disk filled up to 100%, it seemed to have caused a malformation of the used_serial.db database.

I’ve followed the link within the forums to used sqlite to try and correct the db but the process is taking forever. Its been running for about 14 hours and has only complete 40Mb of the 2000Mb db. Is there any way of speeding this up as I think my node will become completely disqualified before the process has finished.

Also, if the db cannot be repaired what are my options as this node has been running for 9 months and has nearly $300 in held back money, will i be able to get this money or at least some of it should the node not be able to be recovered?

Thanks

Ian.

[BrightSilence]

Did you see this tip on that page as well?

Tip . You can use tmpfs to restore your databases. It uses memory instead of disk and should take a lot less time than on HDD (you can read more about usage of tmpfs with Docker in the Use tmpfs mounts guide).

You could also move the db file to an SSD and do the procedure there. But tmpfs is by far the fastest.

[ianllew600]

Hey,

Thaks for that, i’d didn’t see it no. I’ve taken a look through and i think its a little out of my depth. I was able to follow the instructions for the other method as they were very easy to follow.

Ian.

[Alexey]

It’s much simpler than you thought:

docker run -it --rm --mount type=tmpfs,destination=/ramdisk,tmpfs-size=1.5G --mount type=bind,src=Y:\storagenode2\storage\,dst=/data sstc/sqlite3 sh
/data # cp used_serial.db /ramdisk/
/data # sqlite3 /ramdisk/used_serial.db
sqlite> .mode insert
sqlite> .output /ramdisk/dump_all.sql
sqlite> .dump
sqlite> .exit
/data # rm /ramdisk/used_serial.db
/data # cat /ramdisk/dump_all.sql | grep -v TRANSACTION | grep -v ROLLBACK | grep -v COMMIT >/ramdisk/dump_all_notrans.sql
/data # rm /ramdisk/dump_all.sql
/data # sqlite3 /ramdisk/used_serial.db ".read /ramdisk/dump_all_notrans.sql"
/data # mv used_serial.db used_serial.db.bak
/data # cp /ramdisk/used_serial.db used_serial.db
/data # exit

[ianllew600]

@Alexey

exit

BrightSilence, Alexey,

So your advice was spot on i’ve been trying to sort this for 36hours, and with your suggestion and list of commands it was completed in an hour!!!

It looks like my node is back up and running, eccept i’m getting alot of errors that i’m not sure what they mean i’ve copied it below.

2020-06-12T18:32:47.547Z ERROR piecestore download failed {“Piece ID”: “RIBXI6UXEQN4U3I32A4QBBQVYXSNDBLBK6FPH7W6KL5HPBUXWUKA”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

If I look at the dashboard when ‘all satelites’ is selected there is no information in the window for ingress egress and with held amount is $0. If i select individual satlites then all the information for ingress, held amount etc is displayed for the satelite.

I think the error is still something to do with the used_serials db saying its locked but i’m not sure where to go next?

Do you have any other advice?

Thank you so much for your help so far.

Ian

[Alexey]

I think you have other malformed databases as well. Please, stop the storagenode and check all databases.

[ianllew600]

Hi Alexey,

So i did what you said and checked every db in the node. I found that bandwidth.db was malformed also and have repaired this. Also revocations.db when i integrity check it it says it was unable to open database.

Is revocations.db able to be fixed?

I have reloaded all the databases back into the node excpet revocations and they are all ok. The node dashboard is back up and all information is in it but i am still getting usedserialdb database locked. I check the used_serials.db again but it passes ok.

Is there anything else I can check?

Ian

[ianllew600]

This is the error within the log that i keep getting.

2020-06-12T22:21:24.340Z ERROR piecestore download failed {“Piece ID”: “GOBJJCJPNQYUYYXVJLCBAT7ZDM4CZMBDFQ7BQPKFTWZF35ZMBRKQ”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

Its strange because everything looks good everwhere else.

Ian

[Alexey]

@ianllew600

Is revocations.db able to be fixed?

Please stop storagenode, rename this database and start the storagenode back.

@ianllew600

usedserialsdb error: database is locked

Database is locked. What is the reason? What is the possible solution?

Short update from a meeting with the developers today about the db locking: The piece size is way too small and causing an IOPS bottleneck that we need to fix. I am not sure about the details but the long term goal is to write bigger pieces to disk. That should improve performance. Why do we store serial numbers for 48h in a DB? We should keep that in memory and don’t write it to disk at all. That will solve that problem entirely. We still want to write orders into a DB. If the node is restart…

and it’s fixed in the next version. At the moment can help only vacuuming:

"Action": "GET_AUDIT", "error": "usedserialsdb error: database is locked",

I’m running a docker version on Debian GNU/Linux. Here’s the procedure I use to vacuum and check the databases: Stop the node Vacuum the dbs integrity_check the dbs Restart the node Here’s a bash check which should do that for you. Please change the database directory to reflect where yours are on your system. docker stop -t 300 storagenode && dbs=$(ls /opt/storj/storage/*.db) c1="VACUUM;" c2="PRAGMA integrity_check;" for i in $dbs do sqlite3 $i "$c1" done for i in $dbs do sqlite3 $i "…

[ianllew600]

Morning Alexey,

I ran the VACUUM script and everything reported OK, i’ve started the node backup and everything seems to be working but i’m getting a lot of database locked errors, i’m hoping like you say it’ll be fixed in the next version.

I’ll leave the node up and running now and keep my fingers crossed that it doesn’t get disqualified and that the suspensions get lifted.

Unless there’s anything else you can think of that my help?

I just wanted to say thank you for helping me with my problems it was very much appriciated.

Ian.

[Alexey]

@ianllew600

i’ve started the node backup

What do you mean by this? Is it a data backup too?
If so - is your node running at least once since the backup? If so - it can be disqualified because of lost data since backup

[ianllew600]

Sorry bad English. I didn’t mean backup, i meant i’ve turned it on to start normal operation.