"Action": "GET_AUDIT", "error": "usedserialsdb error: database is locked",

Hello I have updated to version 1.3.3 and everything has been ok.
But today when reviewing my dashboard an error appeared in the log, I have read that sometimes this error is fixed, doing a vacuum of the database, how is that done?

Piece ID": “S7TPOGKEFQ3RYPSIHDXMCGD52EPJMI2O2IETQDT6LHPHN2CA2SQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

I had the same once a few hours ago.

2020-05-04T02:53:39.336Z ERROR piecestore download failed {
"Piece ID": "6X4K2SSFE6NBTOTD5PZIFBEKYFMJMJU2QWBUMQQQDXQO5PUOMERQ", 
"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", 
"Action": "GET_REPAIR", "error": "usedserialsdb error: database is locked",
"errorVerbose": "usedserialsdb error: database is locked
  [followed by the same stack trace as above]
}
1 Like

I already posted this in another thread (No diskspace used in dashboard graph), but for completeness I add it here too:

I noticed that the disk usage only updates twice per day on my node, don’t know if that is intended. I saw some days with a big mismatch between remaining disk and disk usage.

I saw three different errors repeated a few times in the last few hours:

2020-05-04T01:59:56.112Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: database is locked”, “errorVerbose”: "bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59…

2020-05-04T02:00:30.675Z ERROR piecestore download failed {“Piece ID”: “73NTFGGVWPYR56LAIIUY5B4XCR23JRKV2B7S23WRHZJNXNAR4Q2Q”, “Satellite ID”: “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE”, “Action”: “GET”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: "usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35…

2020-05-03T01:00:43.841Z ERROR piecestore failed to add order {“error”: “ordersdb error: database is locked”, “errorVerbose”: "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue…

I restarted the node, it seems to be running just fine now, besides the dashboard showing wrong data. I wonder if that is only the local representation or am I loosing income or something?

Im seeing a similar disk space used graph issue on one of my nodes myself. I had no downtime.
I didnt find any “database locked” records in the logs though.

The issue seems to be related to the saltlake satellite only, as disk space used graphs of other satellites are looking good.

1 Like

Good spot! I bet the poor thing can’t keep up with all the data it has sent

I’m running a docker version on Debian GNU/Linux.

Here’s the procedure I use to vacuum and check the databases:

  • Stop the node
  • Vacuum the dbs
  • integrity_check the dbs
  • Restart the node

Here’s a bash check which should do that for you. Please change the database directory to reflect where yours are on your system.

docker stop -t 300 storagenode &&
dbs=$(ls /opt/storj/storage/*.db)
c1="VACUUM;"
c2="PRAGMA integrity_check;"
for i in $dbs
do
  sqlite3 $i "$c1"
done
for i in $dbs
do
  sqlite3 $i "$c2"
done
docker start storagenode

The output looks like this:

# ./vacuum-test.sh
storagenode
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
storagenode

If it works for you, and you have a low spec-ed node, it might be useful to run the script once a week.

You can automate that task using cron:

crontab -e

And add the following line… changing the script location as appropriate… to run every Sunday at 6:30

30 6 * * 0 /root/scripts/vacuum-databases.sh

9 Likes

uhm after using your script im getting this little error …


i hope it is only temporary ?

Did you get any errors?

No errors… But seems to be fine now without that weird number

Thank you for the script!

I have been suspended on several nodes and indeed found database locked records in my logs.

I have now run your script less than 2 hours ago and, since then, I have received again a few dozen database locked errors. But perhaps it is now much less errors than before I ran the script.

Anyway, my dashboard still shows that I am suspended on a couple of nodes and I would like to know whether running this script has produced the intended effect.

How long does it usually take for a node to move from suspended mode to regular operation? In other words, what is the process undertaken by satellites to check that a node can revert to regular operation?

Many thanks!

As the situation was getting worse (I got eventually suspended on all satellites), I unmounted the storj partition, converted it from ext3 to ext4 (dunno whether it really makes a difference?), ran e2fsck -fD /dev/sdX, ran the VACUUM/PRAGMA script again and now it seems that the node is working again perfectly. Hopefully this will last for a bit!


and

EXT4 has numerous improvements from EXT3. Including online defrag and support for large filesystems. EXT4 is much more appropriate for SNO purposes than EXT3. Your change was probably necessary.

For further reading:

Old Article on EXT3 vs. EXT4

1 Like

Hello,
I just want inform you than sometime the database is locked.

2020-05-09T16:21:48.448Z ERROR piecestore download failed {“Piece ID”: “LSXHAMQ6GLWTVOAOTE22ZPWNE7HNMDYSM23EH3WH7GONORHE55LQ”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:523\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Download:471\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:995\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:66\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

Then

2020-05-09T16:23:02.778Z INFO piecestore download started {“Piece ID”: “7EWCGASLUU2EO732PZA4UKHHJ3JV4BSTIDH7TKZGIGHGJSDNDGHA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”}
2020-05-09T16:23:14.956Z INFO piecestore downloaded {“Piece ID”: “7EWCGASLUU2EO732PZA4UKHHJ3JV4BSTIDH7TKZGIGHGJSDNDGHA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”}

without doing anything special.
Maybe it’s a writing problem when storj write data in the database in a same time.
I hope you will find a solution to correct it.
If i need do something tell me.
Have a nice day.

yeah i also get that, generally when the node is doing clear up or just when booting…
might be some timeout that is set a bit to low if i was to hazard an uneducated guess.

For my part it’s happens quite often.
Like today between 2020-05-09T00:00 and 2020-05-09T18:41 it’s happen 884 times for the satellite
118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW

known bug: https://github.com/storj/storj/issues/3871

Hi,

im having the same problems regarding the “database is locked” and tried @beast s vacuumscript.

I seem to miss something trivial, as I’m getting this error:

sudo sh vacuumtest.sh
storagenode
: not foundsh: 1: vacuumtest.sh:
vacuumtest.sh: 6: vacuumtest.sh: Syntax error: word unexpected (expecting “do”)

this is the .sh im running:

docker stop -t 300 storagenode &&
dbs=$(ls /mnt/storj8/storage/*.db)
c1=“VACUUM;”
c2=“PRAGMA integrity_check;”
for i in $dbs
do
sqlite3 $i “$c1”
done
for i in $dbs
do
sqlite3 $i “$c2”
done
docker start storagenode

what do I do wrong?

The script is bash script… other shells may need some modification.

You can add a “sha-bang” as the first line so that the correct shell is called.

I called my script vacuumtest.sh, but you can call it anything you’d like:

  • Copy and paste the script into a new file.
  • Add #!/bin/bash at the top so that it runs as a bash script
  • Name the file vacuumtest.sh
  • Set the execute bit: chmod +x vacuumtest.sh
  • Run the script ./vacuumtest.sh

You can name it something else if you’d like, but the process is the same

1 Like

Thank you for explaining.

I never realised that my node runs a different shell. It runs fine with the of the sha-bang,

thank you ver much.