Suspended on Multiple Satellites, Database Locked Errors, Vacuum Script Unsuccesful

Node Operators troubleshooting
, ,
1

Hello -

I am runing v1.5.2.
Over the last few days, I’ve received several emails about my storage node being suspended on multiple satellites. Looking in the logs, I find multiple entries that include failed audits and ‘database locked’ errors, such as the following:

2020-06-16T01:14:57.373Z ERROR piecestore download failed {“Piece ID”: “YTPW7TAE4YQ5UCNQKE7RDH5VXPOGIB6IEKZANQ5UKLXGDCAEW4FA”, “Satellite ID”: “118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW”, “Action”: “GET_AUDIT”, “error”: “usedserialsdb error: database is locked”, “errorVerbose”: “usedserialsdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*usedSerialsDB).Add:35\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:459\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:1004\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

I should add that the logs also show many normal successful uploads and downloads as well.

From reading advice in these forums, I learned about the vacuum script, which I created and ran on my node…however, it just returns the following:

pi@raspberrypi:~ $ sudo ./vacuum.sh
storagenode
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 7: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
./vacuum.sh: 11: ./vacuum.sh: sqlite3: not found
storagenode

Any ideas?

Thanks!

2

It looks like you don’t have sqlite3 installed on that system. Try installing it by running the following command:
sudo apt install sqlite3

3

Much better results now!

pi@raspberrypi:~ $ sudo ./vacuum.sh
storagenode

ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
ok
storagenode
pi@raspberrypi:~ $

4

Database locks gone?

5

Unfortunately I’m still seeing database locked errors…now about the bandwidth database:
2020-06-16T20:19:19.119Z ERROR piecestore failed to add bandwidth usage {“error”: “bandwidthdb error: database is locked”, “errorVerbose”: “bandwidthdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:59\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:664\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:304\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:320\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:996\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:56\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:111\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:62\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:99\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

I just ran the vacuum script again…hopefully it will help…

Do you guys have any other tips? I really don’t want to get disqualified after working so hard to keep my node up and in good working order…

I should add that the logs don’t have all bad news…there are still many messages about successful uploads and downloads.

6

Your node will not be disqualified because of locking database. The other locks should resolve itself during the time.
However it indicates that your disk is little bit slow.
Is it SMR?