Ordersdb error: database is locked

May be it can be helpful:

I easy reproduce “database is locked”, just start storagenode after regular OS update.

System (docker host):

Inside container from the same period:

CPU: 8 Cores
Disk: ZFS raidz1 8хHDD + SSD LOG + SSD cache (ultra fast storage)

And I got this errors during node start:
docker logs storagenode 2>&1 | grep database

2019-11-26T10:32:15.829Z        ERROR   orders  archiving orders        {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).archiveOne:238\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Archive:202\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches.func2:213\n\tstorj.io/storj/storagenode/orders.(*Service).handleBatches:237\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func1:164\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-11-26T10:32:48.841Z        ERROR   piecestore      failed to add order     {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:52\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:625\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload.func4:584\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doDownload:609\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:391\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:1096\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}

As you can see on the screenshot, we have heavy IO wait on a short time period on ultrafast storage (I afraid on single disk we have catastrophic). This node is 6+ month age and slite database files have a big size.

4 Likes