Could not send delete piece to trash

Node Operators troubleshooting
1

Hello,
I have important I/O slowness due to hardware RAID extension… I don’t want my node to be disqualified but I randomly get this kind of errors :

2023-03-07T19:17:15.880Z        ERROR   piecedeleter    could not send delete piece to trash    {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "D7ONH6J6RB3BBO5FWVZ6QGQ3XZVLZJTRB2IOF3JTK2ZKI2MFNBCA", "error": "pieces error: pieceexpirationdb: database is locked", "errorVerbose": "pieces error: pieceexpirationdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:368\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
[...]
2023-03-07T19:17:26.493Z        ERROR   piecedeleter    could not send delete piece to trash    {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "CVG4NNAET2PW7SA2MJXXGCUZRMGUIWO46ETZP5HNQQVTC63YW2QQ", "error": "pieces error: pieceexpirationdb: database is locked", "errorVerbose": "pieces error: pieceexpirationdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*pieceExpirationDB).Trash:112\n\tstorj.io/storj/storagenode/pieces.(*Store).Trash:368\n\tstorj.io/storj/storagenode/pieces.(*Deleter).deleteOrTrash:185\n\tstorj.io/storj/storagenode/pieces.(*Deleter).work:135\n\tstorj.io/storj/storagenode/pieces.(*Deleter).Run.func1:72\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:75"}
[...]
2023-03-07T19:23:41.403Z        ERROR   piecestore      failed to add bandwidth usage   {"Process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:829\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:778\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:792\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
[...]
2023-03-07T19:23:47.244Z        ERROR   piecestore      failed to add bandwidth usage   {"Process": "storagenode", "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:60\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).beginSaveOrder.func1:829\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:778\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:792\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}

Can I do something ? (Debian 11)

2

You can usually lower the speed of the raid rebuild to limit the impact on other processes. But you’d have to look up how that works for your specific raid setup.

3

This doesn’t solve your problem, but it might help to know that you wouldn’t get disqualified for either of those cases.

In the case of “could not send delete piece to trash”, the node will try to delete the piece again later, so there’s no problem—unless it is failing to delete every piece, or it always fails to delete that same piece, etc.

In the case of “failed to add bandwidth usage”, that means the node failed to record some bandwidth usage that it should have been paid for. If it can’t record the usage, it won’t be able to request payment for it. So you’ve lost some amount of profit, but that doesn’t adversely affect the network, so no disqualification.

2 Likes