Node Suspension - serial already exists in store

Node Operators troubleshooting
, ,
1

Hello,

I’ve been a node operator since February of 2019, and today I’ve noticed my main node (~16.4TB stored) has been suspended from europe-north-1.tardigrade.io:7777 (12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB)

Looking around, I see this GitHub issue: GET REPAIR errors showing up on storage nodes · Issue #4687 · storj/storj · GitHub

The errors I’m receiving are similar to those presented there:

2022-04-19T17:37:22.102Z        INFO    piecestore      download started        {"Piece ID": "HFS3LO35OZYVZRQUCNAWIMLQVEUGQOD4Q45CRBDRIG6ZM3FIHGVA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR"}
2022-04-19T17:37:22.102Z        ERROR   piecestore      download failed {"Piece ID": "HFS3LO35OZYVZRQUCNAWIMLQVEUGQOD4Q45CRBDRIG6ZM3FIHGVA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "used serial already exists in store", "errorVerbose": "used serial already exists in store\n\tstorj.io/storj/storagenode/piecestore/usedserials.insertSerial:263\n\tstorj.io/storj/storagenode/piecestore/usedserials.(*Table).Add:117\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:498\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2022-04-19T17:37:27.431Z        INFO    piecestore      download started        {"Piece ID": "WWVN5WZNDJTM5KWW67ZSVK2ROSILG37CKY6UVE7XFK2YPH75LR4Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR"}
2022-04-19T17:37:29.324Z        INFO    piecestore      downloaded      {"Piece ID": "WWVN5WZNDJTM5KWW67ZSVK2ROSILG37CKY6UVE7XFK2YPH75LR4Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR"}

With the GitHub link I posted above, can someone from Storj confirm that I’m experiencing the same issue? I’m seeing similar errors within the ap1.storj.io (121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6) node as well.

Is there any actions I need to do on my end? My Uptime is going on 300h+, and QUIC is “OK”. I have 5 total nodes, and all appear to be exhibiting this issue (and they’re at different locations).

image
image667×1106 86.9 KB

2

Yes it’s the same. Long discussion here and implications also here.

Update for the satellites causing this issue already on the go.

Crossing fingers. :v:t2:

1 Like
3

If the issue still occurs today or later, please let storjlings know here.

4

suspension
suspension1025×338 29 KB

Hello all,

My node has been running for a little over a year now, but lately I’ve noticed that the suspension score from the satellites across the Atlantic have been sliding down. I really can’t figure out what is causing this. The US based satellites haven’t been affected.

This seems to be the typical error message:

2022-05-26T17:42:20.107Z ERROR piecestore download failed {“Process”: “storagenode”, “Piece ID”: “EQYRH2RNA6XROSM2OFUB2W4CJWLDRP5SAFDIFGTKAES22GZPWY4Q”, “Satellite ID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “GET_REPAIR”, “error”: “used serial already exists in store”, “errorVerbose”: “used serial already exists in store\n\tstorj.io/storj/storagenode/piecestore/usedserials.insertSerial:263\n\tstorj.io/storj/storagenode/piecestore/usedserials.(*Table).Add:117\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).verifyOrderLimit:76\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:498\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:122\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:112\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52”}

I’d like some help in diagnosing the problem.

Thanks in advance!