Node consistently errors

Node Operators troubleshooting
1 
04.619Z       ERROR   piecestoreupload failed    {"Process": "storagenode", "Piece ID": "R3V7SVT55AW66GH25Z7RVL4MER2NNMYNFJUFOXNFRB5I7PR3VQPA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/rpc/rpcstatus.Wrap:75\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload.func5:498\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:529\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:243\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:61\n\tstorj.io/common/experiment.(*Handler).HandleRPC:42\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:124\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:66\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:114\n\tstorj.io/drpc/drpcctx.(*Tracker).track:35", "Size": 768, "Remote Address":

This is consistently happening in one of 2 nodes. This has caused the dashboard to go down and when the node is running the dashboard gets stuck on loading. The logs is getting spammed with errors similar to that.

2

After waiting about 15minutes for it to load, the disk usage chart is empty and the last contact was 3m ago.

3

Looks like a usual long tail cancelation error: your node is too slow.
Almost any “context canceled” is meaning that your node is too slow to provide a piece.
This is only matter when it has this king error (usually you do not see them ever) for GET_AUDIT or GET_REPAIR, all others are usual behavior.

4

If my node is too slow, how do I make it quicker?

5

Depends. What is your current disk connection? If it’s a network filesystem or USB, try to make it internal, or use at least iSCSI for a network attached drives.
If your HDD is SMR (see PSA: Beware of HDD manufacturers submarining SMR technology in HDD's without any public mention), try to replace it to CMR if possible, or add another one node to your network - this will reduce a load on SMR.

But in general - you cannot be close to everyone customer in the world. Part of races is impossible to win.

1 Like