What does this error normally mean?

Node Operators troubleshooting
1

For some reason this node stopped filling at 16GB (It has 550GB of storage).

This is what i’m seeing over and over in the logs:

2020-01-26T06:48:45.786Z INFO piecestore upload failed {“Piece ID”: “5TKKOKVPKM4FGMRI42EQL4S5T3WKUUIJZSXVS2KRL4DI7GJF3DTA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:389\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:268\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:1066\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}

2
3

However if you do not get any succesful uploads then there is a problem.
But you might need to share more information about your setup.

Also use the scripts to see your actual succes/fail rate. Are you on linux/docker or windows?

4

Linux - Using the docker container

Successful: 76
Recoverable failed: 0
Unrecoverable failed: 0
Success Rate Min: 100.000%
Success Rate Max: 100.000%
========== DOWNLOAD ==========
Successful: 1080
Failed: 836
Success Rate: 56.367%
========== UPLOAD ============
Successful: 28586
Rejected: 0
Failed: 403
Acceptance Rate: 100.000%
Success Rate: 98.610%
========== REPAIR DOWNLOAD ===
Successful: 0
Failed: 0
Success Rate: 0.000%
========== REPAIR UPLOAD =====
Successful: 171
Failed: 0
Success Rate: 100.000%

5

There’s absolutely nothing wrong with theses stats. Your node is fine.
Well ok the download is a bit low but might be your setup’s speed or your internet. Upload is good though.

6

The machine is on very stable internet and has low loads across the board, there’s no reason it should be having nearly 50% failure rate on downloads.

It’s strange because I have other nodes I setup the exact same way that are all working fine, but this one and one other have an identical problem. They both stopped downloading at around 15GB, and are both showing 50% download errors.

How do I make sure that none of the satellites disqualified me or something?

7

Are all nodes in the same location? Might be that this node is just a lot further away from the europe satellite which could result in it losing many download races.

So both nodes only got 15GB stored and are getting 50% success rate on downloads? Strange…

You check the dashboard of the nodes and click on every satellite. it’ll show you if you are disqualified.

8

How do I do this with the CLI dashboard? I don’t have the web dashboard enabled.

9

not possible with the cli dashboard afaik. You should enable the web dashboard. if you are concerned about safety then only run it on localhost, so it is not accessible from the network.

10

You can use these scripts: Script for Audits stat by satellites
Or just take satellites data

docker exec -i storagenode wget -qO - localhost:14002/api/dashboard | jq