Since this update, I am getting an incredibly number of upload and download “failed” entries in my log

Node Operators troubleshooting
1

Since this update, I am getting an incredibly number of upload and download “failed” entries in my log. Once in a while an upload or download will finish successfully. I haven’t changed anything at all in my setup. Has anyone else figured this out yet?

2

you are not the only one in the same boat

3

same problem on my node

4

What errors do you see, be more specific, may be part of logs will help to developers.

5

I posted above 1 day 5 hours that is a new event since ver. 0.24.5 never before has it happened over 10k failed downloads it’s not right

6

did you update windows? firewall? pc firewall is on?

7

@mitchellds
The most errors is:

2019-11-03T19:41:54.191743332Z 2019-11-03T19:41:54.191Z INFO    piecestore      upload failed   {"Piece ID": "CY4HBNAR7FZZ35T36EAF3HLU7AWIJC56R7EKBM5LOAPXQDIDX2QA", "SatelliteID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:288\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:162\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-11-03T19:41:54.191809036Z 2019-11-03T19:41:54.191Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:288\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:162\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}

Is it true?

8

Yes. Those are the consistent gRPC errors I am getting along with 90% or so download and upload failures. No windows updates, no firewall changes, etc.

9

Looks like you have low speed or big delay, someone was faster than you, i also have them, but not so mutch, but i have 100/100Mbit network speed over phisical gigabit connection.

10

Not sure that the total amount of data received depends on the network speed. I have been doing some benchmark during last weeks, setting up +5 nodes with VPN and best nodes in terms of amount of data received were these with VPN server located in Germany, receiving 10-20% more data than my base nodes without VPN.
This fact surprised me and i will continue doing test next weeks.

Update 1: all my nodes located in Europe (VPN and non-VPN) received same amount of data, around 30GB per day. Egress seems to be dependant on more parameters (between 10-25GB/day)