I do not understand! A lot of failed uploads

Node Operators troubleshooting
1

Goodmorning,
I have an active nod
e for months,

everything worked perfectly! Suddenly after upgrading to version 0.20.1, I have lots of failed uploads, but I’ve never lost DATA! I have only one HDD connected to a PC dedicated to STORJ!
How is it possible?

DASHBOARD
Storage Node Dashboard ( Node Version: v0.20.1 )

======================

ID 1xfv1DbmFi4dKkTALh4ArXEXcvjy9Lia3rgw2Y8g9pDJvmrYNU
Last Contact 0s ago
Uptime 79h1m1s

               Available         Used      Egress      Ingress
 Bandwidth        4.8 TB     185.7 GB     15.9 GB     169.9 GB (since Sep 1)
       
  Disk      439.5 GB     460.5 GB

Bootstrap bootstrap.storj.io:8888
Internal 127.0.0.1:7778
External my:28967

Neighborhood Size 364

LOG FILE
2019-09-16T13:58:35.895Z ERROR server gRPC stream error response {“error”: “piecestore protocol: rpc error: code = Canceled desc = context canceled”, “errorVerbose”: “piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:242\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:851\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-09-16T13:58:36.731Z INFO piecestore upload failed {“Piece ID”: “4M65ISV4O4UYFHZAQYTAGESBQP6EOFF2WBICQHHW74FEQLD5742A”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “piecestore protocol: rpc error: code = Canceled desc = context canceled”, “errorVerbose”: “piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:242\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:851\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-09-16T13:58:36.731Z ERROR server gRPC stream error response {“error”: “piecestore protocol: rpc error: code = Canceled desc = context canceled”, “errorVerbose”: “piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:242\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:851\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-09-16T13:58:38.073Z INFO piecestore upload started {“Piece ID”: “F2TLN7NK65MSWSXPQBN24RBVQ4FLZP4SGFCIZUNZQSESGZQB3WPA”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”}
2019-09-16T13:58:38.699Z INFO piecestore upload failed {“Piece ID”: “3HW75JKTM6OO3XDZ476WRUMLQQDQ2J325PKH647BXP4WORD5WQSA”, “SatelliteID”: “12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”, “Action”: “PUT”, “error”: “piecestore protocol: rpc error: code = Canceled desc = context canceled”, “errorVerbose”: “piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:242\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:851\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}
2019-09-16T13:58:38.700Z ERROR server gRPC stream error response {“error”: “piecestore protocol: rpc error: code = Canceled desc = context canceled”, “errorVerbose”: “piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:242\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:851\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696”}

========== AUDIT =============
Successful: 988
Recoverable failed: 290
Unrecoverable failed: 0
Success Min: 77.3082942097027%
Success Max: 100%
========== DOWNLOAD ==========
Successful: 12850
Failed: 6810
Success Rate: 65.3611393692777
========== UPLOAD ============
Successful: 23254
Rejected: 3780
Failed: 50728
Acceptance Rate: 86.017607457276
Success Rate: 31.4319699386337
========== REPAIR DOWNLOAD ===
Successful: 0
Failed: 0
Success Rate: 0
========== REPAIR UPLOAD =====
Successful: 0
Failed: 0
Success Rate: 0

I proceeded to make constant backups of the data on another disk, stopping the node, starting the backup and then restarting the node.

Not that the data shown by the dashboard is also incorrect.
170GB are downloaded for download in September, but actually less have been downloaded, around 10Gb of new data.

I don’t understand how it can be said sincerely.

Best Regards,

Tony

1 Like
2

Uploads are uploads to your system. So you do not have failed data, you just fail to receive the uploads.

This is probably because your setup loses the race with other systems. Pieces are uploaded to many systems at once

2 Likes
3

Yep and a change in behavior can be explained by a change in usage patterns on the network. Perhaps customers further away from your node have started uploading causing you to fail more uploads than normally.

4

As said before uploads are to your data and we all compete for them. More nodes = more nodes to compete with for the down and uploads. So what you see is that competition for that transfer. You have not lost any files and that is shown by your log

5

Same problem here… :frowning:

6

I fixed that for you :laughing:

3 Likes