Satellite locations?

Sasha · September 19, 2019, 9:38am

It would be great if we can get a location where each satellite is located geographically.

"satellites": [
      "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",
      "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",
      "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",
      "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW"
    ],

I’ve noticed 118U successfully uploads to my node quite frequently, but 118U has stopped uploading and 12L9 has taken over but I have less than 5% successful upload due to all upload started fail instantly with an error:

2019-09-19T09:36:17.208039779Z 2019-09-19T09:36:17.207Z INFO    piecestore      upload started  {"Piece ID": "3WSO6IJSEQ2DISIEFH4TZODPN2S4L4V7OI765GQPZFFLGYVAZ7AQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-19T09:36:19.009390830Z 2019-09-19T09:36:19.009Z INFO    piecestore      upload failed   {"Piece ID": "3WSO6IJSEQ2DISIEFH4TZODPN2S4L4V7OI765GQPZFFLGYVAZ7AQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:242\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:851\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-19T09:36:19.009673288Z 2019-09-19T09:36:19.009Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:242\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:851\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}

sudo docker logs storagenode 2>&1 | grep '"PUT"' | grep uploaded -c
171
sudo docker logs storagenode 2>&1 | grep '"PUT"' | grep "upload failed" -c
18784

nerdatwork · September 19, 2019, 11:12am

118 is in Germany

As Storj isn’t mining you cannot expect a constant flow of data. In the future SNOs will have an option to pick a satellite to work with which will lead to lesser failures of upload/download. Knowing geolocation of satellites won’t be of much help unless you have access to different locations around the world to setup your node closer to these satellites.

BrightSilence · September 19, 2019, 2:37pm

While the satellites are segmented per region, it is actually the location of the uplink that matters for the transfer speed. That said, here are the satellites with their tardigrade addresses, which give away their location.

12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S	us-central-1.tardigrade.io:7777
12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs	europe-west-1.tardigrade.io:7777
121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6	asia-east-1.tardigrade.io:7777
118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW	satellite.stefan-benten.de:7777 (Germany)

Sasha · September 20, 2019, 8:30am

@nerdatwork, @BrightSilence, Thanks for the info.

The problem I see here is that my storagenode is just getting hammered with 99% upload started then upload failed immediately by satellite 12L9. I’ve just checked stats and 6 uploaded successfully out of 15,983.

I’m not entirely sure how storage nodes are currently being selectively picked but I somehow feel like the storagenodes are not being picked at all and the upload request is simply sent to all storagenodes and whoever returns uploaded first makes it.

To me that makes no sense, if this is what we’re currently doing.
I thought satellite’s would assess the performance and score of a storage node and then select that storage node for the upload request, rather than being a horse race for all which is inefficient use of SNO’s.

Here is just a small sample of last few requests…

2019-09-20T08:23:54.976086447Z 2019-09-20T08:23:54.975Z INFO    piecestore      upload started  {"Piece ID": "BV6HGSMRZWIG6A2BOMQT3G7DAT6XRMNQ7HYCWXE4VWVI5JISAEZA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:23:56.001896512Z 2019-09-20T08:23:56.001Z INFO    piecestore      upload started  {"Piece ID": "76KMTQ4PNAYQYNZVP4HACCXACVWCLA3QVSTD75UNOJNM5GDTFGTQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:23:56.673059120Z 2019-09-20T08:23:56.672Z INFO    piecestore      upload failed   {"Piece ID": "BV6HGSMRZWIG6A2BOMQT3G7DAT6XRMNQ7HYCWXE4VWVI5JISAEZA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:23:56.674092402Z 2019-09-20T08:23:56.673Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:23:57.673803775Z 2019-09-20T08:23:57.673Z INFO    piecestore      upload failed   {"Piece ID": "76KMTQ4PNAYQYNZVP4HACCXACVWCLA3QVSTD75UNOJNM5GDTFGTQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:23:57.674487047Z 2019-09-20T08:23:57.673Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:23:59.019514355Z 2019-09-20T08:23:59.019Z INFO    piecestore      upload started  {"Piece ID": "ZEVJGPTT3GGA3DYZQL237MIGME2FAO5KKLTLBJZOGJS3DJDFF7VQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:23:59.041238007Z 2019-09-20T08:23:59.041Z INFO    piecestore      upload started  {"Piece ID": "TN3G7DX7I7DKT4JXSBIW7NPMEIE3NCSLXYWVV66ASSFPMQMGKHKQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:00.617899434Z 2019-09-20T08:24:00.617Z INFO    piecestore      upload failed   {"Piece ID": "ZEVJGPTT3GGA3DYZQL237MIGME2FAO5KKLTLBJZOGJS3DJDFF7VQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:00.618505748Z 2019-09-20T08:24:00.617Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:02.873518884Z 2019-09-20T08:24:02.873Z INFO    piecestore      upload started  {"Piece ID": "P5WR4GMBZTULD6ULED4ZFV6LOXA45A5XZV732R7WUZD4CUNDKBPQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:04.631807358Z 2019-09-20T08:24:04.631Z INFO    piecestore      upload failed   {"Piece ID": "P5WR4GMBZTULD6ULED4ZFV6LOXA45A5XZV732R7WUZD4CUNDKBPQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:04.632792854Z 2019-09-20T08:24:04.631Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:13.018590574Z 2019-09-20T08:24:13.018Z INFO    piecestore      upload started  {"Piece ID": "6CQSBEWFX2ERQ33I5SOWJGBPVSRF7LJOG32YYC3A74G23AHG3E2A", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:13.833376190Z 2019-09-20T08:24:13.833Z INFO    piecestore      upload failed   {"Piece ID": "6CQSBEWFX2ERQ33I5SOWJGBPVSRF7LJOG32YYC3A74G23AHG3E2A", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:13.833823901Z 2019-09-20T08:24:13.833Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:16.333784487Z 2019-09-20T08:24:16.333Z INFO    piecestore      upload started  {"Piece ID": "W6UOPNTRV4LRMJAMU5NDYTHQZ3RR25OQAM4QG5HGW7F4OMO4J6EA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:18.186564671Z 2019-09-20T08:24:18.186Z INFO    piecestore      upload failed   {"Piece ID": "ZR4WJQK24HMWTSOX5MYPJQ6ATPVKH55MCM7MIZAW2JKP73GCORHQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:18.187313434Z 2019-09-20T08:24:18.186Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:19.277457602Z 2019-09-20T08:24:19.277Z INFO    piecestore      upload failed   {"Piece ID": "TN3G7DX7I7DKT4JXSBIW7NPMEIE3NCSLXYWVV66ASSFPMQMGKHKQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:19.278354399Z 2019-09-20T08:24:19.277Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:19.668435245Z 2019-09-20T08:24:19.668Z INFO    piecestore      upload started  {"Piece ID": "KJVMVFLLL5RQP5QRIWXR4DVP3RWVHOXA2Z4MP4G3TFHAOPJHSBNA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:21.880565287Z 2019-09-20T08:24:21.880Z INFO    piecestore      upload started  {"Piece ID": "LTLQQWMRRBXLUZSYFIO7DVZQ2SS5ODVNIMV5SR4KSPE4AIKH2YGA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:22.335960931Z 2019-09-20T08:24:22.335Z INFO    piecestore      upload failed   {"Piece ID": "KJVMVFLLL5RQP5QRIWXR4DVP3RWVHOXA2Z4MP4G3TFHAOPJHSBNA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:22.336286814Z 2019-09-20T08:24:22.335Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:23.031056128Z 2019-09-20T08:24:23.030Z INFO    piecestore      upload failed   {"Piece ID": "LTLQQWMRRBXLUZSYFIO7DVZQ2SS5ODVNIMV5SR4KSPE4AIKH2YGA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:23.031793880Z 2019-09-20T08:24:23.030Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:23.306564193Z 2019-09-20T08:24:23.306Z INFO    piecestore      upload started  {"Piece ID": "ZHK3NGQAVP6KBFJP2COOIIEFA2GV7QQVUTCAQYSCSPMGB7NO2XVQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:24.885226448Z 2019-09-20T08:24:24.884Z INFO    piecestore      upload failed   {"Piece ID": "ZHK3NGQAVP6KBFJP2COOIIEFA2GV7QQVUTCAQYSCSPMGB7NO2XVQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:24.891599130Z 2019-09-20T08:24:24.885Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:26.437475495Z 2019-09-20T08:24:26.437Z INFO    piecestore      upload started  {"Piece ID": "CRY3TRKIUADR6QKT6WFYBPI3SZ6XWCTIGJYDTOGF7EJ4MGF6SYCA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:26.660683410Z 2019-09-20T08:24:26.660Z INFO    piecestore      upload started  {"Piece ID": "CBECWTSVBD6EDFGWLWXEJU4YOZIHZOW6HNYQ33V3LHE6F3IEUBYA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:27.165034370Z 2019-09-20T08:24:27.163Z INFO    piecestore      upload started  {"Piece ID": "SAVNPKVUS2MZFNQAJK7QFOAGXO773GEBLUXJLEOEUOWB5ZNLJJKQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:28.265779610Z 2019-09-20T08:24:28.265Z INFO    piecestore      upload failed   {"Piece ID": "CBECWTSVBD6EDFGWLWXEJU4YOZIHZOW6HNYQ33V3LHE6F3IEUBYA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:28.266110801Z 2019-09-20T08:24:28.265Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:28.539274907Z 2019-09-20T08:24:28.539Z INFO    piecestore      upload started  {"Piece ID": "YW5FAUSJOVA7Z3EYEEIZQHV6BAQMUJTE7K2ANPGITKD5PKWRKLEA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:28.584163022Z 2019-09-20T08:24:28.583Z INFO    piecestore      upload failed   {"Piece ID": "SAVNPKVUS2MZFNQAJK7QFOAGXO773GEBLUXJLEOEUOWB5ZNLJJKQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:28.584524668Z 2019-09-20T08:24:28.584Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:29.984610778Z 2019-09-20T08:24:29.984Z INFO    piecestore      upload failed   {"Piece ID": "YW5FAUSJOVA7Z3EYEEIZQHV6BAQMUJTE7K2ANPGITKD5PKWRKLEA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:29.985364365Z 2019-09-20T08:24:29.984Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:31.464471045Z 2019-09-20T08:24:31.464Z INFO    piecestore      upload started  {"Piece ID": "6YRHHS5E2HYKFPCEN32Y2JYHU53NQE47C7P7KMG6O4DGWODQMFAQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:32.773333226Z 2019-09-20T08:24:32.773Z INFO    piecestore      upload failed   {"Piece ID": "6YRHHS5E2HYKFPCEN32Y2JYHU53NQE47C7P7KMG6O4DGWODQMFAQ", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:32.946856839Z 2019-09-20T08:24:32.773Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:33.397672994Z 2019-09-20T08:24:33.397Z INFO    piecestore      upload started  {"Piece ID": "5ON7HQPPPYYACQQU7AOA2BVJLSY6ALQCWX3L64AV3RQH4ZD6AB4Q", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-09-20T08:24:34.801693909Z 2019-09-20T08:24:34.801Z INFO    piecestore      upload failed   {"Piece ID": "5ON7HQPPPYYACQQU7AOA2BVJLSY6ALQCWX3L64AV3RQH4ZD6AB4Q", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:34.880765417Z 2019-09-20T08:24:34.801Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:36.333359435Z 2019-09-20T08:24:36.333Z INFO    piecestore      upload failed   {"Piece ID": "W6UOPNTRV4LRMJAMU5NDYTHQZ3RR25OQAM4QG5HGW7F4OMO4J6EA", "SatelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-20T08:24:36.333662431Z 2019-09-20T08:24:36.333Z ERROR   server  gRPC stream error response      {"error": "piecestore protocol: rpc error: code = Canceled desc = context canceled", "errorVerbose": "piecestore protocol: rpc error: code = Canceled desc = context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:267\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:151\n\tstorj.io/storj/pkg/pb._Piecestore_Upload_Handler:1070\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}

nerdatwork · September 20, 2019, 8:45am

This article should help you gather more info on how Storj works.

Sasha · September 20, 2019, 8:51am

We’ve just completed alpha and are now in beta for V3. Are you implying that all those features are currently 100% implemented and in place?

I’m just trying to understand why my SN is getting a log entry that upload started then upload failed, some entire in the log mentioned above are on the exact same second. Why does this occur?

BrightSilence · September 20, 2019, 9:26am

I think other than IP filtering it’s pretty much random. But for each piece 130 nodes are selected and 80 uploads are actually completed. The behavior your seeing might be indicative of someone trying to upload lots of tiny pieces. I may have asked already in a different topic, but how is your storage connected? Are you using SMB or NFS?

The statistical model outlined there is not yet in place. Right now it’s just the uptime and audit scores and I’m not even sure they are currently used for node selection.

Sasha · September 20, 2019, 9:34am

Synology NAS DS918+
2 x 12TB Exo drives SHR RAID 5, BTRFS.
Gbit Lan,
Fiber connection 950Mbps down, 450Mbps up
The NAS is currently only dedicated to StorJ at present and I have even added SSD cache (Read+write) the other day

Hmmm, that’s exactly as I suspected.

BrightSilence · September 20, 2019, 9:39am

An SHR1 array with 2 drives actually uses RAID1. It’ll only get converted to RAID5 if you add another drive. But I’m just nit-picking now.

We actually have really similar setups and I haven’t yet seen (or noticed) the behavior you’re describing. I’m not at home right now, but I’ll be sure to look into it when I get home. Since when are you seeing this?

Sasha · September 20, 2019, 9:44am

ops, I meant 1 HDD fault tolerance raid.

It seems to have started when 118U stopped uploading and then 12L9 took over. It just doesn’t make sense that I get a log entry that upload started and immediately failed. If you look at the snipped of the log above, there are some entries that occur on the exact same second.

So the upload failed already occurred before my storage node even got to write the log that an upload started then failed, haha.

Sasha · September 25, 2019, 7:33am

Did you have a look at your NAS if you were also getting a lot of failed upload since version 20.x?

BrightSilence · September 25, 2019, 7:45am

I did, but I’m not seeing the same behavior.

Unique · October 23, 2019, 12:30am

I see from the location data that that Satellite my node is dealing with is in Germany. This is crazy as one could not get a destination much further away from Australia. It does however explain why I’m getting 60% failures. My node can’t respond any faster as Germany is hundreds of ms away, the latency is massive. My logs show my node has never dealt with any other Satellite but the one in Germany. Asia Satellite would be better or the US one. How can I change my nodes behavior to deal with a closer Satellite?

John.A · October 23, 2019, 4:12am

We can do anything about that.

We just have to wait until more data comes out from other sattelites

BrightSilence · October 23, 2019, 7:06am

Your node accepts data from all tardigrade satellites by default, the one in Germany is just the most active right now as it’s used for most of the internal testing. Once more customers are onboarded you’ll see more traffic from other satellites as well.