I think I was just being audited by some satellites that were not really close to me. Since that time I am up to ~68% on the Success Rate. I will be migrating the data to a larger disk this weekend so I will be recreating my docker instance. I’ll re-post the stats a few days after the node maintenance to see if it is looking better. I am also fully confirmed by all the satellites now which may have been contributing.
2019-09-06T22:12:28.346682382Z 2019-09-06T22:12:28.346Z ERROR server gRPC stream error response {"error": "piecestore: rpc error: code = Canceled desc = context has been canceled", "errorVerbose": "piecestore: rpc error: code = Canceled desc = context has been canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:366\n\tstorj.io/storj/pkg/pb._Piecestore_Download_Handler:877\n\tstorj.io/storj/pkg/server.(*Server).logOnErrorStreamInterceptor:23\n\tgoogle.golang.org/grpc.(*Server).processStreamingRPC:1127\n\tgoogle.golang.org/grpc.(*Server).handleStream:1178\n\tgoogle.golang.org/grpc.(*Server).serveStreams.func1.1:696"}
2019-09-06T22:13:16.898870567Z 2019-09-06T22:13:16.898Z ERROR server gRPC unary error response {"error": "rpc error: code = PermissionDenied desc = untrusted peer 1QzDKGHDeyuRxbvZhcwHU3syxTYtU1jHy5duAKuPxja3XC8ttk"}So I had a few issues but it seems to be running now albeit with errors. Couple of comments:
1-Is it normal to have this gRPC error? It seems like some file upload fails randomly even though my connection is solid
2-Is it normal that the config.yaml file parameter don’t match that of the docker (bandwidth, storage size, user email,…)?
Thank you all for your help.
- You’re showing a very small part of the error, if you could get the logs through ssh and port them in text here we could have a closer look. But these are likely context cancelled errors which just means your node wasn’t among the fastest for that transfer and enough other transfers finished before yours did. This is normal.
- Yes, the run parameters overrule the config.yaml settings. You can ignore what it mentions there.
1-You are most likely right. Here are the details:
Anyhow it looks good and I have been running for 7 hours. I just need to be able to access the dashboard in GUI form instead of CLI and I’ll be all set.
Thank you for the prompt help!
Another rookie question but why is 99% of my data coming from one satellite only?
Thats the sat with with best Specs as i havet understod most of the data comes from this sat so far
Data is coming from customers of that satellite, not from a satellite itself. It’s just a catalog with nodes.
When customer want to upload something, the satellite will give a list of reliable nodes to them.
2019-12-19T19:11:40.841+0100 INFO piecestore uploaded {“Piece ID”: “SD67ZUL6ISM5IBW4PU4ZP4AXBO6SCI7EIST2RBP4KPDILWW2R2OQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”}
2019-12-19T19:17:16.540+0100 INFO version running on version v0.28.2
2019-12-19T19:32:15.930+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-19T19:32:15.958+0100 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 sending {“count”: 6}
2019-12-19T19:32:15.958+0100 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S sending {“count”: 628}
2019-12-19T19:32:15.958+0100 INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW sending {“count”: 408}
2019-12-19T19:32:15.958+0100 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs sending {“count”: 132}
2019-12-19T19:32:16.439+0100 INFO orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs finished
2019-12-19T19:32:16.579+0100 INFO version running on version v0.28.2
2019-12-19T19:32:16.915+0100 INFO orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 finished
2019-12-19T19:32:17.320+0100 INFO orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW finished
2019-12-19T19:32:31.185+0100 INFO orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S finished
2019-12-19T19:47:16.593+0100 INFO version running on version v0.28.2
2019-12-19T20:02:16.598+0100 INFO version running on version v0.28.2
2019-12-19T20:17:16.608+0100 INFO version running on version v0.28.2
2019-12-19T20:32:16.026+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-19T20:32:16.678+0100 INFO version running on version v0.28.2
2019-12-19T20:47:16.665+0100 INFO version running on version v0.28.2
2019-12-19T21:02:16.690+0100 INFO version running on version v0.28.2
2019-12-19T21:17:16.716+0100 INFO version running on version v0.28.2
2019-12-19T21:32:16.118+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-19T21:32:16.739+0100 INFO version running on version v0.28.2
2019-12-19T21:47:16.752+0100 INFO version running on version v0.28.2
2019-12-19T22:02:16.784+0100 INFO version running on version v0.28.2
2019-12-19T22:17:16.824+0100 INFO version running on version v0.28.2
2019-12-19T22:32:16.206+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-19T22:32:16.845+0100 INFO version running on version v0.28.2
2019-12-19T22:47:16.831+0100 INFO version running on version v0.28.2
2019-12-19T23:02:16.874+0100 INFO version running on version v0.28.2
2019-12-19T23:17:16.897+0100 INFO version running on version v0.28.2
2019-12-19T23:32:16.303+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-19T23:32:16.926+0100 INFO version running on version v0.28.2
2019-12-19T23:47:16.961+0100 INFO version running on version v0.28.2
2019-12-20T00:02:16.948+0100 INFO version running on version v0.28.2
2019-12-20T00:17:16.971+0100 INFO version running on version v0.28.2
2019-12-20T00:26:45.935+0100 INFO trust Scheduling next refresh {“after”: “8h45m48.924035118s”}
2019-12-20T00:32:16.401+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-20T00:32:17.019+0100 INFO version running on version v0.28.2
2019-12-20T00:47:17.032+0100 INFO version running on version v0.28.2
2019-12-20T01:02:17.048+0100 INFO version running on version v0.28.2
2019-12-20T01:17:17.079+0100 INFO version running on version v0.28.2
2019-12-20T01:32:16.496+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-20T01:32:17.169+0100 INFO version running on version v0.28.2
2019-12-20T01:47:17.130+0100 INFO version running on version v0.28.2
2019-12-20T02:02:17.149+0100 INFO version running on version v0.28.2
2019-12-20T02:17:17.182+0100 INFO version running on version v0.28.2
2019-12-20T02:32:16.582+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-20T02:32:17.197+0100 INFO version running on version v0.28.2
2019-12-20T02:47:17.226+0100 INFO version running on version v0.28.2
2019-12-20T03:02:17.263+0100 INFO version running on version v0.28.2
2019-12-20T03:17:17.317+0100 INFO version running on version v0.28.2
2019-12-20T03:32:16.685+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-20T03:32:17.312+0100 INFO version running on version v0.28.2
2019-12-20T03:47:17.324+0100 INFO version running on version v0.28.2
2019-12-20T03:54:23.329+0100 INFO piecestore upload failed {“Piece ID”: “NX7SKKHGLH6RXFOGCASVO6PCPBEYBX7ESFANWEGRIJIJFMRMKX6A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.356+0100 INFO piecestore upload failed {“Piece ID”: “N2TCJPJCR5C6HSXNNVGBNBHMIHNADD5GLJ3V672U66QIARL7Z6NA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.421+0100 INFO piecestore upload failed {“Piece ID”: “CQQ7HZO2IC4OUBMXELB433NZI5K5XGL7EU7EAGZ4AKESJT7J76KA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.424+0100 INFO piecestore upload failed {“Piece ID”: “GPBD66BJFXXXNBEDWLOUX2JZ2F6FAIECUWXWUXP7B6JC7AALAPRQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.431+0100 INFO piecestore upload failed {“Piece ID”: “YCRTZ6IGKB64CNN2MOLRX5YZNYMYTWE6HEODDE3TR4S26BVE6BSA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.579+0100 INFO piecestore upload failed {“Piece ID”: “IGV2OCACNQUYYP7RWSJIKEGKYJ64HLRIXSTKHCO4ZUNTNURXYURQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.580+0100 INFO piecestore upload failed {“Piece ID”: “HF7BCDEIGNWQCEQB4AUWOMJVHL3OQZN7NSGQGCIJODU2XLAPE5UQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.580+0100 INFO piecestore upload failed {“Piece ID”: “DX3VGGLTA3AI4QSISIU3B2J7HJNQEQQUCOE7PWZZITVYZHQRCJ2A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.641+0100 INFO piecestore upload failed {“Piece ID”: “RRK4H7XHSKGVNMK3V46LARNFPCYUQ4URJQ5Y35ZPBJ2L45KI7WDA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.672+0100 INFO piecestore upload failed {“Piece ID”: “U2NSUVD4XVP6U2RMLMHLINNCW2O7CHP3XQA5TXTV3XOV45VQD2WA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.707+0100 INFO piecestore upload failed {“Piece ID”: “PTVVBPQZJZGZ53RD7OT5CTZIUD7ZLKS7BVMLW5IDW4GXIZRW5JYA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.733+0100 INFO piecestore upload failed {“Piece ID”: “LB72DVGUSVIGSZRMIKWJAXH263ZTVEHCADCT66JULCR2RLGORO2A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.739+0100 INFO piecestore upload failed {“Piece ID”: “5OX3XBBQ3DT25NPHKGTM7J42ADKUP7SCQHAD5RX4O7W4S5MJLR3A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.740+0100 INFO piecestore upload failed {“Piece ID”: “YYPN2AN3VIIEAW7RLPRA2YXVIWTBXZPRZSYOHTT5MX5X7NZTQB4A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.765+0100 INFO piecestore upload failed {“Piece ID”: “ZYS6EQKNR3R62DO5C2XJ4G7KLC3ORVIQH5YRC27QCBJB6LJAFLQQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.807+0100 INFO piecestore upload failed {“Piece ID”: “OX7LOATXYL3CK7D3D4K7FIBJEFBOLHHCO3BNU7UNCNLLPJUZONXA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.809+0100 INFO piecestore upload failed {“Piece ID”: “ZRS323FL6IHUVIEKBDNUWCD65F6K6SDINSTLD5N27RFENPDBZCMQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.829+0100 INFO piecestore upload failed {“Piece ID”: “MSTMGUVTUOW7JD5QSEH7UG6KG4ODLU23VYYJ3BSCNMBXYEDMAG7A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.857+0100 INFO piecestore upload failed {“Piece ID”: “U646BRKLA5OTFS4S5FQTROJ7UJQXRHWNKXWWJAB4SQDURI42SFEA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.888+0100 INFO piecestore upload failed {“Piece ID”: “Z2ZXA5AXPXHWYTNCIWKRN7W4RTEOWDUWCJ64HRDM7CRQNWDCJBKA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.892+0100 INFO piecestore upload failed {“Piece ID”: “VBVSNJLDNXLAR3BNPYLSNU6FRLXMK52TF6PT5CUBFVX5DSEQYUAQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.898+0100 INFO piecestore upload failed {“Piece ID”: “X53HLQZ3QBE45IJFLVH5HDMCG6GG5V4YW6CFWAHY7DVZBBX4ZHBQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.902+0100 INFO piecestore upload failed {“Piece ID”: “VDWMMCGJKAOTY7Q7OFIBOKUVZMXPCJR2AGTKNX4WRCL5SO4PYKZQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.906+0100 INFO piecestore upload failed {“Piece ID”: “Z4U6W5NBNEJMPCVVIY2LC5BTHSUN7NUTJU6YNL4BKLUMRHNPJ25A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.925+0100 INFO piecestore upload failed {“Piece ID”: “FXSHNB6JR3ZSVTFKSMWLWFEOVSOW4VY7Q663GLSUXLXOVDTQPWZQ”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.927+0100 INFO piecestore upload failed {“Piece ID”: “N4S4ZBR7A7BHBH65FUD3AUE6RM4VQIBCONP4HH2EL643BHSX36NA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.928+0100 INFO piecestore upload failed {“Piece ID”: “7BCQP7GDZK4IHNCV5RTC3MFDPT2VN6DOVE67FO5C5JNIH3TVZNSA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.931+0100 INFO piecestore upload failed {“Piece ID”: “OYTKYONFY3JGIPQWYFD3STFYD65FKUWRR4FA5TPGLNT5XLRMD63A”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.933+0100 INFO piecestore upload failed {“Piece ID”: “R5DIQKMIJWC5BGXLKL2KXWH2BTUL3XFQ4775O6U36WAQVYF7MIWA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.935+0100 INFO piecestore upload failed {“Piece ID”: “IOSM2NF3IMQVMYDEE6PIHEVPUKSIEMGIITXJA3H2SKTFALRQV45Q”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T03:54:23.967+0100 INFO piecestore upload failed {“Piece ID”: “OZESBV7LRIG43QLIIRUDH56OIUB4XDMHQHK5JNDU5SSXATFKBYFA”, “Satellite ID”: “12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”, “Action”: “PUT”, “error”: “piecestore protocol: context canceled”, “errorVerbose”: “piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:346\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:225\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:985\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51”}
2019-12-20T04:02:17.349+0100 INFO version running on version v0.28.2
2019-12-20T04:17:17.367+0100 INFO version running on version v0.28.2
2019-12-20T04:32:16.772+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-20T04:32:17.410+0100 INFO version running on version v0.28.2
2019-12-20T04:47:17.413+0100 INFO version running on version v0.28.2
2019-12-20T05:02:17.833+0100 INFO version running on version v0.28.2
2019-12-20T05:17:17.453+0100 INFO version running on version v0.28.2
2019-12-20T05:32:16.865+0100 INFO bandwidth Performing bandwidth usage rollups
2019-12-20T05:32:17.493+0100 INFO version running on version v0.28.2
2019-12-20T05:47:17.523+0100 INFO version running on version v0.28.2
2019-12-20T06:02:17.523+0100 INFO version running on version v0.28.2
2019-12-20T06:17:13.516+0100 INFO Stop/Shutdown request received.
This is usual “Context canceled” from the current thread - your node too slow to get a piece.
lol?
bevore i get the update :
2019-12-16: recv: 40.17 GB sent: 22.51 GB
2019-12-17: recv: 32.25 GB sent: 21.98 GB
2019-12-18: recv: 26.28 GB sent: 18.57 GB
2019-12-19: recv: 24.35 GB sent: 17.77 GB
Hi all,
On my side, my 4 nodes are not working anymore since upgrade v0.31.12
| 2020-01-31T14:52:43.134Z | INFO | piecestore | upload started | {“Piece ID”: “ID546EHNEHGGQ6Z52EBJG5ABNLM7L3M7N27WUETFAP64FZJPG5XQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, “Action”: “PUT”} |
|---|---|---|---|---|
| 2020-01-31T14:52:43.772Z | INFO | piecestore | upload failed | {“Piece ID”: “ID546EHNEHGGQ6Z52EBJG5ABNLM7L3M7N27WUETFAP64FZJPG5XQ”, “Satellite ID”: “121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”, …} |
1 Like
Hi, yes exactly.
“error”: “context canceled”
Im having those errors for the last 6 hours.
Its perfectly normal then. Your node lost the race for the piece.
Im doing some monitoring using grafana and the graphs are showing a drastic change since last upgrade, all uploads are now failing. What could be the reason ?
Bandwith used evolution over the month has been really regular but not anymore,
1 Like
This could be the reason.
1 Like
I have encountered the same problem.
So my node is less welcome than before?