Upload failed increased SIGNIFICANTLY over the last two days

tankmann · November 24, 2019, 8:02pm

Hey guys,

we were just chatting some days ago that sucessrates have gone up super high, just checking the node after like 3 days later. Download Success Rate is still at 99%, but Upload Success Rate dropped from 89% (9000 pieces) to 66% (20500 pieces)…

As you can see below looks like normal ‘My node is not fast enough’… uptime was 94h34m52s so since last update but in the beginning it looked all normal. Any ideas?
Dashboard also okayish:

2019-11-24T19:57:15.672Z	INFO	piecestore	upload started	{"Piece ID": "F7XHRDZB735CLTFFQVQTBDQANHKFSY7KU4UWIRSJCYP4D4YN4EAQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:57:17.347Z	INFO	piecestore	upload started	{"Piece ID": "JYPBNF4PD42NLTOAVUURI2EJM4XW3XNJXMU6MABNZ3ATHFBRPSAA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:57:19.771Z	INFO	piecestore	upload failed	{"Piece ID": "IVPP3XXCR7QGDDDNY2R7YJGHH67RCW4UIM5KKKBHO2M2S47WESUA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:57:22.395Z	INFO	piecestore	upload failed	{"Piece ID": "VLXMTLNR674QNP67SXWJOPG7ROM4KLQHIG4MECABSBWO7MQCIHNA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:57:23.297Z	INFO	piecestore	upload failed	{"Piece ID": "F7XHRDZB735CLTFFQVQTBDQANHKFSY7KU4UWIRSJCYP4D4YN4EAQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:57:28.769Z	INFO	piecestore	uploaded	{"Piece ID": "ONCGYRASGDBJY4GSO7NF5UURPVZ224GLCP2UE5LCDYC64ZUETEAA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:57:30.743Z	INFO	piecestore	upload failed	{"Piece ID": "JYPBNF4PD42NLTOAVUURI2EJM4XW3XNJXMU6MABNZ3ATHFBRPSAA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:57:31.754Z	INFO	piecestore	upload started	{"Piece ID": "6LIUIWM7KONTWMXTWOQ6FWEEPZ3ED5H7OHTK7NN2HATQHHUX3ZIQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:57:34.345Z	INFO	piecestore	upload started	{"Piece ID": "XEBFLIS535FJQOOA3GUJJB5DLNAKNB6UML5KFCPCIGP4RFYGOXHA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:57:34.675Z	INFO	piecestore	uploaded	{"Piece ID": "XEBFLIS535FJQOOA3GUJJB5DLNAKNB6UML5KFCPCIGP4RFYGOXHA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:57:37.645Z	INFO	piecestore	upload failed	{"Piece ID": "6LIUIWM7KONTWMXTWOQ6FWEEPZ3ED5H7OHTK7NN2HATQHHUX3ZIQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:57:39.847Z	INFO	piecestore	upload started	{"Piece ID": "O5L7MGDOMJNP7EGZV3LDU23VULSFNL6ZBJSLQ6HBX2IKRJZRK2NQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:57:43.301Z	INFO	piecestore	upload started	{"Piece ID": "OVBR5AT3MFVYJNUA2XAHMWFUG5Y3235YVZOXXWGQ5AMYMEJSAPWQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:57:44.017Z	INFO	piecestore	uploaded	{"Piece ID": "OVBR5AT3MFVYJNUA2XAHMWFUG5Y3235YVZOXXWGQ5AMYMEJSAPWQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:57:47.839Z	INFO	piecestore	upload started	{"Piece ID": "DUEXBCRCORCSBWIJKGBOD466OKWORILTQ7SP5DIO4OBJOXXOJHRQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:57:56.147Z	INFO	piecestore	upload started	{"Piece ID": "FIXNI5VJZ2ZWWHY3A34W3M55DKAQVUKWXOBVM6WAM7AJN23L34XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:57:56.742Z	INFO	piecestore	upload started	{"Piece ID": "EQRCH6SOBKD4TJF6XUMHUBASYTJONUDRKO7NHERAEZFOIVIVLHEQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:57:56.785Z	INFO	piecestore	uploaded	{"Piece ID": "EQRCH6SOBKD4TJF6XUMHUBASYTJONUDRKO7NHERAEZFOIVIVLHEQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:57:59.103Z	INFO	piecestore	upload failed	{"Piece ID": "O5L7MGDOMJNP7EGZV3LDU23VULSFNL6ZBJSLQ6HBX2IKRJZRK2NQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:00.553Z	INFO	piecestore	upload started	{"Piece ID": "OGKRRDWXXHHDJQHQU7GOFO2IW4DGGMLPTEWXXMAWXFQAIL47C3PQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:58:00.591Z	INFO	piecestore	upload failed	{"Piece ID": "DUEXBCRCORCSBWIJKGBOD466OKWORILTQ7SP5DIO4OBJOXXOJHRQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:00.681Z	INFO	piecestore	upload failed	{"Piece ID": "FIXNI5VJZ2ZWWHY3A34W3M55DKAQVUKWXOBVM6WAM7AJN23L34XQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:00.703Z	INFO	piecestore	uploaded	{"Piece ID": "OGKRRDWXXHHDJQHQU7GOFO2IW4DGGMLPTEWXXMAWXFQAIL47C3PQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:58:06.892Z	INFO	piecestore	upload started	{"Piece ID": "IVFCNTM3QGEGH2GCXVSTATBITEGEWG4AV6H4JZBB7PXO4MX3JAVA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:58:13.839Z	INFO	piecestore	upload started	{"Piece ID": "2JAOE55RXMF6OW2BO67HITXY6J6SSPPD2MLG5RD4JGUMYPU5QN2Q", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:58:18.373Z	INFO	piecestore	upload started	{"Piece ID": "5JODEKOWK4XGJBRU6MDMDGJW3NSCRXL3SYOLM7YO6EK5COV5X6EQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:58:21.407Z	INFO	piecestore	upload started	{"Piece ID": "L5G3353S7SB7XVRERDMSSCYXI26DKG7NKWY5VC7DAUMTNZZTT3WQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:58:23.762Z	INFO	piecestore	upload failed	{"Piece ID": "5JODEKOWK4XGJBRU6MDMDGJW3NSCRXL3SYOLM7YO6EK5COV5X6EQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:24.273Z	INFO	piecestore	upload failed	{"Piece ID": "L5G3353S7SB7XVRERDMSSCYXI26DKG7NKWY5VC7DAUMTNZZTT3WQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:25.240Z	INFO	piecestore	upload failed	{"Piece ID": "IVFCNTM3QGEGH2GCXVSTATBITEGEWG4AV6H4JZBB7PXO4MX3JAVA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:28.808Z	INFO	piecestore	upload started	{"Piece ID": "M5ZDOXRUFDF6EIMBFXJGFC5FQ64EHTTZPZFMJIO25U6N2X4GTGRA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:58:32.933Z	INFO	piecestore	upload failed	{"Piece ID": "M5ZDOXRUFDF6EIMBFXJGFC5FQ64EHTTZPZFMJIO25U6N2X4GTGRA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:32.979Z	INFO	piecestore	upload failed	{"Piece ID": "2JAOE55RXMF6OW2BO67HITXY6J6SSPPD2MLG5RD4JGUMYPU5QN2Q", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:37.857Z	INFO	piecestore	upload started	{"Piece ID": "45CMO47HF2L2JVB7FYWZIPGKCWWWO7N44RM3U6KX7W3IKWUSKQ3A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:58:38.985Z	INFO	piecestore	upload started	{"Piece ID": "46O7CA3ZGJCN7ZBY5TLUK6D56A7F5A2ACTLHU4FJITBIUU5VTUQA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:58:43.143Z	INFO	piecestore	upload failed	{"Piece ID": "45CMO47HF2L2JVB7FYWZIPGKCWWWO7N44RM3U6KX7W3IKWUSKQ3A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:45.556Z	INFO	piecestore	upload started	{"Piece ID": "OYHSNT2PMNDGXDB4QNDMWFI3P4CJ4ZRTILWPGI36RCJS46ZYM2ZQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:58:53.422Z	INFO	piecestore	upload failed	{"Piece ID": "46O7CA3ZGJCN7ZBY5TLUK6D56A7F5A2ACTLHU4FJITBIUU5VTUQA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:58:59.649Z	INFO	piecestore	upload started	{"Piece ID": "AR2YN4GMG2WXJYQS3DEHJ3YFTREYAR2YCG7TOBJBEXFGEXQ4E43Q", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-24T19:59:02.421Z	INFO	piecestore	upload failed	{"Piece ID": "OYHSNT2PMNDGXDB4QNDMWFI3P4CJ4ZRTILWPGI36RCJS46ZYM2ZQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2019-11-24T19:59:03.803Z	INFO	piecestore	upload started	{"Piece ID": "5UEJNIXE4R4SJAS3AHAHELDOV4GUWFVLFXCTHLU2T5DFBXYZBUMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:59:04.489Z	INFO	piecestore	upload started	{"Piece ID": "PDAZRJ56F7YKBN6VWBTMTCITUJ7BKGH3YRRWRT4HSPZKRMV4Z5UA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:59:05.007Z	INFO	piecestore	uploaded	{"Piece ID": "PDAZRJ56F7YKBN6VWBTMTCITUJ7BKGH3YRRWRT4HSPZKRMV4Z5UA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-24T19:59:08.689Z	INFO	piecestore	upload failed	{"Piece ID": "5UEJNIXE4R4SJAS3AHAHELDOV4GUWFVLFXCTHLU2T5DFBXYZBUMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "piecestore protocol: context canceled", "errorVerbose": "piecestore protocol: context canceled\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:297\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:176\n\tstorj.io/storj/pkg/pb.DRPCPiecestoreDescription.Method.func1:830\n\tstorj.io/drpc/drpcserver.(*Server).doHandle:175\n\tstorj.io/drpc/drpcserver.(*Server).HandleRPC:153\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:114\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:147\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}

Tulip · November 24, 2019, 9:28pm

Hi,

Update from here. Not much change since last. 1% down

========== AUDIT =============
Successful:           445
Recoverable failed:   0
Unrecoverable failed: 0
Success Rate Min:     100.000%
Success Rate Max:     100.000%
========== DOWNLOAD ==========
Successful:           49275
Failed:               3
Success Rate:         99.994%
========== UPLOAD ============
Successful:           84745
Rejected:             0
Failed:               928
Acceptance Rate:      100.000%
Success Rate:         98.917%
========== REPAIR DOWNLOAD ===
Successful:           0
Failed:               0
Success Rate:         0.000%
========== REPAIR UPLOAD =====
Successful:           155
Failed:               0
Success Rate:         100.000%

tankmann · November 24, 2019, 9:41pm

For me it doesn’t look good, maybe I was at the bottom list with my connection/speed and now more SNOs are there and ‘win over me’?
Dropped to 64.950%, it gets worse
Maybe test focus has changed, more uploads, less download traffic?

========== AUDIT ============= 
Successful:           284 
Recoverable failed:   0 
Unrecoverable failed: 0 
Success Rate Min:     100.000%
Success Rate Max:     100.000%
========== DOWNLOAD ========== 
Successful:           16000 
Failed:               10 
Success Rate:         99.938%
========== UPLOAD ============ 
Successful:           20830 
Rejected:             3 
Failed:               11241 
Acceptance Rate:      99.986%
Success Rate:         64.950%
========== REPAIR DOWNLOAD === 
Successful:           5 
Failed:               0 
Success Rate:         100.000%
========== REPAIR UPLOAD ===== 
Successful:           36 
Failed:               0 
Success Rate:         100.000%

Tulip · November 24, 2019, 9:46pm

What connection did you have? Time for a upgrade?

Have you try to reboot router/your node/etc…

hmmm

tankmann · November 24, 2019, 9:57pm

It’s a home ADSL connection 10 mbit/s up and 40mbit/s down… so not fast, but quite stable.
I’ll reboot the router and see if that changes anything but looks like it was online and connected for 10 days so no big changes there.
Router rebooted … no change.

JB273 · November 25, 2019, 1:32am

Hello guys,
Do you have a script for this output?
If so are you willing to share this or point where to find it?
Thank you very much in advance!

cameron · November 25, 2019, 2:35am

@tankmann
Could you try stopping the node and setting the storage2.max-concurrent-requests field in your config.yaml to something other than 0, maybe 50, and then restarting the container?

anon27637763 · November 25, 2019, 2:48am

My understanding is that too many concurrent requests would result in a “Rejection” entry in the log file. Since there aren’t that many rejections listed, changing the concurrency limit is unlikely to be a solution.

cameron · November 25, 2019, 3:05am

That’s correct. However, our current default is unlimited concurrent requests, so there would be no rejection logs if the node has not set a limit

tankmann · November 25, 2019, 9:18am

Thanks - but I thought through the ‘new way’ this was not needed any more, hence it was removed from the config / not used any longer.
Not sure what’s changed but looks like since this morning (didn’t really have time to check details) it’s back on track… maybe just the pressure went down significantly?

2019-11-25T08:45:14.959Z	INFO	piecestore	upload started	{"Piece ID": "7CA6MFCFX4YQGXTR6DI6L2C4W7V3MUOZ634QTGAAFLLIWODNQGSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-25T08:45:16.175Z	INFO	piecestore	download started	{"Piece ID": "HEGLXY4I6CKQDNHI3WMSDYVVE43PZEWCFMFJFPVN3EPKBRYYHRZA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-11-25T08:45:19.560Z	INFO	piecestore	upload started	{"Piece ID": "KVDOQX7VBX2U5FGEKNLF5QFD5FH3KRQ5IK3ZERHT4Q67I2FOUUZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-25T08:45:22.973Z	INFO	piecestore	downloaded	{"Piece ID": "HEGLXY4I6CKQDNHI3WMSDYVVE43PZEWCFMFJFPVN3EPKBRYYHRZA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-11-25T08:45:23.120Z	INFO	piecestore	uploaded	{"Piece ID": "KVDOQX7VBX2U5FGEKNLF5QFD5FH3KRQ5IK3ZERHT4Q67I2FOUUZA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-25T08:45:27.679Z	INFO	piecestore	upload started	{"Piece ID": "NPEMIM52HOQVRGM2F6CXSDATZUZI7LYTLXANN7BG6KRSKRCFNVVQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-25T08:45:28.000Z	INFO	piecestore	uploaded	{"Piece ID": "NG3WRK5IWSKBWP5ARNRLJUZPO4GZOZD2ZMFDMGQ22FSBFNDO7R4A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-25T08:45:28.301Z	INFO	piecestore	download started	{"Piece ID": "4XVMYPRNPCSU6257MTNNJTNEQNFHECKE4X45TAQ4UXVDHAWIZLJA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-11-25T08:45:30.026Z	INFO	piecestore	uploaded	{"Piece ID": "NPEMIM52HOQVRGM2F6CXSDATZUZI7LYTLXANN7BG6KRSKRCFNVVQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-25T08:45:30.056Z	INFO	piecestore	upload started	{"Piece ID": "A44DL23LMDHJBFQBUDQXQRMTL3YFD7SNCQWXOYQUHWHF3NQGTAFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2019-11-25T08:45:30.281Z	INFO	piecestore	uploaded	{"Piece ID": "7CA6MFCFX4YQGXTR6DI6L2C4W7V3MUOZ634QTGAAFLLIWODNQGSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2019-11-25T08:45:30.349Z	INFO	piecestore	upload started	{"Piece ID": "JCD7ZJCPWTNFW5GR7AUYTWH76MHCQYKFK37JQ5CV2CFJ22L5YSZQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:33.906Z	INFO	piecestore	downloaded	{"Piece ID": "4XVMYPRNPCSU6257MTNNJTNEQNFHECKE4X45TAQ4UXVDHAWIZLJA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-11-25T08:45:38.684Z	INFO	piecestore	uploaded	{"Piece ID": "JCD7ZJCPWTNFW5GR7AUYTWH76MHCQYKFK37JQ5CV2CFJ22L5YSZQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:40.991Z	INFO	piecestore	upload started	{"Piece ID": "T3EKDGKZUQE2K53CEUJCGWHKZOA5QZUL4XWC556AVJ46BOOVF5QA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:43.905Z	INFO	piecestore	uploaded	{"Piece ID": "T3EKDGKZUQE2K53CEUJCGWHKZOA5QZUL4XWC556AVJ46BOOVF5QA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:44.572Z	INFO	piecestore	uploaded	{"Piece ID": "A44DL23LMDHJBFQBUDQXQRMTL3YFD7SNCQWXOYQUHWHF3NQGTAFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2019-11-25T08:45:45.170Z	INFO	piecestore	upload started	{"Piece ID": "RCIDVWZPFV75ZUDFJ6T4DHESXZLIGZOVI4O6OMCCUUC5DKMPZWFA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:47.931Z	INFO	piecestore	uploaded	{"Piece ID": "RCIDVWZPFV75ZUDFJ6T4DHESXZLIGZOVI4O6OMCCUUC5DKMPZWFA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:54.499Z	INFO	piecestore	upload started	{"Piece ID": "FTCR44MEOPHQTGWU3TDS7LDP5SLT5TZ6MY4ZKKNE6CPXJGJALWSQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:56.010Z	INFO	piecestore	deleted	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "4XEWQHN6B6ETTH3NWFXXBPRMC3ENE3MNMJE7OHX6A6BCL4EOL6ZA"}
2019-11-25T08:45:59.810Z	INFO	piecestore	upload started	{"Piece ID": "CRQIVKZLLQLDODK65CD5NJC36TF2GB7VWXE4XRGKUHWUFN6LTEUA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:45:59.941Z	INFO	piecestore	upload started	{"Piece ID": "LRPFFSZSUI3UELDMDSHG57UGR73ISRHYKJIFZPEIV6QV5QSWDPDQ", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}
2019-11-25T08:46:01.729Z	INFO	piecestore	download started	{"Piece ID": "2PQW5R6PEWQTAUWDQ3SDY36GFRKKDW5BTII3UKQLSM5E5FV3W65A", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "GET"}
2019-11-25T08:46:02.077Z	INFO	piecestore	upload started	{"Piece ID": "FS4XWR5B5H4KK7JSDJQQL2XIZYSMIZDY4SIZN6GPU36W3QGIUWZA", "Satellite ID": "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "Action": "PUT"}

tankmann · November 25, 2019, 10:15am

Two things:

latest update, so it looks ok again but not so many pieces

image374×606 98.3 KB
When does the standard logfile restart again with docker? I restarted yesterday but a restart doesn’t remove logfiles right? I know I can output them to a file but for now didnt see the need… anyone know standard size / time for docker logs?

Thanks!

donald.m.motsinger · November 25, 2019, 1:08pm

I think it’s this one

github.com

ReneSmeekes/storj_success_rate/blob/da67babb103d9c9133061c91bf69217303fd0726/successrate.sh

#!/usr/bin/env bash
#A StorJ node monitor script: Initial version by turbostorjdsk / KernelPanick, adapted by BrightSilence, original grep statements by Alexey

LOG_SOURCE="${1}"

if [ -e "${LOG_SOURCE}" ]
then
	# the first argument is passed and it's an existing log file
	LOG="cat ${LOG_SOURCE}"
else
	# assumes your docker container is named 'storagenode'. If not, pass it as the first argument, e.g.:
	# bash successrate.sh mynodename
	DOCKER_NODE_NAME="${1:-storagenode}"
	LOG="docker logs $DOCKER_NODE_NAME"
fi

#Node Success Rates

echo -e "\e[96m========== AUDIT ============= \e[0m"
#count of successful audits

This file has been truncated. show original

JB273 · November 25, 2019, 7:37pm

Thank you for sharing, will try this today!

jocelyn · November 26, 2019, 3:36am

3 posts were split to a new topic: Bandwidth limitations and Node-per-ip

Alexey · November 26, 2019, 7:13am

2 posts were merged into an existing topic: Bandwidth limitations and Node-per-ip

BrightSilence · November 27, 2019, 9:54am

Logs are reset when you remove and recreate the container, which is also what happens when the node is updated. So manual recreations and updates is when the log resets.

tankmann · November 27, 2019, 1:17pm

Yeah I get that:

remove / recreate container = logreset
same for autoupdate

But my logs sometimes just restart even so uptime is then like 10 days or so.

nerdatwork · November 27, 2019, 2:04pm

Why not redirect to a file ?

BrightSilence · November 27, 2019, 2:10pm

Do you perhaps have log rotation set for the docker logger?

anon27637763 · November 27, 2019, 2:12pm

I recommend log redirection…

However, the log file grows rapidly. I have over 500 MB of logs for a two month time frame. So, for GNU/Linux hosts, I recommend log redirection and logrotate …

See logrotate man page for configuration details.