downed my Storagenode yesterday and upgraded my server…
got my node up and running again and everything looked okay… then today i was running successrate.sh
and got a near 1% failed rate on my downloads (egress) which is usually more like 0.3% (avg ipv4 failure rate).
i proceeded to check my logs, where i found some issues that made me slightly worried.
maybe disk latency, my ZFS ARC was still adapting after my reboot, or because i’m running with a setting of storage2.max-concurrent-requests: 14
I like my max-concurrent-requests at 14, it makes my disk backlogs much more reasonable.
peak disk backlog 99.9% of the time is below 20ms on my ssd L2ARC/SLOG and below 30ms on my raidz1 5 x hdd 7200rpm nonSMR vdev and my ARC/ZIL has 40-44GB RAM accessible to help manage everything.
After the Cleaning process is done the failed to add order issue vanishes.
i’m thinking some kind of code issue when it goes that high over the max concurrent-requests…
but what do i know 
i hope this isn’t an error that may adverse affect my node…
i’ve included a successrate.sh result of the full log below.
i’m sure it’s just the logs way of writing rejected in another way or something like that, so not really that important, but it seem like a novel error report which might be useful maybe figuring out another problem, and i wanted to verify my node is okay…
btw i looked after the audit check tool, but didn’t have much luck finding that, if anyone got a link for that would be great… i searched the documentation.storj.io and support.storj.io but neither gave me a result that i think i was looking for…
2020-04-23T16:59:15.082Z DEBUG orders cleaning
2020-04-23T16:59:15.436Z INFO piecestore upload started {"Piece ID": "ADEN6P3YJOZEXD6WQRY7RGY53BXW2YTQ5KNP4MWR2GEEIFGMBE3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1198557048324}
2020-04-23T16:59:15.663Z INFO piecestore upload started {"Piece ID": "WZD2U6XFRB3C4HC74Z75XPQDLQIXFWINS2QDUW2GMP247VGI52TQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198554728452}
2020-04-23T16:59:15.674Z INFO piecestore upload started {"Piece ID": "7V75JE5RIJQBCRTXWFDC7C46V2RJISO4RL5HENGG5TFWAVOONITA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198554728452}
2020-04-23T16:59:16.201Z INFO piecestore upload started {"Piece ID": "RY5MPX3CS4XYYZBE4SH7X4AEIVG7AMYMD3Y7KIPINHSTFZGQ3I2A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198552408580}
2020-04-23T16:59:17.206Z INFO piecestore upload started {"Piece ID": "I6DICKAJV4M4XXUZR5YUGT6DTZTTYSO4XIPQQS2OXQAIHZ3NEQRA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 1198546723076}
2020-04-23T16:59:18.280Z INFO piecestore upload started {"Piece ID": "5DODM2BRTDEXRZX4BTJY637YIHNAT2XWT2437AVVO33Y7F4ZGDNQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 1198544403204}
2020-04-23T16:59:19.144Z INFO piecestore upload started {"Piece ID": "ELS2ZBLXZKFYENOKOP5M2J25JMWBW2XLMENDRFY27CKJNLCCMRMQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198542083332}
2020-04-23T16:59:19.998Z INFO piecestore upload started {"Piece ID": "HTEIH2YOCHYZCB2OERT6ZQ7Z6LK2X6RMEZACJ2KVTI5XFWJDCHLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 1198539763460}
2020-04-23T16:59:20.775Z INFO piecestore upload started {"Piece ID": "BB3UFDKKPBKFIOPB27EONAO4RPBQEFCGOGS3LXYLFFISA4WLBVKA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198537443588}
2020-04-23T16:59:20.988Z INFO piecestore download started {"Piece ID": "P2CYISEMYYUQ4TLDQCMVUIXFNZ2FVW2WXEASJ25BRVBBYDUEWIUA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-04-23T16:59:21.468Z INFO piecestore upload started {"Piece ID": "AVTOYTJEOI7HAPM5ZRRQ7PVD3QMQ5DH27QMUAK6OHCVO2VF5JOHA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1198537443588}
2020-04-23T16:59:21.607Z INFO piecestore upload started {"Piece ID": "NGGPXW2L3TX3JQIMMG2LNXZGMRUQMJVW6KJRPRU6ZAXRQPHPEG6Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198535120388}
2020-04-23T16:59:22.762Z INFO piecestore upload started {"Piece ID": "M7FORCTEVQ4TRNJUTGKDJS2RC42UBKWR673GCFORJFLDBNCCKIGA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198534074628}
2020-04-23T16:59:22.905Z ERROR piecestore upload rejected, too many requests {"live requests": 15}
2020-04-23T16:59:23.182Z ERROR piecestore upload rejected, too many requests {"live requests": 15}
2020-04-23T16:59:23.991Z INFO piecestore download started {"Piece ID": "JIAYTLWCJYZZXMZ6TFSDZPE6JVIXFDFG72BJZA23UZTRAPOYPZLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-04-23T16:59:24.014Z ERROR piecestore upload rejected, too many requests {"live requests": 16}
2020-04-23T16:59:24.081Z INFO piecestore download started {"Piece ID": "PIOA4QIVLCYGJUNKAKIFSTWH5ZXVXCCW7KM4BWQG4UY2TZG6PGPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-04-23T16:59:24.292Z ERROR piecestore upload rejected, too many requests {"live requests": 17}
2020-04-23T16:59:24.682Z ERROR piecestore upload rejected, too many requests {"live requests": 17}
2020-04-23T16:59:24.773Z ERROR piecestore upload rejected, too many requests {"live requests": 17}
2020-04-23T16:59:24.776Z ERROR piecestore upload rejected, too many requests {"live requests": 17}
2020-04-23T16:59:25.205Z ERROR piecestore upload rejected, too many requests {"live requests": 17}
2020-04-23T16:59:25.374Z INFO piecestore download started {"Piece ID": "FSB4W4ZLTANJBXTZYQUWH7GBSCIAXCRRZFNT4IC3DHWUQ6PGRUUA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-04-23T16:59:25.560Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:25.561Z INFO piecestore uploaded {"Piece ID": "GHFUYHFMWUVWHIVGAWUV4YPO3LR4QNFICW3W6ZYBQCNY5NNCSIYQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:25.695Z ERROR piecestore upload rejected, too many requests {"live requests": 17}
2020-04-23T16:59:26.230Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:26.231Z INFO piecestore uploaded {"Piece ID": "ADEN6P3YJOZEXD6WQRY7RGY53BXW2YTQ5KNP4MWR2GEEIFGMBE3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2020-04-23T16:59:26.319Z ERROR piecestore upload rejected, too many requests {"live requests": 16}
2020-04-23T16:59:26.491Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:26.492Z INFO piecestore upload canceled {"Piece ID": "WZD2U6XFRB3C4HC74Z75XPQDLQIXFWINS2QDUW2GMP247VGI52TQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:26.501Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:26.501Z INFO piecestore uploaded {"Piece ID": "7V75JE5RIJQBCRTXWFDC7C46V2RJISO4RL5HENGG5TFWAVOONITA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:26.682Z INFO piecestore deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "QV5QRYDFNIV4AJFX2277AQ2DZ4PBUQ6WMW7V6AE5LGHYEGAJ7KYQ"}
2020-04-23T16:59:26.683Z INFO piecestore deleted {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "RFROCKS7G3CENOFQAJ5CFZDQRSL5J3QBXU6W2BZEOSG4RKU2KTTQ"}
2020-04-23T16:59:26.710Z INFO piecestore upload started {"Piece ID": "3IM3XAK2MU2CNBWNGFOBVJET23APWAZUV3SBQ6UOWHRVRUVQA4YA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 1198536394500}
2020-04-23T16:59:26.821Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:26.822Z INFO piecestore uploaded {"Piece ID": "RY5MPX3CS4XYYZBE4SH7X4AEIVG7AMYMD3Y7KIPINHSTFZGQ3I2A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:26.958Z INFO piecestore upload started {"Piece ID": "E52R7AVAAEPO2ODHQ2ODB4W6E4M7BMWKJ6FBM37RD2WRV3YGTGJQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198536394500}
2020-04-23T16:59:27.030Z ERROR piecestore upload rejected, too many requests {"live requests": 15}
2020-04-23T16:59:27.116Z ERROR piecestore upload rejected, too many requests {"live requests": 15}
2020-04-23T16:59:27.158Z ERROR piecestore upload rejected, too many requests {"live requests": 15}
2020-04-23T16:59:28.218Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:28.218Z INFO piecestore upload canceled {"Piece ID": "I6DICKAJV4M4XXUZR5YUGT6DTZTTYSO4XIPQQS2OXQAIHZ3NEQRA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:28.290Z INFO piecestore upload started {"Piece ID": "H5HFBXHSKRZXH4CZNB7FFUP3YT5MJ6INZEEAHZUDCMQC7MKU6OJA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 1198531754756}
2020-04-23T16:59:28.955Z ERROR piecestore upload rejected, too many requests {"live requests": 15}
2020-04-23T16:59:29.210Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:29.210Z INFO piecestore upload canceled {"Piece ID": "5DODM2BRTDEXRZX4BTJY637YIHNAT2XWT2437AVVO33Y7F4ZGDNQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:29.922Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:29.923Z INFO piecestore upload canceled {"Piece ID": "ELS2ZBLXZKFYENOKOP5M2J25JMWBW2XLMENDRFY27CKJNLCCMRMQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:30.004Z INFO piecestore upload started {"Piece ID": "N6EX7VTGRKFWVEE4SUVCUKLRALW6XIEOQKDQ2OWO2ENGHXTX5R3A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198529434884}
2020-04-23T16:59:30.236Z INFO piecestore upload started {"Piece ID": "4QXR3JBQHRSJLPB7ITO7XEUI2XW5J4LPLKQV6DNODE65WI7DMNAA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 1198529434884}
2020-04-23T16:59:30.371Z ERROR piecestore upload rejected, too many requests {"live requests": 15}
2020-04-23T16:59:30.623Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:30.624Z INFO piecestore uploaded {"Piece ID": "HTEIH2YOCHYZCB2OERT6ZQ7Z6LK2X6RMEZACJ2KVTI5XFWJDCHLQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT"}
2020-04-23T16:59:30.762Z INFO piecestore upload started {"Piece ID": "6WTDTCTUJIH2UYXGJZGTA6S3TC5Z7WSYHHNFWZAPJB7PM4WC4WUA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 1198526298372}
2020-04-23T16:59:31.511Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:31.511Z INFO piecestore uploaded {"Piece ID": "AVTOYTJEOI7HAPM5ZRRQ7PVD3QMQ5DH27QMUAK6OHCVO2VF5JOHA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT"}
2020-04-23T16:59:31.710Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:31.710Z INFO piecestore upload canceled {"Piece ID": "BB3UFDKKPBKFIOPB27EONAO4RPBQEFCGOGS3LXYLFFISA4WLBVKA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:32.174Z ERROR piecestore failed to add order {"error": "ordersdb error: database is locked", "errorVerbose": "ordersdb error: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*ordersDB).Enqueue:53\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).saveOrder:714\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:443\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:32.174Z INFO piecestore upload canceled {"Piece ID": "NGGPXW2L3TX3JQIMMG2LNXZGMRUQMJVW6KJRPRU6ZAXRQPHPEG6Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:32.760Z INFO piecestore upload started {"Piece ID": "AS3W5MSBZZZOIN2P5RC2JYWBT5M4AXZ5D7SZTSDZZJD3BTFYHEIQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198523978500}
2020-04-23T16:59:32.862Z INFO piecestore upload started {"Piece ID": "SKNK6457DPSJ6RLPLLIN26VFPF2FAX26VBMOMAD2UJZHNTFRTQBA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198523978500}
2020-04-23T16:59:33.000Z INFO piecestore upload started {"Piece ID": "VXAKHILTVZ3LGFYNYNCVMWDG546NDMYHG5C65OHK3S6RVFLEMUNA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "Available Space": 1198523978500}
2020-04-23T16:59:33.150Z INFO piecestore uploaded {"Piece ID": "M7FORCTEVQ4TRNJUTGKDJS2RC42UBKWR673GCFORJFLDBNCCKIGA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:33.151Z INFO piecestore upload canceled {"Piece ID": "3IM3XAK2MU2CNBWNGFOBVJET23APWAZUV3SBQ6UOWHRVRUVQA4YA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:33.158Z INFO piecestore uploaded {"Piece ID": "6WTDTCTUJIH2UYXGJZGTA6S3TC5Z7WSYHHNFWZAPJB7PM4WC4WUA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-04-23T16:59:33.160Z INFO piecestore uploaded {"Piece ID": "N6EX7VTGRKFWVEE4SUVCUKLRALW6XIEOQKDQ2OWO2ENGHXTX5R3A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:33.183Z INFO piecestore downloaded {"Piece ID": "PIOA4QIVLCYGJUNKAKIFSTWH5ZXVXCCW7KM4BWQG4UY2TZG6PGPQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-04-23T16:59:33.422Z INFO piecestore upload started {"Piece ID": "LS3WDI6WBE3WFXWTB66I2YFO4DR5QISBVIS3WOFEGEL6MBFHJOFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 1198523978500}
2020-04-23T16:59:33.505Z INFO piecestore uploaded {"Piece ID": "4QXR3JBQHRSJLPB7ITO7XEUI2XW5J4LPLKQV6DNODE65WI7DMNAA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-04-23T16:59:33.614Z INFO piecestore downloaded {"Piece ID": "P2CYISEMYYUQ4TLDQCMVUIXFNZ2FVW2WXEASJ25BRVBBYDUEWIUA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2020-04-23T16:59:33.724Z INFO piecestore uploaded {"Piece ID": "H5HFBXHSKRZXH4CZNB7FFUP3YT5MJ6INZEEAHZUDCMQC7MKU6OJA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT"}
2020-04-23T16:59:33.726Z INFO piecestore uploaded {"Piece ID": "AS3W5MSBZZZOIN2P5RC2JYWBT5M4AXZ5D7SZTSDZZJD3BTFYHEIQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:33.727Z INFO piecestore upload canceled {"Piece ID": "SKNK6457DPSJ6RLPLLIN26VFPF2FAX26VBMOMAD2UJZHNTFRTQBA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT", "error": "context canceled", "errorVerbose": "context canceled\n\tstorj.io/common/pb/pbgrpc.init.0.func3:70\n\tstorj.io/common/rpc/rpcstatus.Wrap:77\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).doUpload:452\n\tstorj.io/storj/storagenode/piecestore.(*drpcEndpoint).Upload:215\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:987\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:107\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:105\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:56\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:93\n\tstorj.io/drpc/drpcctx.(*Tracker).track:51"}
2020-04-23T16:59:33.739Z INFO piecestore uploaded {"Piece ID": "VXAKHILTVZ3LGFYNYNCVMWDG546NDMYHG5C65OHK3S6RVFLEMUNA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:33.744Z INFO piecestore uploaded {"Piece ID": "E52R7AVAAEPO2ODHQ2ODB4W6E4M7BMWKJ6FBM37RD2WRV3YGTGJQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "PUT"}
2020-04-23T16:59:33.838Z INFO piecestore downloaded {"Piece ID": "JIAYTLWCJYZZXMZ6TFSDZPE6JVIXFDFG72BJZA23UZTRAPOYPZLA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-04-23T16:59:33.840Z INFO piecestore download started {"Piece ID": "QJCYTR35GABSHGP75O6OIQEQ3TTN7XIZ2YPXGUIPJSXDCLCRWLNA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2020-04-23T16:59:33.867Z DEBUG orders cleanup finished {"items deleted": 49135}
successrate.sh result
========== AUDIT ==============
Critically failed: 0
Critical Fail Rate: 0.000%
Recoverable failed: 0
Recoverable Fail Rate: 0.000%
Successful: 181
Success Rate: 100.000%
========== DOWNLOAD ===========
Failed: 82
Fail Rate: 0.963%
Canceled: 64
Cancel Rate: 0.752%
Successful: 8365
Success Rate: 98.285%
========== UPLOAD =============
Rejected: 49
Acceptance Rate: 99.955%
---------- accepted -----------
Failed: 0
Fail Rate: 0.000%
Canceled: 21042
Cancel Rate: 19.490%
Successful: 86919
Success Rate: 80.510%
========== REPAIR DOWNLOAD ====
Failed: 0
Fail Rate: 0.000%
Canceled: 0
Cancel Rate: 0.000%
Successful: 4
Success Rate: 100.000%
========== REPAIR UPLOAD ======
Failed: 0
Fail Rate: 0.000%
Canceled: 240
Cancel Rate: 19.769%
Successful: 974
Success Rate: 80.231%
========== DELETE =============
Failed: 0
Fail Rate: 0.000%
Successful: 22381
Success Rate: 100.000%