Odd suspension score

SGC · November 10, 2021, 8:22pm

over a long period now i’ve been seeing inconsistent suspension scores, they are very transient and usually very low deviations from the 100%, i have yet to find a real cause of these issues.

usually happens on node updates, after the most recent update i got this

95.25% on my near 16TB node, this is a massive deviation
so i decided to investigate further, so i ran a successrate.sh on my logs for the last 7 or so.
since nothing showed up in the current date… also pretty sure it happened today or very recently atleast.

its my understanding that suspension score drops from failed audits, so i’m a bit puzzles whats going on here… anyone else seeing something like this after updates or downtime without being able to define the cause…?

if it was less of a deviation i wouldn’t really think much about it, because it seems to happen a bit from time to time and wasn’t of any significance.

deathlessdd · November 10, 2021, 8:23pm

Suspension also can happen when a file isnt accessible had an issue on one of my nodes where permission wasnt the right owner and it happened randomly an my suspension score fell.

SGC · November 10, 2021, 8:30pm

guess i’m going back to my logs … lol

SGC · November 10, 2021, 9:08pm

2021-11-09T23:35:10.219Z    INFO    piecestore    download started    {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

2021-11-10T00:18:32.771Z ERROR piecestore download failed {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.( *Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(* Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.( *Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(* Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.( *Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(* Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.( *Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(* Tracker).track:52"}

seems like this has to be the cause.

however if i do a

grep '27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA' 2021-11-10-sn0001*
2021-11-10T00:18:32.771Z        ERROR   piecestore      download failed {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2021-11-10T02:18:53.487Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T02:20:37.603Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T02:51:30.198Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T02:51:36.200Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T03:26:16.729Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T03:26:20.291Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T03:55:52.330Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T03:56:30.893Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T04:12:39.970Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T04:18:36.754Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T04:18:56.824Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T04:33:46.829Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T05:35:33.335Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T05:35:35.559Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T05:36:51.170Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T05:37:56.556Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T05:45:58.634Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T05:46:43.389Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T06:48:27.432Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T06:49:16.934Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T07:07:30.163Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T07:45:39.676Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T07:46:02.781Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T07:46:02.890Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T07:47:02.680Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T07:59:21.370Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T07:59:30.783Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:02:55.421Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:15:09.854Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:15:10.586Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:27:17.628Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:27:47.822Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:34:05.770Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:34:17.118Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:56:47.580Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:59:05.241Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T08:59:20.558Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T09:14:42.768Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T09:14:50.208Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T09:15:11.976Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T09:15:22.694Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T09:33:13.545Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T09:34:53.079Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T09:40:24.395Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:05:45.093Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:05:51.247Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:12:32.775Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:12:44.217Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:29:57.330Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:30:41.701Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:34:08.449Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:34:22.807Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:34:35.628Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:34:49.881Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:35:01.862Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:35:02.469Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:35:17.976Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:35:17.988Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:35:32.511Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:40:58.639Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:45:17.677Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:46:41.714Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:47:11.596Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:48:26.563Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:49:15.111Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:51:03.769Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:51:37.217Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:51:38.855Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:53:58.116Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:57:13.278Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T10:59:52.850Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:01:56.270Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:01:59.560Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:03:40.475Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:10:30.293Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:16:23.334Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:19:03.396Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:20:28.180Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:22:34.619Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:29:55.115Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:29:55.777Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:32:04.924Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:32:05.894Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:39:10.637Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:39:15.306Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:48:35.543Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T11:48:38.593Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T12:00:24.813Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T12:00:53.192Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T12:14:55.243Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T12:15:46.461Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:24:37.381Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:25:24.959Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:30:55.677Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:36:11.024Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:36:19.859Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:48:03.636Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:55:44.568Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:57:18.780Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T13:57:19.539Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T14:05:46.445Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T14:05:59.247Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T14:12:14.810Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T14:14:27.247Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T14:46:44.608Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T14:47:07.091Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:26:12.483Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:26:20.931Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:31:13.217Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:31:54.720Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:43:17.622Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:43:50.647Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:46:35.313Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T15:47:14.778Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:02:12.919Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:02:23.742Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:02:26.508Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:02:38.397Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:02:40.558Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:02:54.701Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:03:50.838Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:03:52.331Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:29:42.478Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-10T16:30:10.976Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

it was accessed like 151 times just today.

SGC · November 10, 2021, 9:09pm

and if i check the 9th

grep '27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA' 2021-11-09-sn0001*
2021-11-09T06:46:38.871Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T06:53:32.783Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T06:59:54.693Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:03:40.815Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:13:17.509Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:13:35.221Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:32:12.663Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:32:21.997Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:43:46.627Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:46:04.810Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T07:58:33.049Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:01:03.176Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:01:27.959Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:01:36.065Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:06:39.835Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:08:44.843Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:19:16.764Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:19:52.328Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:27:26.098Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:30:01.967Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T08:49:27.878Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:01:25.813Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:01:40.038Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:45:53.493Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:46:26.787Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:46:51.623Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:46:53.832Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:52:12.309Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T09:53:02.217Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T10:12:56.267Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T10:13:52.286Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T10:22:19.136Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T10:23:50.231Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T10:36:50.226Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T10:37:08.834Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T11:00:27.190Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T11:00:38.887Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T11:07:54.780Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T11:08:09.366Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T11:25:20.510Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T11:39:39.610Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T11:42:39.416Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:15:47.200Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:16:21.087Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:26:30.009Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:26:44.625Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:38:35.645Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:40:00.941Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:42:24.799Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T12:42:41.398Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:07:16.812Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:08:02.817Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:09:07.794Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:09:19.320Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:16:01.698Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:16:11.309Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:17:14.332Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:17:15.777Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:17:17.577Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:17:17.756Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:17:22.760Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:17:25.795Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:19:32.858Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:22:15.734Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:26:48.152Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:27:12.146Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:27:13.483Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:28:22.658Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:41:32.710Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:44:42.004Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:56:10.498Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T13:57:04.621Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T14:01:33.194Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T14:15:56.956Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T15:03:12.021Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T15:03:19.151Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T15:35:29.191Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T15:36:34.674Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T15:59:04.091Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T15:59:04.835Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T16:01:05.743Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T16:01:06.510Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T16:01:09.825Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T16:01:10.238Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T16:56:33.880Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T16:58:17.689Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T17:06:38.376Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T17:06:59.971Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T17:16:08.427Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T17:20:32.177Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T17:21:01.013Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T17:47:50.895Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T17:48:23.910Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:03:39.217Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:03:43.348Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:07:56.884Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:08:20.844Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:15:47.379Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:15:59.063Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:27:22.625Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:27:43.968Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:52:34.625Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T18:54:04.034Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T19:32:14.096Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T19:32:16.503Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T19:32:38.196Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T19:34:31.532Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T21:34:12.613Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T21:34:36.161Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T22:25:19.956Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T22:26:54.374Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T22:27:43.958Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T22:27:51.752Z        INFO    piecestore      downloaded      {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}
2021-11-09T23:35:10.219Z        INFO    piecestore      download started        {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

so the file clearly exists and works, infact its most likely cached in my 400gb ssd
making it highly unlikely that this is a issue on part of my system.
has to be something else going on here.

had to cut down the log output a bit to size. lol

and ofc the system continues to operate normally for all other requests meanwhile…

SGC · November 10, 2021, 9:14pm

as exampled here, but the full log around the error… which doesn’t make sense

2021-11-10T00:18:09.396Z	INFO	piecestore	uploaded	{"Piece ID": "3PWSNGFRLAQSJPLPNNFGZEGTWSQUNLRK6IJGLTF5RAVZNCMTYJSQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 348928}
2021-11-10T00:18:10.274Z	INFO	piecestore	download started	{"Piece ID": "HLENFX2ZSKADVANBI27IRLQLPXOL5KQFTOO5L4NYR7IQG36U7VSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR"}
2021-11-10T00:18:10.387Z	INFO	piecestore	downloaded	{"Piece ID": "HLENFX2ZSKADVANBI27IRLQLPXOL5KQFTOO5L4NYR7IQG36U7VSQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR"}
2021-11-10T00:18:11.996Z	INFO	piecestore	download started	{"Piece ID": "OR4PU6HSGA5SN3XLUA2K6ITY4OEGC7PAKJYE3QGRSDYTEPOVUDHQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:12.293Z	INFO	piecestore	download started	{"Piece ID": "6IKYOEUCC2Z4FK3ECFBUIY3OG63HWP4VJQYAV5HJZAW7MZEIMR5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:14.237Z	INFO	piecestore	uploaded	{"Piece ID": "2GYFFHM7QO4LUED3GR5KS65OM5FHRE5XOMSPSAQHCIGOKQOZOLXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 181504}
2021-11-10T00:18:16.969Z	INFO	piecestore	downloaded	{"Piece ID": "OR4PU6HSGA5SN3XLUA2K6ITY4OEGC7PAKJYE3QGRSDYTEPOVUDHQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:17.076Z	INFO	piecestore	upload started	{"Piece ID": "3YSNY4Z267E6VWRJ2YOLYH3VQ2FKIZPDABI7MJT4PVWO4OXFMYXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 10630625492992}
2021-11-10T00:18:17.324Z	INFO	piecestore	download started	{"Piece ID": "E36MKMW5WNLCNQH2MQMYCNOPPLG3F4SO2Q2TYJEZTPGYAT7XOQVA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:17.391Z	INFO	piecestore	download started	{"Piece ID": "WI6AQ7NGAVM2PBJNQ4CB2BSP35N4HL4R43Q47LBQHK3ZWDDSFF6A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:17.547Z	INFO	piecestore	downloaded	{"Piece ID": "6IKYOEUCC2Z4FK3ECFBUIY3OG63HWP4VJQYAV5HJZAW7MZEIMR5Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:17.588Z	INFO	piecestore	uploaded	{"Piece ID": "3YSNY4Z267E6VWRJ2YOLYH3VQ2FKIZPDABI7MJT4PVWO4OXFMYXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 348928}
2021-11-10T00:18:18.508Z	INFO	piecedeleter	delete piece sent to trash	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "ES6NL4GAQHEOBE3BSF3T6IGCKFB3WYKO3EER4QX5Y5DOK2KISMKA"}
2021-11-10T00:18:18.807Z	INFO	piecedeleter	delete piece sent to trash	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "ALMDBQWXXK43C6HC6AOAPZWU4YMA4P7GNWDIL3QFVJTLMAS2QZJQ"}
2021-11-10T00:18:20.511Z	INFO	piecestore	upload started	{"Piece ID": "CUCJZI6AT26DCMLWSYEL75W35R3HMM3ADJVK3N4DX6RWNFT2ARDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 10630589579264}
2021-11-10T00:18:20.515Z	INFO	piecestore	uploaded	{"Piece ID": "CUCJZI6AT26DCMLWSYEL75W35R3HMM3ADJVK3N4DX6RWNFT2ARDA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 4608}
2021-11-10T00:18:22.538Z	INFO	piecestore	downloaded	{"Piece ID": "WI6AQ7NGAVM2PBJNQ4CB2BSP35N4HL4R43Q47LBQHK3ZWDDSFF6A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:22.563Z	INFO	piecestore	downloaded	{"Piece ID": "E36MKMW5WNLCNQH2MQMYCNOPPLG3F4SO2Q2TYJEZTPGYAT7XOQVA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:26.867Z	DEBUG	retain	About to move piece to trash	{"Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Piece ID": "7MHCBLEXPHWHEDLZ6IZFI67AVZWDGRO4NW62VWLR3AWAKYBQEONQ", "Status": "enabled"}
2021-11-10T00:18:28.725Z	INFO	piecestore	download started	{"Piece ID": "DJOR2OEW6XXK6TBNYFXFHLPUPOFLT5YKM5RBV75X7WMSN2FB5SYA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:28.766Z	INFO	piecestore	download started	{"Piece ID": "3ALKONFCRYR34KHUK2NRS2MZMGJUH4RCN2CE5CFFFZBAUUTY73JQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR"}
2021-11-10T00:18:30.723Z	INFO	piecestore	upload started	{"Piece ID": "K3K3JIPRGLX362EZBZSQYLWGBQDY4ZNMCUPSHE7FPOMTT4DNASKA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 10630498877440}
2021-11-10T00:18:30.944Z	INFO	piecestore	upload started	{"Piece ID": "5FHT6M2IJMCCGGHKBWG7BKBZX4LUS4PEN5EBOSYHUGQ23AYTYMJA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Available Space": 10630494945280}
2021-11-10T00:18:31.780Z	INFO	piecedeleter	delete piece sent to trash	{"Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "PPUDIKTD457PAGMWEZJORGJAVGG5TV2B6KS5P6XRU4B5OLDMCBTQ"}
2021-11-10T00:18:32.771Z	ERROR	piecestore	download failed	{"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2021-11-10T00:18:33.627Z	INFO	piecestore	upload started	{"Piece ID": "PH7XYA3O54P5AVUUA2V4HLTG76WME2NAV22Z7QAJL573XGDGJCIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 10630457196544}
2021-11-10T00:18:33.744Z	INFO	piecedeleter	delete piece sent to trash	{"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Piece ID": "VR2IBI4XHQAHEQKR7HGLVU2I7JPZJU5MW2V264NE7ECXE3ULP66A"}
2021-11-10T00:18:33.955Z	INFO	piecestore	downloaded	{"Piece ID": "DJOR2OEW6XXK6TBNYFXFHLPUPOFLT5YKM5RBV75X7WMSN2FB5SYA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:34.159Z	INFO	piecestore	download started	{"Piece ID": "YIPH44ZG3QQNOO2XFKGE6I7H7B4SMZW66YWBIRCTQKAJTHQH74MA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:34.544Z	INFO	piecestore	downloaded	{"Piece ID": "3ALKONFCRYR34KHUK2NRS2MZMGJUH4RCN2CE5CFFFZBAUUTY73JQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR"}
2021-11-10T00:18:34.939Z	INFO	piecestore	upload started	{"Piece ID": "LQDRCTOXARAZ2NEDEGWGZSMXSTCLPNQRPGDCRMENXJOGYD2EM6PQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 10630453526528}
2021-11-10T00:18:35.048Z	INFO	piecestore	uploaded	{"Piece ID": "LQDRCTOXARAZ2NEDEGWGZSMXSTCLPNQRPGDCRMENXJOGYD2EM6PQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 13568}
2021-11-10T00:18:35.122Z	INFO	piecestore	upload started	{"Piece ID": "V3EGW6X7ELFRG3TBITIVTM5MKT4W6T7LPIAGMDAT3M2JULS2IBIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 10630449594368}
2021-11-10T00:18:35.126Z	INFO	piecestore	uploaded	{"Piece ID": "V3EGW6X7ELFRG3TBITIVTM5MKT4W6T7LPIAGMDAT3M2JULS2IBIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 3840}
2021-11-10T00:18:37.256Z	INFO	piecestore	uploaded	{"Piece ID": "K3K3JIPRGLX362EZBZSQYLWGBQDY4ZNMCUPSHE7FPOMTT4DNASKA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "Size": 181504}
2021-11-10T00:18:37.992Z	INFO	piecestore	uploaded	{"Piece ID": "PH7XYA3O54P5AVUUA2V4HLTG76WME2NAV22Z7QAJL573XGDGJCIQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 2319360}
2021-11-10T00:18:39.682Z	INFO	piecestore	downloaded	{"Piece ID": "YIPH44ZG3QQNOO2XFKGE6I7H7B4SMZW66YWBIRCTQKAJTHQH74MA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:39.862Z	INFO	piecestore	download started	{"Piece ID": "E36MKMW5WNLCNQH2MQMYCNOPPLG3F4SO2Q2TYJEZTPGYAT7XOQVA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET"}
2021-11-10T00:18:40.288Z	INFO	piecestore	downloaded	{"Piece ID": "7L45TOP76B3WILVIBBACI763FB5OLVUUPSPNZHVQCQBAKKZZCIUA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET"}

SGC · November 10, 2021, 9:35pm

@littleskunk how is this possible.
i’m running zfs, on a server with ECC RAM, my ZFS pool is raidz1, has a SLOG and L2ARC.
my RAM and ZFS pool is scrubbed on a regular basis, and the ZFS reports no issues of any kind.

Seems very odd to me that the system just “forgets” to send the file

Alexey · November 10, 2021, 9:42pm

You should search for ERROR and (GET_AUDIT or GET_REPAIR), the GET doesn’t affect neither suspension nor audit score as far as I know.

Suspension affected when your node answers on audit request with unknown error. File not found, disk i/o error, timeout are known errors, so they will affect audit score directly.

BrightSilence · November 11, 2021, 10:19am

I don’t think it even can, since the satellite isn’t involved in that transfer.

Aside from just looking at GET_AUDIT and GET_REPAIR, you can assume that it won’t be errors regarding missing files, corrupt data being sent or time outs. As all of those would hit the audit score, not the suspension score.

Last note: 95.25% might sound low, but a single failure drops it to 95% and it slowly recovers from there. So it’s not an indication of consistent failures, just an indication that one happened recently. Hope that helps.

SGC · November 11, 2021, 10:56am

this is not the same node, just another example of how it usually is minor deviation, and seems random where it pops up, only when i restart nodes or update nodes and it will usually be gone again within the same day if not less.

the 95.25% suspension was also back to 100% when i checked it this morning.

ill try and go through my errors for the 10th, usually not that many, so should be possible.
and i sort of have already

i only have these errors for the entire day
did find a few more of those deadline exceeded

grep 'ERROR' 2021-11-10-sn0001*
2021-11-10T00:00:00.551Z        ERROR   piecestore      download failed {"Piece ID": "CTESSJTAHV54VGUXQQ6BFZGJFKL5DWFLYRBBRK6Q5HW3VGSNBKQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:35894: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:35894: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T00:13:07.343Z        ERROR   piecestore      upload failed   {"Piece ID": "4DVMDSRIIY4VGTEHBNSUOL4UXEYMBEUHUUSYXEOUDYHKREGB67AA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T00:18:32.771Z        ERROR   piecestore      download failed {"Piece ID": "27HV5AFLAVY4SDPJKUFAF6FKVSFGWGSZNAASLVAHQ6BZD6CXNYFA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2021-11-10T00:50:17.976Z        ERROR   piecestore      download failed {"Piece ID": "3K3VSPA36HDKGO3LEMLJ4MO37S3W25C5OF46DHPQWPQX4DXRL65Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->174.89.56.181:59973: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->174.89.56.181:59973: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T01:23:26.646Z        ERROR   piecestore      upload failed   {"Piece ID": "W7NKGOMWQ3YEBJ4JFG22Z7MOLJI5AFF3ROR3U3MMY3NABRQGNCMA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T02:13:07.868Z        ERROR   piecestore      download failed {"Piece ID": "DZPQWB227V2MOT6UK26YDCBQ75BAVUE2RTSCS6L2NVAV6UAJ7SKA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2021-11-10T02:22:40.483Z        ERROR   piecestore      upload failed   {"Piece ID": "WKWMQDCPY5Q2IEK6CAHIQH2PLMIMBYACYHKNWILZA54YXPZKLHKQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T02:27:27.224Z        ERROR   piecestore      download failed {"Piece ID": "DZPQWB227V2MOT6UK26YDCBQ75BAVUE2RTSCS6L2NVAV6UAJ7SKA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2021-11-10T04:35:23.658Z        INFO    piecestore      upload started  {"Piece ID": "ERRORAM6HASDF26J7VW72R7LJY6BH3V7HFIJHKT3N66EFYV6H3PQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Available Space": 10517221212160}
2021-11-10T04:35:23.662Z        INFO    piecestore      uploaded        {"Piece ID": "ERRORAM6HASDF26J7VW72R7LJY6BH3V7HFIJHKT3N66EFYV6H3PQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "Size": 2560}
2021-11-10T04:39:59.922Z        ERROR   piecestore      download failed {"Piece ID": "BR2QL4PJOAWG7QLD53VN7N4LT5JADQDFOQBVRTRWTM43M2EL5I4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.148.54.71:53232: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.148.54.71:53232: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T05:35:54.908Z        ERROR   piecestore      download failed {"Piece ID": "NBY42DHBAQ5C6LNOCYBUPHBBNIL2I62IQIMSUSKKTMNNQSTXWYUA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->145.40.73.87:48544: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->145.40.73.87:48544: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T06:33:58.370Z        ERROR   piecestore      download failed {"Piece ID": "EVK4A7Z2RPRGCY6UDJUJAHLBBCUUKWARDJNTS5BQFDBKB6O35N3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->145.40.73.87:35926: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->145.40.73.87:35926: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T06:45:46.738Z        ERROR   piecestore      download failed {"Piece ID": "XFOHHOXWQ5OCOZ6SKTRNBXVCG5VLTILJ3BNRBNI467ILSAJWDC5Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:43870: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:43870: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T07:23:10.011Z        ERROR   piecestore      download failed {"Piece ID": "EVK4A7Z2RPRGCY6UDJUJAHLBBCUUKWARDJNTS5BQFDBKB6O35N3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:50486: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:50486: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T07:27:06.686Z        ERROR   piecestore      download failed {"Piece ID": "NOEOCCQ3L5RASGTTSXHWYACHFAFPQPK2SNEEV2PIODUD5C2O6OOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:45680: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:45680: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T08:02:22.175Z        ERROR   piecestore      download failed {"Piece ID": "BYKG7XF46MDLLMA4JGFNFSXJR2N24TV566WFDGPPICD7B65BDU6Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:52394: write: broken pipe", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:52394: write: broken pipe\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T08:19:13.958Z        ERROR   piecestore      download failed {"Piece ID": "MCITR2ZGE6TVHYWAYRNGUCYVCFEGHD77RQHQOQDMTN3EISNXAW4Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->95.216.20.146:51996: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->95.216.20.146:51996: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T08:52:26.275Z        ERROR   piecestore      download failed {"Piece ID": "EVK4A7Z2RPRGCY6UDJUJAHLBBCUUKWARDJNTS5BQFDBKB6O35N3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->145.40.73.79:53496: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->145.40.73.79:53496: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T09:12:34.761Z        ERROR   piecestore      download failed {"Piece ID": "I52OA4EKWTMQF22FQEG4SYXZ6NNPSEFFWGR37P5N7LDA5BEMVJ4A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->145.40.73.87:43742: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->145.40.73.87:43742: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T09:17:21.770Z        ERROR   piecestore      upload failed   {"Piece ID": "EI42WJ7AOHYDYRLEUPQIFPQKPGJLKVGC6KUVDQ5TTDQTY4JGTQIQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T10:52:19.057Z        ERROR   piecestore      upload failed   {"Piece ID": "IFUOPLEZPUJAH3X3IC3XP63PBMISO7RSPHCBUOTL3SRHWKXCOJ6A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 311296}
2021-11-10T11:50:14.062Z        ERROR   piecestore      upload failed   {"Piece ID": "VNJUICD74ZGUPHP7SDBXQ5KJFPBZGKDC5DU7C6AO72EY3BWPWEQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T15:03:26.563Z        ERROR   piecestore      upload failed   {"Piece ID": "VYYM42ELAQCEJNNS6W7X7WLS3X6RES5HRI7E6UDN6GT7HBSG6XHQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T15:06:30.814Z        ERROR   piecestore      download failed {"Piece ID": "EVK4A7Z2RPRGCY6UDJUJAHLBBCUUKWARDJNTS5BQFDBKB6O35N3A", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func6:656\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download:674\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func2:228\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52"}
2021-11-10T15:07:48.092Z        ERROR   piecestore      upload failed   {"Piece ID": "62FD767HAA2GKSRGSCPB622HLEUGDN55ZGBO2VR72XNWGYJRMGOQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T16:06:30.624Z        ERROR   piecestore      download failed {"Piece ID": "AKQVDT677WSFSAAUD5FLOXQNQ3D67VNPRASSCAMDV3ZQO5JWLSCA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET", "error": "write tcp 172.17.0.2:28967->144.76.136.153:49802: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->144.76.136.153:49802: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T20:08:48.818Z        ERROR   piecestore      upload failed   {"Piece ID": "VLD6FHK6VX6LAUTN35PIGMWFSPJUWVYFWV2UDGZZOJ6LTBXPYEHQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 163840}
2021-11-10T20:17:30.490Z        ERROR   piecestore      download failed {"Piece ID": "AOMVT552PD5N53N4LZLXH2DZFQQGT27EC3VBYADPRRDZW2VUZ2LQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:60524: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:60524: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T20:47:06.852Z        ERROR   piecestore      upload failed   {"Piece ID": "URDCJSGY3JAT75L537OFMF2J6LHASCFFEFEX3WRB2A6KSXOALZYQ", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "PUT", "error": "context deadline exceeded", "errorVerbose": "context deadline exceeded\n\tstorj.io/common/rpc/rpcstatus.Wrap:73\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:344\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 163840}
2021-11-10T20:55:57.051Z        ERROR   piecestore      upload failed   {"Piece ID": "2QYWDNIDNRECDCXWVNPKILXCZEYUO2EU6V6QPZZQLVZOWZ3APF6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T21:46:17.417Z        ERROR   piecestore      upload failed   {"Piece ID": "GZCCNODZYXC4F2NTR7VNTCXY4ZTXHANUQM4B3YOCEXG6X437ZAIA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T22:37:17.647Z        ERROR   piecestore      upload failed   {"Piece ID": "U3KMMRZWK4ROXSFA3COLQNP4ACILBRLZFMTRLVAKYFNGZA5UDYOQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 65536}
2021-11-10T23:03:54.046Z        ERROR   piecestore      upload failed   {"Piece ID": "2F3XHSDFUZIAIIU54BPLUDRXHRDDFTILQFRRSEE4EYGURZ7IXP3Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 65536}
2021-11-10T23:07:05.728Z        ERROR   piecestore      download failed {"Piece ID": "CSMTRWUDKD7NSPAAJUWUPAIQD72G6LGKQDVBCKOBONISHGKU4C2Q", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:58070: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:58070: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T23:10:47.951Z        ERROR   piecestore      upload failed   {"Piece ID": "H2KNT374IVEYFL4PJ6PUTM6W2IAGQIWMQBC4OREUGQBK3OCOBD3Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "error": "unexpected EOF", "errorVerbose": "unexpected EOF\n\tstorj.io/common/rpc/rpcstatus.Error:82\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Upload:342\n\tstorj.io/common/pb.DRPCPiecestoreDescription.Method.func1:220\n\tstorj.io/drpc/drpcmux.(*Mux).HandleRPC:33\n\tstorj.io/common/rpc/rpctracing.(*Handler).HandleRPC:58\n\tstorj.io/drpc/drpcserver.(*Server).handleRPC:104\n\tstorj.io/drpc/drpcserver.(*Server).ServeOne:60\n\tstorj.io/drpc/drpcserver.(*Server).Serve.func2:97\n\tstorj.io/drpc/drpcctx.(*Tracker).track:52", "Size": 0}
2021-11-10T23:14:46.594Z        ERROR   piecestore      download failed {"Piece ID": "L66KOFVHKFXLS4UZGAPPB3KBQ4AVXJXP5XCKLSWA6G4GRUBEUAXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:55322: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:55322: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-11-10T23:17:26.333Z        ERROR   piecestore      download failed {"Piece ID": "L66KOFVHKFXLS4UZGAPPB3KBQ4AVXJXP5XCKLSWA6G4GRUBEUAXQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "error": "write tcp 172.17.0.2:28967->184.104.224.99:59230: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->184.104.224.99:59230: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}

Alexey · November 11, 2021, 8:20pm

Only GET_AUDIT + failed, GET_REPAIR + failed, blobs and database could lower the suspension score, so search only for them.

SGC · November 12, 2021, 9:44am

grep -E "GET_AUDIT|GET_REPAIR" 2021-11-10-sn0001* | grep failed

returns nothing, tried using started instead of failed and it works fine.
blobs gives nothing useful

grep 'blobs' 2021-11-10-sn0001*
2021-11-10T00:48:52.120Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.122Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 122880}
2021-11-10T00:48:52.124Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 4352}
2021-11-10T00:48:52.126Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 109056}
2021-11-10T00:48:52.128Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 109056}
2021-11-10T00:48:52.130Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.133Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.135Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.138Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.140Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.153Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.155Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.157Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.159Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.160Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 109056}
2021-11-10T00:48:52.162Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 109056}
2021-11-10T00:48:52.164Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 109056}
2021-11-10T00:48:52.167Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.169Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.171Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T00:48:52.173Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 109056}
2021-11-10T00:48:52.175Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T01:48:52.116Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T01:48:52.117Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 69376}
2021-11-10T01:48:52.120Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1792}
2021-11-10T01:48:52.122Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T01:48:52.124Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T01:48:52.126Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T01:48:52.128Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 512}
2021-11-10T01:48:52.130Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 72960}
2021-11-10T01:48:52.133Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 956160}
2021-11-10T01:48:52.135Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2048}
2021-11-10T01:48:52.137Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 512}
2021-11-10T01:48:52.139Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 768}
2021-11-10T01:48:52.140Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1536}
2021-11-10T02:48:52.125Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 651008}
2021-11-10T02:48:52.128Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1024}
2021-11-10T02:48:53.193Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 981760}
2021-11-10T02:48:53.196Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1024}
2021-11-10T02:48:53.198Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1024}
2021-11-10T02:48:53.200Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1024}
2021-11-10T02:48:53.201Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1024}
2021-11-10T02:48:53.203Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1024}
2021-11-10T02:48:53.206Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1671680}
2021-11-10T02:48:53.209Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1280}
2021-11-10T02:48:53.210Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1280}
2021-11-10T02:48:53.212Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1280}
2021-11-10T02:48:53.214Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1536}
2021-11-10T02:48:53.215Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1536}
2021-11-10T03:48:52.117Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T03:48:52.119Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T03:48:52.121Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 768}
2021-11-10T03:48:52.123Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 69376}
2021-11-10T03:48:52.125Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 3072}
2021-11-10T03:48:52.128Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1792}
2021-11-10T03:48:52.130Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 182784}
2021-11-10T03:48:52.133Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1792}
2021-11-10T03:48:52.135Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1280}
2021-11-10T03:48:52.137Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T03:48:52.138Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1792}
2021-11-10T03:48:52.139Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1792}
2021-11-10T03:48:52.141Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1792}
2021-11-10T03:48:52.143Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 177920}
2021-11-10T04:48:52.715Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 512}
2021-11-10T04:48:52.721Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T04:48:52.723Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 72960}
2021-11-10T04:48:52.735Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T04:48:52.739Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2048}
2021-11-10T04:48:52.742Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T04:48:52.745Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 7168}
2021-11-10T04:48:52.748Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 929536}
2021-11-10T04:48:52.752Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T04:48:52.756Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T04:48:52.759Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T05:48:52.122Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T05:48:52.125Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 440064}
2021-11-10T05:48:52.127Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 13824}
2021-11-10T05:48:52.129Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T05:48:52.134Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T05:48:52.136Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T05:48:52.139Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 32000}
2021-11-10T05:48:52.141Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 69376}
2021-11-10T05:48:52.144Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T05:48:52.147Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1536}
2021-11-10T05:48:52.150Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T06:48:52.573Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 558592}
2021-11-10T06:48:52.576Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1536}
2021-11-10T06:48:52.578Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 516864}
2021-11-10T06:48:52.583Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 9216}
2021-11-10T06:48:52.585Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T06:48:52.587Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T07:48:52.161Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T07:48:52.164Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 257536}
2021-11-10T07:48:52.170Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 16128}
2021-11-10T07:48:52.174Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T07:48:52.178Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 878592}
2021-11-10T07:48:52.182Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T07:48:52.185Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 665088}
2021-11-10T08:48:52.565Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 9216}
2021-11-10T08:48:52.578Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 512}
2021-11-10T08:48:52.585Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T08:48:52.600Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T08:48:52.610Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T08:48:52.623Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T08:48:52.655Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T08:48:52.707Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.122Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 539904}
2021-11-10T09:48:52.125Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1280}
2021-11-10T09:48:52.127Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 136704}
2021-11-10T09:48:52.129Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.132Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.133Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 6912}
2021-11-10T09:48:52.136Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 1280}
2021-11-10T09:48:52.138Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 69376}
2021-11-10T09:48:52.140Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 613376}
2021-11-10T09:48:52.143Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.145Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.148Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.153Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.155Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.157Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T09:48:52.158Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 512}
2021-11-10T10:48:52.545Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.549Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.553Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.556Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.560Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.563Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2560}
2021-11-10T10:48:52.569Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.574Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 81664}
2021-11-10T10:48:52.577Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.581Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.585Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.589Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.591Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 65792}
2021-11-10T10:48:52.597Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.600Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.603Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.606Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 205568}
2021-11-10T10:48:52.608Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.612Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.615Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.617Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T10:48:52.619Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T11:48:52.367Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T11:48:52.370Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T11:48:52.374Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 21760}
2021-11-10T11:48:52.377Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T11:48:52.379Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T11:48:52.382Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T11:48:52.386Z        DEBUG   blobscache      deleted piece   {"Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "disk space freed in bytes": 2319360}
2021-11-10T19:12:23.333Z        DEBUG   services        started {"items": ["version", "trust", "contact:chore", "PieceDeleter", "pieces:trash", "piecestore:cache", "piecestore:monitor", "retain", "orders", "nodestats:cache", "console:endpoint", "gracefulexit:blobscleaner", "gracefulexit:chore", "collector", "bandwidth"]}

and searching for database or Database only returns this… i have in the past seen bandwidth.db issues… tried to search for bandwidth also, but no luck.

grep ‘Database’ 2021-11-10-sn0001*
2021-11-10T19:12:19.826Z INFO db.migration Database Version {“version”: 53}
2021-11-10T19:12:19.831Z DEBUG db Database version is up to date {“version”: 53}

and bandwidth is the same returns nothing useful

will have removed the end of the blobs search, since it was also unhelpful and most likely takes to much space for me to post this.

mr.Viking · November 12, 2021, 1:24pm

The same problem after update docker image to v1.42.4.
Some nodes periodically drop and restore suspension.

SGC · November 12, 2021, 1:29pm

most likely this issue with EU1 from today, ofc it’s possible that you have the same issue as me but this is not a new thing, been happening for a long time and there seems to be no cause i can find anyways.

if you are seeing EU1 sat drop in suspension score please post it on this thread.