Freezed onlinescore

[Solu]

After 2 days planned maintenance downtime my node shows up again on 2022-11-08T23:00:00Z with ~89% onlinescore orange colored on dashboard.

A few days later my node updated to v1.67 (watchtower) the onlinescore hasnt changed so far
but the onlinescore now shows up red colored related to Release Preperation v1.67 i guess.

Today my node was suspended on the first satellite
"Your Storage Node on the europe-west-1 Satellite was suspended because it produced errors too often during audits.

You were suspended on 2022-11-25 at 16:15 UTC."

Weird because i cannot find any issues, the onlinescore seems to be frozen since my node went online again on 2022-11-09… Node runs flawless the last 3 Years.

Auditscore:

for sat in `wget -qO - localhost:14002/api/sno | jq .satellites[].id -r`; do wget -qO - localhost:14002/api/sno/satellite/$sat | jq .id,.audits; done
"12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"
{
  "auditScore": 1,
  "suspensionScore": 0.9998400391155746,
  "onlineScore": 0.8907883188140794,
  "satelliteName": "us2.storj.io:7777"
}
"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.8938881790326841,
  "satelliteName": "saltlake.tardigrade.io:7777"
}
"121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.8956448345397156,
  "satelliteName": "ap1.storj.io:7777"
}
"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.8964160471553075,
  "satelliteName": "us1.storj.io:7777"
}
"12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.8959536633828439,
  "satelliteName": "eu1.storj.io:7777"
}
"12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB"
{
  "auditScore": 1,
  "suspensionScore": 1,
  "onlineScore": 0.8927838837750933,
  "satelliteName": "europe-north-1.tardigrade.io:7777"
}

Logs:

docker logs storagenode 2>&1 | grep -E "GET_AUDIT|GET_REPAIR" | grep failed
2022-11-16T17:41:58.354Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "FDID6DIMRB7IGMCS2Y2NSL7VDAEZ6ORFZY4S6CM3VPQVJVQOFKXA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.130.154:45912: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.130.154:45912: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:326\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:628\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-11-16T17:43:30.276Z        ERROR   piecestore      download failed {"Process": "storagenode", "Piece ID": "ZHR2ZNDL5RZJEA6UE6AZQZNSGUCSDZZD4VFLTMNP24S4YT5XGOUA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->5.161.130.152:45544: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:28967->5.161.130.152:45544: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:326\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:404\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:628\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}

docker logs storagenode 2>&1| grep FATAL
2022-11-14T20:11:18.368Z        INFO    piecestore      upload started  {"Process": "storagenode", "Piece ID": "KFVMO3GSFYTFVBE6BJIEUHVBMFATALH6BAMEVYSWXSHSRDQZVCNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Available Space": 2491652895488}
2022-11-14T20:11:21.895Z        INFO    piecestore      uploaded        {"Process": "storagenode", "Piece ID": "KFVMO3GSFYTFVBE6BJIEUHVBMFATALH6BAMEVYSWXSHSRDQZVCNQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "PUT", "Size": 2174464}
2022-11-21T18:34:36.270Z        INFO    piecedeleter    delete piece sent to trash      {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Piece ID": "UCEOGMD4MXN7HCKBR3POST2QJFATALJFKQ6JH3EACGVBN24PCNYQ"}

Auditlogs from europe-west limited on 2022-11-25T23:00:00Z

docker logs --since 2022-11-26T00:00:00Z storagenode 2>&1| grep GET_AUDIT | grep 12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
2022-11-26T00:00:48.665Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "4DGIH26FM3HB3RVW4RZW26HJMAMCIKY6BT22EWDWXPPNS6GYRXBA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T00:00:48.712Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "4DGIH26FM3HB3RVW4RZW26HJMAMCIKY6BT22EWDWXPPNS6GYRXBA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T00:01:19.951Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "MN2DZVDR4Q44YCYFTF7RYFZYMVEX25DTDWUKLRMJKZSXSAVT4ECQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:01:05.069Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "MN2DZVDR4Q44YCYFTF7RYFZYMVEX25DTDWUKLRMJKZSXSAVT4ECQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:11:59.770Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "TJEGDA7S5RXQSWD2LT5ATX6MHYWB6XDYPVTGCMCFKKT2SNWXW5CA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:11:59.838Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "TJEGDA7S5RXQSWD2LT5ATX6MHYWB6XDYPVTGCMCFKKT2SNWXW5CA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:26:40.627Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "L37CAGPUTWTQHO4WSXDWAUEKLKSMZ63FP3URNTGUDLOIN4WW336A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:26:40.678Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "L37CAGPUTWTQHO4WSXDWAUEKLKSMZ63FP3URNTGUDLOIN4WW336A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:27:59.368Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "JAMSZDILLAAHNSYNTLJXJZ3BVWPUAABBA7TFXU6QQV4QMQE2JBXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:27:59.422Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "JAMSZDILLAAHNSYNTLJXJZ3BVWPUAABBA7TFXU6QQV4QMQE2JBXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:33:54.255Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "G66OA4ZBLAG244SYWE36AUM5V6HTODEJRODTKOCFMMNY3YPJ245A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:33:54.309Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "G66OA4ZBLAG244SYWE36AUM5V6HTODEJRODTKOCFMMNY3YPJ245A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:36:44.762Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "5DYAUI3PL7AFE6WP63M4CPAFAZT7COU4O675IF6LTRM75JL6BLMQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T08:36:44.816Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "5DYAUI3PL7AFE6WP63M4CPAFAZT7COU4O675IF6LTRM75JL6BLMQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T09:19:59.520Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "6Y5DNGECE67SAMNVNSKP7DNJNRI7TEYISSWDS44WGXSOG4IV7WXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T09:19:59.572Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "6Y5DNGECE67SAMNVNSKP7DNJNRI7TEYISSWDS44WGXSOG4IV7WXQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T09:22:24.154Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "DMLCLVO4QWLKHHOAAYJ6TIO6RDMA6FBXMCIBRYG2BJTP2TKRZSIQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T09:22:24.216Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "DMLCLVO4QWLKHHOAAYJ6TIO6RDMA6FBXMCIBRYG2BJTP2TKRZSIQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T09:56:45.485Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "CS3ZZRIXJ2RZBVQ6N2HM6NERI5RWSPY66AC7VCOJFUOCZQIUNCCQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T09:56:45.553Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "CS3ZZRIXJ2RZBVQ6N2HM6NERI5RWSPY66AC7VCOJFUOCZQIUNCCQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:04:03.697Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "AMHLKOZGVN3DEDX7OQWF2JBRIPSPFVKF3VETJ7PUHWMQPNVE5PVQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:04:03.769Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "AMHLKOZGVN3DEDX7OQWF2JBRIPSPFVKF3VETJ7PUHWMQPNVE5PVQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:07:36.711Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "E3ADJZYP3WJSAUXOG2CE4RX3T2HMUWH4KEBWOUQ4TPJX5GSTG2PQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:07:36.757Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "E3ADJZYP3WJSAUXOG2CE4RX3T2HMUWH4KEBWOUQ4TPJX5GSTG2PQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:12:00.340Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "SZ55BX6F4TZX3AKQBIROUELT4YUBJJWUVDITYHWBC6MOKUDDLNGQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:12:00.392Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "SZ55BX6F4TZX3AKQBIROUELT4YUBJJWUVDITYHWBC6MOKUDDLNGQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:15:32.911Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "PTC723PI3DQMCZLEPJLZO2IVYM2BDZFOFWVI62RI4VVREYRNTS7Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:15:32.966Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "PTC723PI3DQMCZLEPJLZO2IVYM2BDZFOFWVI62RI4VVREYRNTS7Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:32:26.313Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "WDIJZDGOCJGXP3CR55F26J4DG4GLH6XWTIK65YXAPCE4Z6AHDKRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:32:26.365Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "WDIJZDGOCJGXP3CR55F26J4DG4GLH6XWTIK65YXAPCE4Z6AHDKRQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:35:03.608Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "VSDLWR3FKUJVOHCJ2X5QAQXF4GFNRDK3UQL2JYUY4WU3AKFONW2Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:35:03.648Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "VSDLWR3FKUJVOHCJ2X5QAQXF4GFNRDK3UQL2JYUY4WU3AKFONW2Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:37:56.446Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "NA4SQG7HZB5KKMFJXMWU5ETZO5VDIKCGPBVZ4CNHQ4JB6IJDGWEQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:37:56.511Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "NA4SQG7HZB5KKMFJXMWU5ETZO5VDIKCGPBVZ4CNHQ4JB6IJDGWEQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:42:50.543Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "ITVQ4NZS47YBWRZLAL2QHQUJCGB4SZST5XQFCTBPYVZJLTHH3IHA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T10:42:50.595Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "ITVQ4NZS47YBWRZLAL2QHQUJCGB4SZST5XQFCTBPYVZJLTHH3IHA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:11:15.569Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "FX2ZRJNWBPFIBR7WSWNFUJX7Q6UBYNPEGPI5XZGHMUVQYE7KKYTA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:11:15.634Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "FX2ZRJNWBPFIBR7WSWNFUJX7Q6UBYNPEGPI5XZGHMUVQYE7KKYTA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:21:48.676Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "54A3IJGO6ACTVEYHR457VEDWOJZR2H3C6DFBBRDOVSPGBVSDAEJQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:21:48.780Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "54A3IJGO6ACTVEYHR457VEDWOJZR2H3C6DFBBRDOVSPGBVSDAEJQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:26:53.825Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "7B2UHQY3FDV25N66JG7RDLIVSYPHI2QTTLCFFS3PPBUHTYBEGWDA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:26:53.885Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "7B2UHQY3FDV25N66JG7RDLIVSYPHI2QTTLCFFS3PPBUHTYBEGWDA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:31:39.321Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "NVUZKXT3WRDXLSEB4ABZ5GOKA47GZOEK6KGNB2CCQVJBMYDZXAQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:31:39.373Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "NVUZKXT3WRDXLSEB4ABZ5GOKA47GZOEK6KGNB2CCQVJBMYDZXAQQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:38:29.031Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "DZDN6DGR42BKM352P22AA7ZH2KI7B5OYN6BIL6DWU6FOF6ETNMGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:38:29.088Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "DZDN6DGR42BKM352P22AA7ZH2KI7B5OYN6BIL6DWU6FOF6ETNMGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:43:53.746Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "TR4AT2WFKPDCRSNH7HA5C5LKAH65MKW3A3OGFESGEIDOAX4LTMTA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T11:43:53.800Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "TR4AT2WFKPDCRSNH7HA5C5LKAH65MKW3A3OGFESGEIDOAX4LTMTA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:06:54.113Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "V2AJH2HVEIKS7R7NECLR325OBCACEGDFEK3QSPHERPQJ4NCTIF6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:06:54.170Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "V2AJH2HVEIKS7R7NECLR325OBCACEGDFEK3QSPHERPQJ4NCTIF6Q", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:09:11.226Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "VKWYH727VNRSYWVPDBIFOSS6IA3UXNRGEQIDPLGYTA2HKUZEH6RA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:09:11.282Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "VKWYH727VNRSYWVPDBIFOSS6IA3UXNRGEQIDPLGYTA2HKUZEH6RA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:21:49.027Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "CE3HCKXJYRFCVYZ6NRKSABTISVBUVEOF2WENXDY6URCWHP5VUBGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:21:49.084Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "CE3HCKXJYRFCVYZ6NRKSABTISVBUVEOF2WENXDY6URCWHP5VUBGA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:22:41.609Z        INFO    piecestore      download started        {"Process": "storagenode", "Piece ID": "TQWZXRFYBKXYPEM7IQCKZEEZVLO7GZDCTA3NFP5QX655UTPJLGAQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}
2022-11-26T12:22:41.678Z        INFO    piecestore      downloaded      {"Process": "storagenode", "Piece ID": "TQWZXRFYBKXYPEM7IQCKZEEZVLO7GZDCTA3NFP5QX655UTPJLGAQ", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_AUDIT"}

Now, what am i missing here?
If the node was able to get audits, then the node was surely online.
Any clues why the onlinescore is still frozen?

Thank you a lot.

[Roberto]

What version do you have? 1.67.1 is full of bugs, it has been replaced with 1.67.3

[Solu]

Node version is v.1.67.1

I read V1.67.1 Suspension percentage with all its bug reports but no one seems similar.

Would be hard to see my node disqualified one some satallites caused by bugs.
It seems v.1.67.3 is not yet available for linux?

docker pull storjlabs/storagenode:latest
latest: Pulling from storjlabs/storagenode
Digest: sha256:9cf6953623af2609e34572f8c084e9b472b0864f3cf204589aafea1acbb29fea
Status: Image is up to date for storjlabs/storagenode:latest
docker.io/storjlabs/storagenode:latest

[Roberto]

The storagenode update happens automatically inside the container, you have to wait, I’m still on 1.66.1. It’s some tricky docker stuff

Autoupdate to pre-release versions

Is it normal watchtower to autoupdate my docker nodes to pre-release versions? I know that in the run command of storagenode and watchtower there is a “latest” tag. I see on github that ver. 1.66.1 is latest. Than why part of my nodes are at ver. 1.67.1 and 1.67.3, that are tagged “pre-release”? Is there a way to opt-out from pre-releases?

[Alexey]

@Solu

Your Storage Node on the europe-west-1 Satellite was suspended because it produced errors too often during audits.

This is not related to downtime, it’s related to failed audits with unknown error, so this message related to the suspension score, not the online score.
Your node will be suspended if the suspension score drop below 60% (0.6), however, if your node started to pass audits, the suspension score will recover with each successful GET_AUDIT or GET_REPAIR.
And as you can see it’s recovering (0.9998400391155746):

@Solu

"12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo"
{
  "auditScore": 1,
  "suspensionScore": 0.9998400391155746,
  "onlineScore": 0.8907883188140794,
  "satelliteName": "us2.storj.io:7777"
}

The message is come when you/updater restarted the node, it is checked-in on the satellite and received the message related to the past state because of this bug: [storagenode] The email about suspension is coming only after check-in to satellite · Issue #4235 · storj/storj · GitHub