Got Disqualified from saltlake

[allenyllee]

Hi, I’ve suddenly received an email with title “Your Node Is Disqualified on the saltlake Satellite”. Actually, just in few days ago, my office(where i hosting storj node)'s firewall policy had changed, so one can’t access node from internet, but I didn’t aware of that, untill I got this email. According to this thread:

Disqualified on one satellite

Hello @krilson, Welcome to the forum! Because of failed audits. Please, check your logs and search for GET_AUDIT and failed to see a reason. Also, the audit will be treated as failed if you have GET_AUDIT and download started, but not downloaded at the end. And of course, if audited piece is exist but corrupted, the audit will be treated as failed too. If audit score fall below 0.6 (60% on the dashboard), the node will be disqualified.

I’ve checked my logs, but I didn’t find any “GET_AUDIT” and “failed” keywords. Doesn’t that mean I should not got disqualified?

my node id is: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg

Could anyone help?
Thanks.

[Toyoo]

@allenyllee

I’ve checked my logs, but I didn’t find any “GET_AUDIT” and “failed” keywords. Doesn’t that mean I should not got disqualified?

Well, your node wouldn’t know about the satellites trying to audit it, because the satellites couldn’t contact your node. So your node had nothing to log.

[SGC]

if the node is offline it shouldn’t get DQ… atleast not before having like a month of downtime…

saltlake is a most active satellite for new nodes, so maybe it hit the month limit first…
are you sure the node has even been online recently.

[Alexey]

@allenyllee

I’ve checked my logs, but I didn’t find any “GET_AUDIT” and “failed” keywords.

Do you have logs before the date of disqualification?
Please, search also for GET_REPAIR and failed. They can be accounted against the audit score.

[allenyllee]

But isn’t there any way to notify node operator when node is offline from satellite before get DQ? For example, sending an email to notify node operator before actually get DQ?

[allenyllee]

Yes, but only trace back to date 8/5 because watchtower pulled the latest image on that day.
I try to search “GET_REPAIR” and “failed” but still nothing…

[allenyllee]

I think it is a bug. The node should know there isn’t any satellite trying to audit for a period (may be 5 mins?) so that node can leave logs about didn’t contact to satellites.

[Alexey]

If logs are deleted - in fresh logs you would not have error messages anymore, the node is already dead on that satellite.

@allenyllee

node can leave logs about didn’t contact to satellites.

The node is doing exactly that. You can easily check - disable the port forwarding rule and check your logs.

The offline doesn’t affect an audit score, it affects the online score.
So, to fail audits and being disqualified your node must be online.
It can fail audits in three main ways:

1. missed piece - file does not exist. Affects audit score immediately. Will be written into logs
2. timeouts - the node answers on audit request, but cannot provide a piece for the next 5 minutes. And did the same two more times for the same piece. Such audit will be considered as failed. You cannot log it into logs - the node write a message only when the download of audit is started. However, in such cases the node is usually unable to write to log too (because it’s partially hangs or too slow to response due to hardware or software problems with the host). But if it can write to logs - you will see that audit is started, but never finished.
3. corrupted piece - the node returned a piece, but it’s corrupted. Affects audit score immediately. The audit failure doesn’t logged into logs, there would be usual “download started” and “downloaded”.

The satellite never reports which piece is failed audit. The file does not exist just an error from the OS and if you see such error - you may be sure - this audit is definitely failed.

[allenyllee]

Here is my log file started from date 8/5 and filtered by satellite ID “1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE” and “failed”.

2021-08-05T11:00:44.188Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T12:00:45.938Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T12:15:13.846Z	ERROR	piecestore	download failed	{"Piece ID": "RE2ZKRBS54SW7MMIL4BPAQI6IFJWPZWA7XKDZD43XRUEXHHSE6QA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.47:41766: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.47:41766: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T13:00:45.611Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T13:43:52.256Z	ERROR	piecestore	download failed	{"Piece ID": "JZODDF5KOEFQLZZ2CBW5ULPMT7MVGJ4Q7QPPWV2LE7DPEIXNQB7A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.54:41146: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.54:41146: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T14:00:44.647Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T15:00:44.466Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T16:00:46.396Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T16:10:16.652Z	ERROR	piecestore	download failed	{"Piece ID": "MAUVZZV662UI3BYEIVCLHZMRZM7GA3UC6SPX5ZR4DGHMZ2F7HJGA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.48:48450: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.48:48450: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T16:36:12.836Z	ERROR	piecestore	download failed	{"Piece ID": "TZ55SPVRDMBQ5TURXBCOBB4TWKYDLEKCAM7PJS3IONVXXSBMDP4Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.51:42440: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.51:42440: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T17:00:44.788Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T17:13:22.869Z	ERROR	piecestore	download failed	{"Piece ID": "A5BIE5HUZBASY6ZPBLJTRHW2TKBXNW6CIQYKFQNCJTED4XJ2JJVA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.52:38264: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.52:38264: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T18:00:45.588Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T19:00:45.663Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T19:05:08.208Z	ERROR	piecestore	download failed	{"Piece ID": "RRLAWRD5DQ3PGA7SIBRAD4SIDVNTGABZNXPQQEZV43GEAYMVZKCA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.52:38960: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.52:38960: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T19:38:37.466Z	ERROR	piecestore	download failed	{"Piece ID": "OF7GO4G5N4AHZ4HGVHQDQ4JY56PXNVBWHY62H4TBRN6QHSBY7POA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.53:34166: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.53:34166: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T20:00:44.400Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T21:00:45.431Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T22:00:44.441Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-05T22:18:10.993Z	ERROR	piecestore	download failed	{"Piece ID": "YPMYIAAAAD5GQHBMSAI6CFGKOWXSZD4DHBW7SPUOZEYANH4UKY6A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.47:39750: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.47:39750: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-05T23:00:45.393Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T00:00:45.014Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T00:01:25.689Z	ERROR	piecestore	download failed	{"Piece ID": "I54IDNKHJJTWAYHOXQZM5VTDNVM5WCOIGEGNEYZL6EU7IINSJVWQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.53:39260: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.53:39260: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T01:00:45.295Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T01:36:39.873Z	ERROR	piecestore	download failed	{"Piece ID": "Y4CCQOXG3WZYBWP4P4GFT3C7FCXIHYLHLJ2UPUAHJR3MI7QWZJ7Q", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->95.217.161.92:37644: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->95.217.161.92:37644: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T02:00:47.374Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T03:00:45.426Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T03:04:48.896Z	ERROR	piecestore	download failed	{"Piece ID": "SGLQKO742CXL67ASWLNNDJMZRXXNPREHNIB43GYYNX3TSCG3K4DA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->95.217.161.92:40544: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->95.217.161.92:40544: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T03:07:18.923Z	ERROR	piecestore	download failed	{"Piece ID": "TCHEFOB2YMJCFO2CU67NSDKBLRXIO3VO2QNMGD2IUD676DASWTQA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.48:38122: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.48:38122: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
......
2021-08-06T13:00:44.376Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T14:00:45.482Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T14:11:51.442Z	ERROR	piecestore	download failed	{"Piece ID": "KCGYO6HUO5S7EPXVFE4DNVFNQBGY7IWP3ISZ2GIFWGJCYVVBELAQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.52:32884: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.52:32884: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T15:00:44.529Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T15:56:59.199Z	ERROR	piecestore	download failed	{"Piece ID": "PSZH2OIGPOYR6UB2RHYD5FMK3CSTNUUDNTUBI5J7ZX4WN3RM644A", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->95.217.161.92:50780: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->95.217.161.92:50780: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T16:00:45.052Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T16:28:59.047Z	ERROR	piecestore	download failed	{"Piece ID": "A5BIE5HUZBASY6ZPBLJTRHW2TKBXNW6CIQYKFQNCJTED4XJ2JJVA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.50:35792: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.50:35792: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T16:31:59.713Z	ERROR	piecestore	download failed	{"Piece ID": "HLFX3H6JT3UU5LFMLRXZPUPZ4N3PD2ATKKLT2JQ4AX5C6ZAK5VBA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.51:50564: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.51:50564: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T17:00:45.849Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T17:02:23.723Z	ERROR	piecestore	download failed	{"Piece ID": "O7ZCSTSE5HPCMD7XFBWAVH3TZZHEGGHXR4QTVBHOBAAVHYYWV6VQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.51:58214: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.51:58214: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T17:09:45.912Z	ERROR	piecestore	download failed	{"Piece ID": "VQQBBR3YED3G245QDYXVKRZTUIQWMLQMRNQ25QEVEFNLUK6KDYKQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.50:37428: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.50:37428: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T18:00:44.560Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T19:00:44.524Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T20:00:44.797Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T21:00:17.985Z	ERROR	piecestore	download failed	{"Piece ID": "J74N2XFBV34S6HBFLMTN2XG5DIE2LN6NKI5ORK5S3H64OHAMP4MA", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET", "error": "write tcp 172.17.0.2:28967->135.181.29.47:34744: use of closed network connection", "errorVerbose": "write tcp 172.17.0.2:28967->135.181.29.47:34744: use of closed network connection\n\tstorj.io/drpc/drpcstream.(*Stream).pollWrite:211\n\tstorj.io/drpc/drpcwire.SplitN:29\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:261\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:315\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:302\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:608\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2021-08-06T21:00:44.484Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T22:00:44.936Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-06T23:00:46.040Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T00:00:44.201Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T01:00:44.493Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T02:00:44.655Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T03:00:44.432Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T04:00:45.307Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T05:00:45.043Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T06:00:44.567Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T07:00:44.867Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T07:04:13.452Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T07:05:01.548Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}
2021-08-07T08:05:01.917Z	WARN	contact:service	Your node is still considered to be online but encountered an error.	{"Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Error": "contact: failed to dial storage node (ID: 15ZiPFXywuYhVi53kn5Ki2oujqBfdybVt54i7uTTPgUQJpdaCg) at address storj.allenyl.ml:28967 using QUIC: rpc: quic: timeout: no recent network activity"}

see full log: 2021-08-05T11:00:44.188Z WARN contact:service Your node is still considered to b - Pastebin.com

As you can see, it says “contact:service Your node is still considered to be online but encountered an error.”

There are few download failed error, but I don’t think that it is the main reason of DQ.

[Alexey]

These warns doesn’t lead to disqualification, you need to forward UDP port and that’s all

Experimenting with UDP based protocols

Hi everyone, we are excited to inform you that we have been working on performance improvement in our network. One area that we have been exploring is UDP based transport protocols. For example, We have done some internal testing and have seen some good result from QUIC protocol. Now we would like to gather more realistic performance metrics with UDP based protocols. In order to do that, we want to ask you, our storage node operators, to update your port-forwarding rules to also include UDP tra…

@allenyllee

ERROR piecestore download failed ... use of closed network connection

The client drop connection - your node in the long tail cancelation backlog.
Nothing to worry about.

Please, temporary disable the port forwarding rule - you will see, how the real offline error is looks like.
Then enable back.

It’s not related to disqualification though.

[allenyllee]

I do, but I still got disqualified. see:

Actually, I saw yellow mark in other satellite yesterday, but now it’s almost recovered. But the saltlake didn’t seem to be recovered, I think it is a mistake, it should be recovered!

The DQ state can not be recovered?

[allenyllee]

After check, my office MIS says, the firewall policy had changed just begin from 8/6, and my logs since 8/5 doesn’t have any strange, that means it can’t be disqualified before 8/6(the offline date), but I got disqualified email at 8/7, I think It is too soon to be disqualified!

[allenyllee]

Hi Alexey, here’s my full logs(without filter out anything since 8/5 the latest storj image had been pulled and container restarted, only masked my email and wallet address):

https://gist.github.com/allenyllee/954f965c56c425f00705abe488e5e085

I can’t see any strange though…

[SGC]

yeah not noticing your node being offline for a month straight is pretty difficult… and ofc it would also have to happen at the wrong time…(when you aren’t looking for 30 days straight)

[Alexey]

The disqualification is permanent and not reversible if audit score drop lower than 60%, it cannot recover automagically.
All suggestions was only to figure out why is it happened to do not repeat this situation again.

My example with disabling port forwarding was only to show how the offline message is looks like and that it’s not affecting audit score unlike failed audits.
However, I would recommend to do not delete data for that satellite until we figure out, why is it failed too much audits and was disqualified.

The reasons of failed audits I showed above.
Now we only need to figure out which one from them has been applied to your storagenode.

Do you have older logs than 05/08?
Please, pack them before upload. You can use transfer.sh for share.

From analyzing the provided log:

$ grep -E "GET_AUDIT" storj_logs.txt  | grep "2021-08-05" | grep "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | grep started -c
49

$ grep -E "GET_AUDIT" storj_logs.txt | grep "2021-08-06" | grep "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | grep started -c
78

$ grep -E "GET_AUDIT" storj_logs.txt | grep "2021-08-07" | grep "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | grep started -c
3

Your node has been disqualified on Saltlake 08/07/21 03:41
Let’s see the last two days dates

$ grep -E "GET_AUDIT" storj_logs.txt | grep -E "2021-08-06|2021-08-07" | grep "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | grep started | awk '{print $1}'
2021-08-06T01:39:10.227Z
2021-08-06T01:44:45.731Z
2021-08-06T02:08:39.037Z
2021-08-06T02:12:47.864Z
2021-08-06T03:39:47.115Z
2021-08-06T03:42:33.563Z
2021-08-06T04:04:11.047Z
2021-08-06T04:09:30.074Z
2021-08-06T04:15:44.396Z
2021-08-06T04:17:46.687Z
2021-08-06T05:02:35.713Z
2021-08-06T05:19:15.637Z
2021-08-06T05:22:19.045Z
2021-08-06T05:38:46.017Z
2021-08-06T05:39:34.873Z
2021-08-06T06:40:20.213Z
2021-08-06T07:09:40.523Z
2021-08-06T07:52:10.378Z
2021-08-06T07:54:46.413Z
2021-08-06T07:58:10.573Z
2021-08-06T08:21:15.573Z
2021-08-06T08:30:49.799Z
2021-08-06T08:40:17.101Z
2021-08-06T08:41:06.015Z
2021-08-06T08:43:07.652Z
2021-08-06T08:49:34.713Z
2021-08-06T08:55:37.703Z
2021-08-06T08:56:12.284Z
2021-08-06T09:17:08.155Z
2021-08-06T10:04:18.982Z
2021-08-06T10:16:14.477Z
2021-08-06T10:23:27.897Z
2021-08-06T10:27:39.048Z
2021-08-06T10:30:32.024Z
2021-08-06T10:57:18.001Z
2021-08-06T11:39:36.635Z
2021-08-06T11:56:59.403Z
2021-08-06T12:00:01.999Z
2021-08-06T12:21:02.103Z
2021-08-06T12:21:42.099Z
2021-08-06T12:27:59.834Z
2021-08-06T12:57:20.280Z
2021-08-06T13:04:43.144Z
2021-08-06T13:10:11.315Z
2021-08-06T13:16:01.890Z
2021-08-06T13:21:39.751Z
2021-08-06T13:50:46.553Z
2021-08-06T14:03:46.424Z
2021-08-06T14:06:28.374Z
2021-08-06T14:24:11.355Z
2021-08-06T14:43:28.290Z
2021-08-06T15:13:17.618Z
2021-08-06T15:28:52.756Z
2021-08-06T16:28:07.078Z
2021-08-06T16:29:32.721Z
2021-08-06T16:45:54.923Z
2021-08-06T17:05:18.818Z
2021-08-06T17:14:54.303Z
2021-08-06T18:09:23.548Z
2021-08-06T18:31:53.098Z
2021-08-06T18:45:38.463Z
2021-08-06T19:18:54.599Z
2021-08-06T19:33:16.774Z
2021-08-06T20:15:08.398Z
2021-08-06T20:16:50.400Z
2021-08-06T20:27:55.027Z
2021-08-06T20:35:21.704Z
2021-08-06T20:36:10.216Z
2021-08-06T20:39:04.123Z
2021-08-06T20:44:01.043Z
2021-08-06T21:15:32.523Z
2021-08-06T21:53:14.903Z
2021-08-06T22:11:29.981Z
2021-08-06T22:21:55.367Z
2021-08-06T22:34:43.681Z
2021-08-06T22:48:33.899Z
2021-08-06T22:57:18.455Z
2021-08-06T23:17:05.500Z
2021-08-07T00:12:31.320Z
2021-08-07T00:16:01.163Z
2021-08-07T00:40:16.540Z

From the excerpt we can see, that your node has been audited by Saltlake between 2 and 7 times per hour. But there is only one entry for 2021-08-06T23:17:05.500Z and three for 2021-08-07 00:***
Looks like the problem has been started after 2021-08-07T00:40:16.540Z, because there is no one audit until it got disqualified.

Can you check your system and firewall logs - what’s was the problem between 2021-08-07T00:40:16.540Z and 2021-08-07T03:42:00Z with your system?

[allenyllee]

Do you have older logs than 05/08?

No, since the watchtower auto update storj node image and restart the container, the logs older then 05/08 had been wiped.

Looks like the problem has been started after 2021-08-07T00:40:16.540Z, because there is no one audit until it got disqualified.

I got disqualified email at about UTC 04:00, just 4 hours later since it started no audit.

Can you check your system and firewall logs - what’s was the problem between 2021-08-07T00:40:16.540Z and 2021-08-07T03:42:00Z with your system?

I checked my syslog and other log files in /var/log, doesn’t find any strange.

The only thing I can confirm is that my office network had been changed since 06/08/2021, my computer seems very normal, and hard disk still spinning without error.

So, how can I do next? Can I directly analysis the data come from that satellite? Or are there any chance the satellite mistakenly disqualified my node?

[Alexey]

@allenyllee

I got disqualified email at about UTC 04:00, just 4 hours later since it started no audit.

@Alexey

Your node has been disqualified on Saltlake 08/07/21 03:41

There is very small chances that satellite wrongly disqualified your node. Usually such can happen when we introduce a bug.
But I created a request to the on-call engineer anyway. Please, do not remove data for that satellite and keep logs too. I would notify you, if there is something extraordinary happened.

From this point I think the reason that your node failed audits because of timeouts. The worst case - because pieces are corrupted. If the last - your node likely can be disqualified on other satellites as well. The data corruption is rare affects only one folder on the disk.
Changing the network cannot affect the audit score unless you use a network-attached drives.

Since it’s disqualified only on one satellite, you can keep it running, the remained satellites will still pay for the service.

[jammerdan]

@Alexey

I would notify you, if there is something extraordinary happened.

Please let us know. One of my nodes got disqualified on Saltlake and Europe-North this weekend as well. The difference is that the node had an issue that I did not notice. So disqualification might be justified. However dashboard says:

saltlake.tardigrade.io:7777
Suspension 100 %
Audit 59.63 %
Online 97.32 %

europe-north-1.tardigrade.io:7777
Suspension 100 %
Audit 57.31 %
Online 97.26 %

This leaves me wondering how fast it has happened as it seems it got disqualified within a day. Is that intentional? For non enterprise home users that seems to be barely enough time to notice a failure and resolve it.

[allenyllee]

Hi Alexey,
Thanks for your cleanly explanation, I’ll keep the data on the Disk and leave the node running.
Do I need to stop my node to avoid data override by other running satellite?

[Alexey]

@jammerdan Please, search in logs - do you see any error from https://support.storj.io/hc/en-us/articles/4403035941780-Why-is-my-node-disqualified-

@allenyllee

Do I need to stop my node to avoid data override by other running satellite?

No, you do not need to stop it. The data belonging to the disqualified satellite will not be removed automatically.