Suspension Problems on 3 different nodes

kbch · January 30, 2022, 11:39am

Facing the issue on 3 different node sthat the suspension is going down. One node even got suspended. Not sure if this is related to v1.47.3. Interesting is also that us is still 100%. What can I do?

kbch · January 30, 2022, 11:59am

Restarted the node and now it seems that the numbers are recovering. No idea what is happening with it.

Could not working QUIC be the issue? Configured it to work on all nodes now…

Alexey · January 30, 2022, 1:36pm

No.
The suspension score is falling when something wrong with pieces.
Please, check for the reasons: https://support.storj.io/hc/en-us/articles/360042257912-Suspension-mode

kbch · January 30, 2022, 1:54pm

Ok, looks like this was a temporary Problem. This means that the suspension value should recover again over time and my node will get unsuspended at some point?

2022-01-30T07:01:21.566Z	ERROR	piecestore	download failed	{"Piece ID": "QF4EO7L6SFHWQCTLFRLSU7PZHOZJAX3XEPPYG5RMYJ6RRMY6H26Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:39288: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:39288: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T07:09:39.998Z	ERROR	piecestore	download failed	{"Piece ID": "4CVGETDUCVTWKARG7ATOFWU2VPAXF6CVG6EV2LSZUQ7P5V55OQSA", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:50060: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:50060: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T07:17:34.809Z	ERROR	piecestore	download failed	{"Piece ID": "SIRMA56PVV3KBZEWADQPVLQJPQZGKBD43HK2FZMR6KNKUCICPJ7A", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:44124: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:44124: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T07:17:51.516Z	ERROR	piecestore	download failed	{"Piece ID": "EQSBGAYFKHEU3W6F2NQE2PKZVXDL3U2P3R6DWOZUB5AZQZMJNH6Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:55550: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:55550: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T07:35:40.574Z	ERROR	piecestore	download failed	{"Piece ID": "FHANAMKSC3SCGBCISHXN2XIIYH5O2UHK6CRSSGK3XGBTHIQQTX7Q", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:54162: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:54162: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T07:45:27.580Z	ERROR	piecestore	download failed	{"Piece ID": "5J76SPO23RQ6MGQM5KUOUKRZSGFXRE5II37QVP3V5NLOBOGR26XA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:48518: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:48518: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T07:59:33.661Z	ERROR	piecestore	download failed	{"Piece ID": "S3LNFDLBFXELG35BQGUIT6VCEDCHUPJFRAHKMZ3UMNLM65OXZQIA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:55444: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:55444: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T07:59:49.789Z	ERROR	piecestore	download failed	{"Piece ID": "V4UIO4IWOCLI7PKKOXSUYYCSIFTDXESMXKY325VEJ5IDIKUIPB6Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:60566: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:60566: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}
2022-01-30T08:03:12.892Z	ERROR	piecestore	download failed	{"Piece ID": "5CFQB64P2UOLHTTMHKWCH4ADFQOV2S557V2AL4I3UPPFVXVDWRTQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "GET_REPAIR", "error": "write tcp 172.17.0.3:28967->10.34.0.1:37350: write: connection timed out", "errorVerbose": "write tcp 172.17.0.3:28967->10.34.0.1:37350: write: connection timed out\n\tstorj.io/drpc/drpcstream.(*Stream).rawWriteLocked:317\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:392\n\tstorj.io/common/pb.(*drpcPiecestore_DownloadStream).Send:317\n\tstorj.io/storj/storagenode/piecestore.(*Endpoint).Download.func5.1:619\n\tstorj.io/common/rpc/rpctimeout.Run.func1:22"}

Alexey · January 30, 2022, 2:45pm

If the problem is fixed, the suspension score will recover with each successful audit and GET_REPAIR.
Looking on your errors I see a network issue. Perhaps you need to reboot this PC, or at least to recreate the container (please note, if you didn’t redirect your logs to the file, then your logs will be deleted with the container).
The reboot in this case is better, because the container will not be re-created and the logs will remain.