The satellite lag while GE

Node Operators troubleshooting
1

Previously, there were “connection reset by peer” errors only from bad slow nodes, but now the satellite has started to give me such an error! I have more than 8 thousand such entries in the log

{"L":"ERROR","T":"2023-04-22T07:18:54.351Z","N":"gracefulexit:chore.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE@saltlake.tardigrade.io:7777","M":"failed to send notification about piece transfer.","Process":"storagenode","Satellite ID":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","error":"write tcp 10.16.0.2:33892->34.94.153.46:7777: write: connection reset by peer","errorVerbose":"write tcp 10.16.0.2:33892->34.94.153.46:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}

image
image947×116 17.6 KB

image
image1452×532 63 KB

2

Notified the team. If you tuned GE parameters, please reset them to default (especially workers).

3

I was too lazy to change the parameters for this small node, so everything is standard. In principle, I don’t care about the loss of 4 cents from this failed exit, I just wanted to get beautiful statistics in my dashboard instead of DQ status.

4

Yes, I understand. There is a work in progress regarding GE and timeouts: GE's failing? for no apparent reason - #31 by thepaul
So it hopefully would be fixed.