Failed to settle orders for satellite "error": "order: sending settlement agreements returned an error: timeout: no recent network activity"

Bivvo · March 30, 2022, 4:26pm

I’ve added more log details from the 3 issues mentioned here. All around successful uploads, but turning into errors with order settlements:

github.com/storj/storj

Failed to settle orders errors: timeout + context cancelled + EOF issues

opened 02:31PM - 30 Mar 22 UTC

dusselmann

Bug

https://forum.storj.io/t/failed-to-settle-orders-for-satellite-error-order-sendi…ng-settlement-agreements-returned-an-error-timeout-no-recent-network-activity/17787/20 order: sending settlement agreements returned an error: timeout: no recent network activity ``` 2022-03-23T18:05:10.043Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 rpc client when sending new orders settlements {"error": "order: sending settlement agreements returned an error: timeout: no recent network activity", "errorVerbose": "order: sending settlement agreements returned an error: timeout: no recent network activity \tstorj.io/storj/storagenode/orders.(*Service).settleWindow:264 \tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205 \tgolang.org/x/sync/errgroup.(*Group).Go.func1:57", "request": {"limit":{"serial_number":"AAAAAADCHXXNEESKNWNUHB5TMU","satellite_id":"121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6","uplink_public_key":{},"storage_node_id":"1veqEG5xuBNkt6cK1NbmL5fbNxXq6E7JzB7CiiaptoapoWnB1i","piece_id":"KIWVOLO4ZO7TLEUJEVLFMAUPMBIGGWP774WEK2JR3OIZE32GLAUA","limit":2319360,"action":1,"piece_expiration":"0001-01-01T00:00:00Z","order_expiration":"2022-03-25T16:33:22.002663311Z","order_creation":"2022-03-23T16:33:22.002663311Z","encrypted_metadata_key_id":"4VwE2lxbA3w=","encrypted_metadata":"yhM8tiv9R2xW6BwPrIfWAZ8iMjp2/7AmVPL0RH40e/I45PwrGl88lletsqQdb8qQEVQ5+/xiXhzoJlwMZAma3w==","satellite_signature":"MEUCIBMatFEGFt7LpPJAifbAIzCjxboyN31UEjucZkk+rUQDAiEAvK7YjR07RDbZW1zic0HYnnV0eKWj1UHifiqQyGY+Q7Y="},"order":{"serial_number":"AAAAAADCHXXNEESKNWNUHB5TMU","amount":768,"uplink_signature":"LEURmLZurGkbaxCb9xOjZPTwWejX6nFMV0nDvRxJUWaadHhOmm83aHSfYRtu30NLyPF4jAhuViO8JmE1PF9NDQ=="}}} 2022-03-23T18:05:10.044Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 failed to settle orders for satellite {"satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "order: sending settlement agreements returned an error: timeout: no recent network activity", "errorVerbose": "order: sending settlement agreements returned an error: timeout: no recent network activity \tstorj.io/storj/storagenode/orders.(*Service).settleWindow:264 \tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205 \tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} ``` order: CloseAndRecv settlement agreements returned an error: context canceled ``` 2022-03-27T04:05:01.763Z ERROR orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo rpc client error when closing sender {"error": "order: CloseAndRecv settlement agreements returned an error: context canceled", "errorVerbose": "order: CloseAndRecv settlement agreements returned an error: context canceled \tstorj.io/storj/storagenode/orders.(*Service).settleWindow:275 \tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205 \tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} 2022-03-27T04:05:01.763Z ERROR orders.12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo failed to settle orders for satellite {"satellite ID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "error": "order: CloseAndRecv settlement agreements returned an error: context canceled", "errorVerbose": "order: CloseAndRecv settlement agreements returned an error: context canceled \tstorj.io/storj/storagenode/orders.(*Service).settleWindow:275 \tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205 \tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} ``` order: sending settlement agreements returned an error: EOF ``` 2022-03-25T19:04:59.856Z ERROR orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE rpc client when sending new orders settlements {"error": "order: sending settlement agreements returned an error: EOF", "errorVerbose": "order: sending settlement agreements returned an error: EOF \tstorj.io/storj/storagenode/orders.(*Service).settleWindow:264 \tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205 \tgolang.org/x/sync/errgroup.(*Group).Go.func1:57", "request": {"limit":{"serial_number":"AAAAAADCICOTJ5WA2KWJO2U5RY","satellite_id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","uplink_public_key":{},"storage_node_id":"1veqEG5xuBNkt6cK1NbmL5fbNxXq6E7JzB7CiiaptoapoWnB1i","piece_id":"KMWHHFOATLSIW3HFURIHIEL74EXYYA2GPRSUJLXN7JU3A4LT5UJQ","limit":2317056,"action":2,"piece_expiration":"0001-01-01T00:00:00Z","order_expiration":"2022-03-27T17:21:56.060000195Z","order_creation":"2022-03-25T17:21:56.060000195Z","encrypted_metadata_key_id":"QDs91VL1vug=","encrypted_metadata":"uODGH1Cc8qNASu0Enc8oO0HPj44MN9WJOsLqC0IQAr6E9OEc63k=","satellite_signature":"MEQCIFadOCNgDRQ7bRCAldr3B6l8tBrQZ6yVZ4OF45LIGhClAiBs/Nn4e37Jm9d+yb3dKSoUriDnz9Y1NUNt3aD2sgtxYw=="},"order":{"serial_number":"AAAAAADCICOTJ5WA2KWJO2U5RY","amount":2105344,"uplink_signature":"0/c37HCIRn08fif9YRSY5c7amb7aIpHEVKXoH7R5ZHwTB65xIQkII8KKWS9VHMr6mDOJWXMP+vwDic1NPKk3BA=="}}} 2022-03-25T19:04:59.857Z ERROR orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE failed to settle orders for satellite {"satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "order: sending settlement agreements returned an error: EOF", "errorVerbose": "order: sending settlement agreements returned an error: EOF \tstorj.io/storj/storagenode/orders.(*Service).settleWindow:264 \tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func1:205 \tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"} ``` @thepaul : "The “no recent network activity” error message is quic-go’s version of ETIMEDOUT (“connection timed out”). It’s a very confusing message, I agree."

cc @deathlessdd