Unsent orders stop sending on old version

Node Operators troubleshooting
1

I just reviewed payouts this month and noticed that my node had 0 payout this month. I manually updated the node executable from 1.12.3 to 1.13.3 (as well as the updater). I also found thousands of these orders unsent which have started processing down since updating/restarting. What is the role of the orders DB and would this error (which persisted all through September based on my logs) affect payouts in October?

UPDATE: unsent orders just stopped with 125 items… oldest is 8/29 - 9/3 and then the rest are dated 10/8 based on file modified date.

UPDATE: the unsent orders from 8/29 - 9/3 are all from satellite 118U/AW which is offline now. SO I assume this can be deleted?

2

Please, enable an automatic updates as soon as possible, because if the minimum version is introduced, your node will stop receive traffic and send orders to be paid.

3

Yes, the unsent orders from 118 satellite can be deleted because this satellite has shutdown.

4

this is something that has started to show in my logs about an hour ago it’s unrelated to sat 118,2020-10-10T19:54:09.048Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB rpc client error when closing sender {“error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:491\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-10T19:54:09.048Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2020-10-10T19:54:09.048Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB failed to settle orders for satellite {“satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:491\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

6

Me too. Seems there is contention at server end.

7

Looks like sat went down. seems back to normal now

8

no it keeps resending still,2020-10-10T20:27:25.173Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB sending {“count”: 896}and,2020-10-10T20:29:02.267Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB rpc client error when closing sender {“error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:491\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-10T20:29:02.267Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2020-10-10T20:29:02.267Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB failed to settle orders for satellite {“satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “error”: “order: CloseAndRecv settlement agreements returned an error: context canceled”, “errorVerbose”: “order: CloseAndRecv settlement agreements returned an error: context canceled\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:491\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}according to uptime robot it has been up for 557 hrs, 24 mins

9

Yep, its not fixed and sat doesn’t show down too. Its PUTing without any problem

10

Yup your right I thought it completed but then I started getting more errors again…

11

I’ve also issues there.

2020-10-10T20:31:51.441Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: write tcp 172.17.0.2:40556->35.228.191.177:7777: use of closed network connection”, “errorVerbose”: “order: sending settlement agreements returned an error: write tcp 172.17.0.2:40556->35.228.191.177:7777: use of closed network connection\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:480\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“AAAAAAC7QSSXG2X5HYO2HZ45Y4”,“satellite_id”:“12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”,“uplink_public_key”:{},“storage_node_id”:“12B5suGhHi6Wfqx562gRRxm9zD4nuvb2Tonrk9tzkERnzSgRngf”,“piece_id”:“LTRNRT4C2FXGNLVSFRUGE4YVM6S3BECQQJHIISRE5REW3TYAOWMQ”,“limit”:2319360,“action”:1,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-10-12T18:50:27.466774601Z”,“order_creation”:“2020-10-10T18:50:27.466774601Z”,“satellite_signature”:“MEUCIDDRVq1UmhV+pdPoRO9oT+1LOm5nP6MKYBM+I7FMmu9PAiEA8uXwgnVChr8dSPKIZtAFg4JC8k0Cpic9iEPe8Mkf44Y=”,“satellite_address”:{}},“order”:{“serial_number”:“AAAAAAC7QSSXG2X5HYO2HZ45Y4”,“amount”:768,“uplink_signature”:“zSsRatT9BVAQht+7hd0AKdhsFlOgYcEryJs2KIFuhDEWZCjlI3XuxOC87npZjoTxgbGK2NRg7QkruQXWCm33Bw==”}}}
2020-10-10T20:31:51.442Z INFO orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB finished
2020-10-10T20:31:51.442Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB failed to settle orders for satellite {“satellite ID”: “12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”, “error”: “order: sending settlement agreements returned an error: write tcp 172.17.0.2:40556->35.228.191.177:7777: use of closed network connection”, “errorVerbose”: “order: sending settlement agreements returned an error: write tcp 172.17.0.2:40556->35.228.191.177:7777: use of closed network connection\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:480\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

12

agreed, have you checked your web dash lately for saltlake sat on this node it went to this for no reasonScreen Shot 2020-10-10 at 16.35.01 at least that I can see

13

Mine still is fine. still both 100%

1 Like
14

mine is at 100% too but I got errors submitting orders too now… connection reset by peer so something server-side

15

hope the genius bar is open, and I don’t know what if anything I can do about the above suspension score :upside_down_face:

16

Sending orders from DB didn’t cause this much heartache to my node.

17

Thats not the satellite with the errors though, its europe-north-1.tardigrade

18

It only started a few hours ago. Maybe someone who’s going to be online a while longer can contact the support? Maybe that’s the fastest way of making them aware of a server-side problem? (I’m going to bed in a few minutes otherwise I’d do it)

2020-10-10T20:26:17.910Z        ERROR   orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB      rpc client when sending new orders settlements  {"error": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:41518->35.228.191.177:7777: write: broken pipe", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:41518->35.228.191.177:7777: write: broken pipe\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:480\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57", "request": {"limit":{"serial_number":"AAAAAAC7QSR3LY4X27U6MNP4ZI","satellite_id":"12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB","uplink_public_key":{},"storage_node_id":"1cYRm7virx4Z7xXvgG9SxMBfJqkEA2XDoy6bWXfxkf2vLsmWjV","piece_id":"UWB26PYKTWSFLOJ6CDV426P3RRSB3MU7J5PJSKXFQSDL6X7JUGSA","limit":2319360,"action":1,"piece_expiration":"0001-01-01T00:00:00Z","order_expiration":"2020-10-12T18:43:01.957141704Z","order_creation":"2020-10-10T18:43:01.957141704Z","satellite_signature":"MEQCIBTAbS/0HTZwbc6iUCINXcvxcl6MhxbDViNskFfG2eiUAiAJhDt5UTaWF199bq2ys0a8XJHDhFtv3am4WarHlzPxBQ==","satellite_address":{}},"order":{"serial_number":"AAAAAAC7QSR3LY4X27U6MNP4ZI","amount":768,"uplink_signature":"64Z7vWN/HYSL591fTDA9pqJ0kg/l+ACxxyKtrQPzUu2G8r4AReb1dFOQs8MxaLjGHRiUfBY/vg85tXb6MTycDw=="}}}
2020-10-10T20:26:17.910Z        ERROR   orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB      failed to settle orders for satellite   {"satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:41518->35.228.191.177:7777: write: broken pipe", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:41518->35.228.191.177:7777: write: broken pipe\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:480\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2020-10-10T20:34:55.022Z        ERROR   orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB      rpc client when sending new orders settlements  {"error": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:50582->35.228.191.177:7777: write: broken pipe", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:50582->35.228.191.177:7777: write: broken pipe\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:480\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57", "request": {"limit":{"serial_number":"AAAAAAC7QSQQJ7WQARQSVPHGHU","satellite_id":"12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB","uplink_public_key":{},"storage_node_id":"1cYRm7virx4Z7xXvgG9SxMBfJqkEA2XDoy6bWXfxkf2vLsmWjV","piece_id":"MGJZABFTTMYM6HDTYVC22ECMHU2XZ7O3OSJ4UOMD7JUZQ3VDD7CA","limit":2319360,"action":1,"piece_expiration":"0001-01-01T00:00:00Z","order_expiration":"2020-10-12T18:31:32.967011747Z","order_creation":"2020-10-10T18:31:32.967011747Z","satellite_signature":"MEUCIA7MrThrza1ziyLZ6q0S7ssy6+rrgXCfYhQreNdH8kzzAiEAq5wZh8VUJSI3G2bW9fWK0wd/g+JKt3aGHJFzhlOjU5M=","satellite_address":{}},"order":{"serial_number":"AAAAAAC7QSQQJ7WQARQSVPHGHU","amount":768,"uplink_signature":"DcPrQi8nbDdgTHH6S1EZjzxV9uVhIFFvICWQJlk2JVBGS0d1VpUvYQBfwnpXlTE6opIBgDs50RYisNUeWRwUDg=="}}}
2020-10-10T20:34:55.022Z        ERROR   orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB      failed to settle orders for satellite   {"satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:50582->35.228.191.177:7777: write: broken pipe", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.17.0.14:50582->35.228.191.177:7777: write: broken pipe\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:480\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromFileStore.func1:422\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
19

@deathlessdd no its not but I have not made any changes other than ip add.

20

are we all docker as I am not seeing the error on the .exe node

21

Yeah I am on docker. I see the same error on all 4 of my nodes…