Failed to settle orders: EOF, deadlock detected?

naxbc · September 2, 2019, 7:46am

Hi guys.
I still have this error in my main node after the update to 0.19.5
Is this an expected behaviour?

ifraixedes · September 2, 2019, 11:05am

Please see Workaround for order sender errors (execute this before they expire)

Thanks for reporting it.

naxbc · September 2, 2019, 11:53am

Just rebooted my SNO as in the workaround…issue remains

ifraixedes · September 2, 2019, 12:07pm

You may need to restart the Storage Node several times, but even with that, it may not be fixed.

We have implemented a workaround and we’re now planning to release it as soon as possible.

Thanks for your patience.

naxbc · September 2, 2019, 1:00pm

Thank you, but in that case I´ll just wait. I prefer not to rebooted as I´m away from my node, and don´t want to risk being DQ´ed. I´ll wait for a release fix.

cdhowie · September 2, 2019, 7:45pm

@ifraixedes The update to v0.19.6 has not resolved the issue on my node. Here is the relevant log snippet:

2019-09-02T19:34:44.813Z        INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       sending {"count": 24}
2019-09-02T19:34:45.066Z        INFO    version running on version v0.19.6
2019-09-02T19:35:30.262Z        ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       gRPC client error when receiveing new order settlements       {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-02T19:35:30.265Z        INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       finished
2019-09-02T19:35:30.267Z        ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       failed to settle orders       {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}

Please advise what I should do to correct this problem. My node has been doing this for several days (on v0.19.5 as well).

Dylan · September 2, 2019, 9:42pm

Hey @cdhowie and welcome to the forums!

Please, check out this thread:

https://forum.storj.io/t/workaround-for-order-sender-errors-execute-this-before-they-expire/1399/3

cdhowie · September 2, 2019, 10:04pm

@Dylan That fix works on occasion. I’m mostly commenting here to point out that the fix in v0.19.6 has not resolved the problem; these errors persist after upgrading.

naxbc · September 2, 2019, 10:17pm

Any chance these errors cause a disqualification or reputation downgrade?

ifraixedes · September 3, 2019, 6:23am

No, they don’t cause any of them.

naxbc · September 3, 2019, 7:34am

Ok, thank you. I´ll just leave it at it is then.

ifraixedes · September 3, 2019, 12:19pm

We are figuring out why the deadlocks are happening.
As we commented the v0.19.6 is a workaround not a solution.

I’m asking my mates if they can give you some advise because I cannot.

Thanks for your patience.

naxbc · September 3, 2019, 12:39pm

Thank you.
I´m good with them until a solution is figured out, as long as it doesn´t cause DQ or reputation harm. This is what a Beta is all about no worries.

xyphos10 · September 3, 2019, 2:25pm

Hello everyone, my node is the same, even in 0.19.6 I am getting the deadlock message.

2019-09-03T10:45:38.497446826Z 2019-09-03T10:45:38.497Z INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      sending {"count": 11}
2019-09-03T10:45:38.497495808Z 2019-09-03T10:45:38.497Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      sending {"count": 54}
2019-09-03T10:45:38.497504213Z 2019-09-03T10:45:38.497Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       sending {"count": 632}
2019-09-03T10:45:38.497585155Z 2019-09-03T10:45:38.497Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      sending {"count": 5}
2019-09-03T10:45:39.206828353Z 2019-09-03T10:45:39.206Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      finished
2019-09-03T10:45:39.312526769Z 2019-09-03T10:45:39.311Z INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      finished
2019-09-03T10:45:39.623264785Z 2019-09-03T10:45:39.623Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      finished
2019-09-03T10:45:59.890343581Z 2019-09-03T10:45:59.889Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       gRPC client error when receiveing new order settlements {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-03T10:45:59.890392923Z 2019-09-03T10:45:59.890Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       finished
2019-09-03T10:45:59.890505163Z 2019-09-03T10:45:59.890Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       failed to settle orders {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-03T11:45:38.510625093Z 2019-09-03T11:45:38.494Z INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      sending {"count": 1}
2019-09-03T11:45:38.510635652Z 2019-09-03T11:45:38.494Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       sending {"count": 677}
2019-09-03T11:45:38.510639590Z 2019-09-03T11:45:38.494Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      sending {"count": 48}
2019-09-03T11:45:38.510643187Z 2019-09-03T11:45:38.494Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      sending {"count": 8}
2019-09-03T11:45:39.182965267Z 2019-09-03T11:45:39.182Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      finished
2019-09-03T11:45:39.398953207Z 2019-09-03T11:45:39.398Z INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      finished
2019-09-03T11:45:39.797840844Z 2019-09-03T11:45:39.797Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      finished
2019-09-03T11:46:58.743741991Z 2019-09-03T11:46:58.740Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       gRPC client error when receiveing new order settlements {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-03T11:46:58.743794289Z 2019-09-03T11:46:58.741Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       finished
2019-09-03T11:46:58.743800681Z 2019-09-03T11:46:58.741Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       failed to settle orders {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-03T12:45:38.494997679Z 2019-09-03T12:45:38.494Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      sending {"count": 6}
2019-09-03T12:45:38.495086756Z 2019-09-03T12:45:38.494Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      sending {"count": 51}
2019-09-03T12:45:38.495167076Z 2019-09-03T12:45:38.494Z INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      sending {"count": 1}
2019-09-03T12:45:38.495183517Z 2019-09-03T12:45:38.495Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       sending {"count": 731}
2019-09-03T12:45:39.299850481Z 2019-09-03T12:45:39.299Z INFO    orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      finished
2019-09-03T12:45:39.307749743Z 2019-09-03T12:45:39.307Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      finished
2019-09-03T12:45:39.592468396Z 2019-09-03T12:45:39.592Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      finished
2019-09-03T12:46:35.226016925Z 2019-09-03T12:46:35.225Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       gRPC client error when receiveing new order settlements {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-03T12:46:35.226092807Z 2019-09-03T12:46:35.225Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       finished
2019-09-03T12:46:35.226104909Z 2019-09-03T12:46:35.225Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       failed to settle orders {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-03T13:45:38.498570764Z 2019-09-03T13:45:38.498Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      sending {"count": 6}
2019-09-03T13:45:38.498632730Z 2019-09-03T13:45:38.498Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      sending {"count": 45}
2019-09-03T13:45:38.498678746Z 2019-09-03T13:45:38.498Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       sending {"count": 781}
2019-09-03T13:45:39.420122660Z 2019-09-03T13:45:39.419Z INFO    orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      finished
2019-09-03T13:45:39.651850706Z 2019-09-03T13:45:39.651Z INFO    orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      finished
2019-09-03T13:46:11.924526591Z 2019-09-03T13:46:11.923Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       gRPC client error when receiveing new order settlements {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}
2019-09-03T13:46:11.924605308Z 2019-09-03T13:46:11.923Z INFO    orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       finished
2019-09-03T13:46:11.924618593Z 2019-09-03T13:46:11.923Z ERROR   orders.118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW       failed to settle orders {"error": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected", "errorVerbose": "order: failed to receive settlement response: rpc error: code = Unknown desc = pq: deadlock detected\n\tstorj.io/storj/storagenode/orders.(*Service).settle:311\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:191\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrders.func2:170\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57"}