Failed to receive settlement response: only specified storage node can settle order

Node Operators troubleshooting
1

Currently have 2330 unsettled orders in unsent. The errors below keep popping up every 5 mins. I moved all the unsent orders out of the unsent folder. New unsent orders were generated and the errors came back. Node is running 1.13.3. Ubuntu minimal 20.04.

2020-10-06T09:35:33.474Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-06T09:35:33.474Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“AAAAAAC7FYDFO4I2QGANV22QIY”,“satellite_id”:“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,“uplink_public_key”:{},“storage_node_id”:“1kZjTB2oHVYtdoHsnB5pkLkDBbCurgBAqwxXxUpSjz6gSra3SR”,“piece_id”:“U6G7RFLAPJIWZS425ITY5XDQN6WUCVPIQKHH4RLRQ4YP2UB3QH6Q”,“limit”:2319360,“action”:5,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-08-08T01:56:39.358176311Z”,“order_creation”:“2020-08-06T01:56:39.358610191Z”,“satellite_signature”:“MEUCIQCc3UQ10gPfczq9yH39b9AgHuJf/c8TVextk/CPBlNfugIgEvcfSkfd4EvdDNcxMe+ujw8LeF00nfsKzdNOq700Tjo=”,“satellite_address”:{}},“order”:{“serial_number”:“AAAAAAC7FYDFO4I2QGANV22QIY”,“amount”:2319360,“uplink_signature”:“Dx/y9YNszNu2L13QJ28pHd98zvvy5lU+klxuaXB8GXVhpEyB7smYbWxn6ygrve2aeEjb4xWRRkhk6/iZYnZCDg==”}}}
2020-10-06T09:35:33.475Z ERROR orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-06T09:35:33.596Z ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-06T09:35:33.796Z ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“AAAAAAC7FYDG4WNLGGXF6LUB4M”,“satellite_id”:“12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S”,“uplink_public_key”:{},“storage_node_id”:“1kZjTB2oHVYtdoHsnB5pkLkDBbCurgBAqwxXxUpSjz6gSra3SR”,“piece_id”:“YRLLFF2Y63IO2UDGCGKC332A5SSKMVRRJH4QNTREP5WFP2RYUPKQ”,“limit”:1236224,“action”:5,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-08-08T01:57:02.338246846Z”,“order_creation”:“2020-08-06T01:57:02.340568343Z”,“satellite_signature”:“MEUCIH0PUGZQXPp2uHp+YVWTw5HruTng7Dpz1FNDuISeHYNkAiEAlRtouFV04jQ9W7j6jj+5uQWRlIWXy/Mn2CEceYVP/+M=”,“satellite_address”:{}},“order”:{“serial_number”:“AAAAAAC7FYDG4WNLGGXF6LUB4M”,“amount”:1236224,“uplink_signature”:“EHn23bMQa+BbJlUdAFxpc44YxmduTad1fWXFSLs6G2YhqXJwXuKE0NaV0MhnYgzFUo2PSXQLa2FjfM9nXsctCQ==”}}}
2020-10-06T09:35:33.798Z ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-06T09:35:33.878Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-06T09:35:33.879Z ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs rpc client error when receiving new order settlements {“error”: “order: failed to receive settlement response: only specified storage node can settle order”, “errorVerbose”: “order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-06T09:35:33.880Z ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“AAAAAAC7FYDYN7BSUFVFDKA4XM”,“satellite_id”:“12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs”,“uplink_public_key”:{},“storage_node_id”:“1kZjTB2oHVYtdoHsnB5pkLkDBbCurgBAqwxXxUpSjz6gSra3SR”,“piece_id”:“ULK4Z25LOJYCULKKHK5VUB77BEEH2LQR73SX3TX5OHV6AKSGPYWQ”,“limit”:2319360,“action”:1,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-08-08T02:01:42.565818262Z”,“order_creation”:“2020-08-06T02:01:42.569935841Z”,“satellite_signature”:“MEUCIESYLfja501oCkJBgBMxGVYhGwitulQnzls2+qrmCMMSAiEA9GVA5NbvmQbyMOQvMuPHA0VpUc9CYJl3nPjU8Y1m2CU=”,“satellite_address”:{}},“order”:{“serial_number”:“AAAAAAC7FYDYN7BSUFVFDKA4XM”,“amount”:4608,“uplink_signature”:“LT2UIe0D0Ts7tOfnF/z3GcbwXvcYtGng27oanB/0RU/Rvr9XOlDtCLgCOQTDCWo0gPBkVvbVBme6SPx29O61AQ==”}}}
2020-10-06T09:35:33.882Z ERROR orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}
2020-10-06T09:35:33.882Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB rpc client when sending new orders settlements {“error”: “order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”, “request”: {“limit”:{“serial_number”:“AAAAAAC7FZH2CAYOHF57XARAWM”,“satellite_id”:“12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB”,“uplink_public_key”:{},“storage_node_id”:“1Fgu7ESXYYhkQ15ceCdmAbb7QWh8QM3rcNZw8Dfs84E9Jhai4Y”,“piece_id”:“P7D4NRUWRLC4PWA25KW65YAE3U362A53TJDAPVTTDSR7F2ILPG4A”,“limit”:2319360,“action”:1,“piece_expiration”:“0001-01-01T00:00:00Z”,“order_expiration”:“2020-08-08T07:09:21.596477844Z”,“order_creation”:“2020-08-06T07:09:21.596624398Z”,“satellite_signature”:“MEYCIQC6VY21O+NCUdsR5ayiBgyoQMjZO+D+kyBkbQvj+NNwQAIhAO0GPt+nw7qadFmZEAdXgIeD0DHNmgA82gNuyCZfmlB/”,“satellite_address”:{}},“order”:{“serial_number”:“AAAAAAC7FZH2CAYOHF57XARAWM”,“amount”:768,“uplink_signature”:“6zHbvKkMrIzGyuUmJ+cOE2Z/yZHSAci93RASZXT8C3b14SFpf0ZQlogVd7Unz+WLMvr2X4XeQ4zFjwtEEx7rCw==”}}}
2020-10-06T09:35:33.884Z ERROR orders.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB failed to settle orders {“error”: “order: failed to receive settlement response: only specified storage node can settle order; order: sending settlement agreements returned an error: EOF”, “errorVerbose”: “group:\n— order: failed to receive settlement response: only specified storage node can settle order\n\tstorj.io/storj/storagenode/orders.(*Service).settle:355\n\tstorj.io/storj/storagenode/orders.(*Service).Settle:246\n\tstorj.io/storj/storagenode/orders.(*Service).sendOrdersFromDB.func2:224\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57\n— order: sending settlement agreements returned an error: EOF\n\tstorj.io/storj/storagenode/orders.(*Service).settle.func2:327\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:57”}

2

Something is wrong with your identity. You have orders for a different identity in your database. Now the question is how did you mange to do that? Merging 2 nodes is not possible.

3

it is very easy, if he moved nodes or has 2 nodes there.

4

I did have a think about this yesterday and came to the same conclusion. I decided to drop the orders database and recreate a blank one. Once I did that everything ran well. The unsent orders got processed. It took a while, there were a lot.

I have been doing a upgrade of the OS and OS disk. Went minimal OS as the whole desktop thing is always a pain on a server. One issue with Ubuntu is it automounts USB drives and the automount can over ride fstab on bootup even when I use blkid for each drive. I have now disabled automount but perhaps not soon enough as I do run two nodes on this platform, each on a separate drive. Easy to see how incorrect drive mounting could cause node ID issues.