US 1 down and can't accept storagenode orders

Since there is an outage at US1 orders are not getting settled in and throw this error.

2024-06-11T04:14:21Z ERROR orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S rpc client error when closing sender      { "error": "order: CloseAndRecv settlement agreements returned an error: process order with window transaction: create storagenode bandwidth: satellitedb: ERROR: replica unavailable: (n17,s17):117 unable to serve request to r23907:/Table/273/1/\"\\x9{b\\xf2\\xe5\\xfa\\x920\\xea\\xcd\\xc0!\\xfa\\xa7\\xacq՜Ԝ5\\xd4wC˜#A\\xa3\\xf0\\x00\\x00\\x00\\x00\"/2021-12-03T10:00:00Z/4-f\\xae\\xe9\\x8f\\v~\\U000192d4+\\xbb)\\xef-@\\x83\\x9c'\\x05\\xbd\\xe67\\x8e\\x18\\x9f\\x05@\\x00\\x00\\x00\\x00\"/2023-03-09T01:00:00Z/4} [(n17,s17):117, (n18,s18):118, (n29,s29):119, next=120, gen=895]: lost quorum (down: (n18,s18):118,(n29,s29):119); closed timestamp: 1718072717.789146557,0 (2024-06-11 02:25:17); raft status: {\"id\":\"75\",\"term\":1423,\"vote\":\"75\",\"commit\":79589750,\"lead\":\"0\",\"raftState\":\"StatePreCandidate\",\"applied\":79589750,\"progress\":{},\"leadtransferee\":\"0\"}: have been waiting 63.00s for slow proposal ConditionalPut [/Table/273/1/\"\\x9c\\xea\\x1c\\x1c\\x12\\xfe\\xbb\\x1bc\\x12}\\x8c\\xbc\\xf1E\\xd8o\\xf7ƚ\\xffTQ2.\\xf6\\n\\xd0\\x00\\x00\\x00\\x00\"/2024-06-11T00:00:00Z/4/0], [txn: 84a3c3ba] (SQLSTATE XXUUU)", "errorVerbose": "order: CloseAndRecv settlement agreements returned an error: process order with window transaction: create storagenode bandwidth: satellitedb: ERROR: replica unavailable: (n17,s17):117 unable to serve request to r23907:/Table/273/1/\"\\x9{b\\xf2\\xe5\\xfa\\x920\\xea\\xcd\\xc0!\\xfa\\xa7\\xacq՜Ԝ5\\xd4wC˜#A\\xa3\\xf0\\x00\\x00\\x00\\x00\"/2021-12-03T10:00:00Z/4-f\\xae\\xe9\\x8f\\v~\\U000192d4+\\xbb)\\xef-@\\x83\\x9c'\\x05\\xbd\\xe67\\x8e\\x18\\x9f\\x05@\\x00\\x00\\x00\\x00\"/2023-03-09T01:00:00Z/4} [(n17,s17):117, (n18,s18):118, (n29,s29):119, next=120, gen=895]: lost quorum (down: (n18,s18):118,(n29,s29):119); closed timestamp: 1718072717.789146557,0 (2024-06-11 02:25:17); raft status: {\"id\":\"75\",\"term\":1423,\"vote\":\"75\",\"commit\":79589750,\"lead\":\"0\",\"raftState\":\"StatePreCandidate\",\"applied\":79589750,\"progress\":{},\"leadtransferee\":\"0\"}: have been waiting 63.00s for slow proposal ConditionalPut [/Table/273/1/\"\\x9c\\xea\\x1c\\x1c\\x12\\xfe\\xbb\\x1bc\\x12}\\x8c\\xbc\\xf1E\\xd8o\\xf7ƚ\\xffTQ2.\\xf6\\n\\xd0\\x00\\x00\\x00\\x00\"/2024-06-11T00:00:00Z/4/0], [txn: 84a3c3ba] (SQLSTATE XXUUU)\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:310\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}

This is just a post to make others aware they are not alone and there is no reason to panic.

5 Likes

This issue is already resolved.