Orders are not sending for US1 and Saltlake

Node Operators troubleshooting
, ,
1

I had a look and saw this error this month

2024-06-09T10:59:04+02:00       ERROR   orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      rpc client when sending new orders settlements  {"Process": "storagenode", "error": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "request": {"limit":{"serial_number":"AAAAAADGM2NFKLMEBETN3M7IPA","satellite_id":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","uplink_public_key":"GKWPCMPSIFSGBFOHOB33IV4L6MOAJVPAZH6W3AEJJABW52VG76PA","storage_node_id":"12vjmGaoza79G3BwA9KkvED8KbAY5q1geZvrNfWLnUj7JJSNYU1","piece_id":"RNYVTADYWZHBLD3WFIZ2UBOCUHBUDG2XLS7XA3IU2N42RUDUGJ2Q","limit":2319360,"action":1,"piece_expiration":"0001-01-01T00:00:00Z","order_expiration":"2024-06-10T06:16:53.732902055Z","order_creation":"2024-06-09T06:16:53.732902055Z","encrypted_metadata_key_id":"+Qu3n5/wGic=","encrypted_metadata":"Spu0mBoHsSNaFKIxfDAmXxQrKvqcYuGRfXdSTFqvYDlvPjsbUurdsQyINp328ZkSkUr/vWpCrWJpWw==","satellite_signature":"MEUCIQDCmXuHJo+v+uYvb6uqZnG81vzKes9U6XbBvBXC3cCqsAIgcGdWoAhE2YmVpaAoDqbb1H8BuGPfE1tRzeoAHeV5/is="},"order":{"serial_number":"AAAAAADGM2NFKLMEBETN3M7IPA","amount":262144,"uplink_signature":"mYNAvhYZ1I+710olHtIJTZPkJaRryJ3A9f+LkRg2rE9EOh7SfYDPPx/KEkffVugGfl1heXBVrZhBNGQ1S0s5Cw=="}}}
2024-06-09T10:59:04+02:00       ERROR   orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}

2024-06-03T23:32:22+02:00       ERROR   orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       rpc client when sending new orders settlements  {"Process": "storagenode", "error": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "request": {"limit":{"serial_number":"AAAAAADGL5SXAZ3URYDH7LSH2E","satellite_id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","uplink_public_key":"QGJATLLFZP57C7CGKX5RAPYTK45LKM2A3JDCXHPXNWDQXSZN4LIA","storage_node_id":"1CPsVCepUBZgo81W5vk5mhnoj6yA3E4qjMDAkjakXxEKq3u9WZ","piece_id":"XZZX7P4GH5HWSMBVDT3PYHBPIDOFO6GV7KTE2DGGNGGZJCFDL2WQ","limit":4203520,"action":1,"piece_expiration":"2024-07-03T19:05:20.52287Z","order_expiration":"2024-06-04T19:05:20.566908198Z","order_creation":"2024-06-03T19:05:20.566908198Z","encrypted_metadata_key_id":"QDs91VL1vug=","encrypted_metadata":"eS7pDVDgV27ZMUR4N1btQBimpHOyOTw1JcW81CvX4Gh3Dp5HxpGTP5mRaIMQB+U=","satellite_signature":"MEUCIG7pbo7zg1rSwPVUBRqFS7jyp/qhBsgAt4cyynOqtjyiAiEAha9sj+ORpfGWtGmSPgf2jKD+PneWDlBp54ZipBq1QIY="},"order":{"serial_number":"AAAAAADGL5SXAZ3URYDH7LSH2E","amount":262144,"uplink_signature":"qnCRh4WAVWcS9uxPZNiDyBgofgbRioinmF/fwOWzH19gwFBwHZjgrG44DY2nDBJCKPs62spAb5vOxdr6uE2KDg=="}}}
2024-06-03T23:32:22+02:00       ERROR   orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2

How many such errors in the logs?
Do you have other errors related to orders?

3

these are all errors since 01.05.
I think the filewalker error came from a node restart

Node 1 
2024-05-02T04:47:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-02T04:47:03+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "context canceled"}
2024-05-02T04:47:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-02T04:47:03+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
2024-05-02T04:47:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-02T04:47:03+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2024-05-02T04:47:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-13T22:46:45+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-13T22:46:45+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "context canceled"}
2024-05-13T22:46:45+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-13T22:46:45+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
2024-05-13T22:46:45+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-13T22:46:45+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2024-05-13T22:46:45+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:738\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-15T13:46:51+02:00       ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.126.92.94:7777: connect: no route to host", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.126.92.94:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-05-15T13:46:51+02:00       ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.159.134.91:7777: connect: no route to host", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.159.134.91:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-05-15T13:46:51+02:00       ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: connect: no route to host", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.150.199.48:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-05-15T13:46:51+02:00       ERROR   contact:service ping satellite failed   {"Process": "storagenode", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "attempts": 1, "error": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.94.153.46:7777: connect: no route to host", "errorVerbose": "ping satellite: rpc: tcp connector failed: rpc: dial tcp 34.94.153.46:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2024-05-16T22:32:06+02:00       ERROR   nodestats:cache Get stats query failed  {"Process": "storagenode", "error": "context canceled"}
2024-05-16T22:32:06+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-16T22:32:06+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "context canceled"}
2024-05-16T22:32:06+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-16T22:32:06+02:00       ERROR   collector       error during collecting pieces:         {"Process": "storagenode", "error": "pieces error: context canceled", "errorVerbose": "pieces error: context canceled\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:574\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:83\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:56\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:52\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-16T22:32:06+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
2024-05-16T22:32:06+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-16T22:32:06+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2024-05-16T22:32:06+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-06-09T10:59:04+02:00       ERROR   orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      rpc client when sending new orders settlements  {"Process": "storagenode", "error": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "request": {"limit":{"serial_number":"AAAAAADGM2NFKLMEBETN3M7IPA","satellite_id":"12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S","uplink_public_key":"GKWPCMPSIFSGBFOHOB33IV4L6MOAJVPAZH6W3AEJJABW52VG76PA","storage_node_id":"12vjmGaoza79G3BwA9KkvED8KbAY5q1geZvrNfWLnUj7JJSNYU1","piece_id":"RNYVTADYWZHBLD3WFIZ2UBOCUHBUDG2XLS7XA3IU2N42RUDUGJ2Q","limit":2319360,"action":1,"piece_expiration":"0001-01-01T00:00:00Z","order_expiration":"2024-06-10T06:16:53.732902055Z","order_creation":"2024-06-09T06:16:53.732902055Z","encrypted_metadata_key_id":"+Qu3n5/wGic=","encrypted_metadata":"Spu0mBoHsSNaFKIxfDAmXxQrKvqcYuGRfXdSTFqvYDlvPjsbUurdsQyINp328ZkSkUr/vWpCrWJpWw==","satellite_signature":"MEUCIQDCmXuHJo+v+uYvb6uqZnG81vzKes9U6XbBvBXC3cCqsAIgcGdWoAhE2YmVpaAoDqbb1H8BuGPfE1tRzeoAHeV5/is="},"order":{"serial_number":"AAAAAADGM2NFKLMEBETN3M7IPA","amount":262144,"uplink_signature":"mYNAvhYZ1I+710olHtIJTZPkJaRryJ3A9f+LkRg2rE9EOh7SfYDPPx/KEkffVugGfl1heXBVrZhBNGQ1S0s5Cw=="}}}
2024-06-09T10:59:04+02:00       ERROR   orders.12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S      failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:45744->34.150.199.48:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
Node 2 
2024-05-15T03:02:23+02:00       ERROR   orders.12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs      failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "order: failed to start settlement: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 127.0.0.1:53: read udp 127.0.0.1:57342->127.0.0.1:53: i/o timeout", "errorVerbose": "order: failed to start settlement: rpc: tcp connector failed: rpc: dial tcp: lookup eu1.storj.io on 127.0.0.1:53: read udp 127.0.0.1:57342->127.0.0.1:53: i/o timeout\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:272\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:227\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-15T03:02:23+02:00       ERROR   orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "order: failed to start settlement: rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 127.0.0.1:53: read udp 127.0.0.1:50538->127.0.0.1:53: i/o timeout", "errorVerbose": "order: failed to start settlement: rpc: tcp connector failed: rpc: dial tcp: lookup saltlake.tardigrade.io on 127.0.0.1:53: read udp 127.0.0.1:50538->127.0.0.1:53: i/o timeout\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:272\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:227\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-15T03:02:23+02:00       ERROR   orders.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6      failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "order: failed to start settlement: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 127.0.0.1:53: read udp 127.0.0.1:41375->127.0.0.1:53: i/o timeout", "errorVerbose": "order: failed to start settlement: rpc: tcp connector failed: rpc: dial tcp: lookup ap1.storj.io on 127.0.0.1:53: read udp 127.0.0.1:41375->127.0.0.1:53: i/o timeout\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:272\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:227\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-19T05:47:30+02:00       ERROR   pieces:trash    emptying trash failed   {"Process": "storagenode", "error": "pieces error: lazyfilewalker: signal: killed", "errorVerbose": "pieces error: lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkCleanupTrash:187\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:419\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2024-05-19T05:47:30+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-19T05:47:30+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "error": "context canceled"}
2024-05-19T05:47:30+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-19T05:47:30+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2024-05-19T05:47:30+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-19T05:47:30+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "context canceled"}
2024-05-19T05:47:30+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-30T19:47:57+02:00       ERROR   orders  failed to add bandwidth usage   {"Process": "storagenode", "satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "action": "GET_AUDIT", "amount": 6400, "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:76\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:249\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-30T19:48:07+02:00       ERROR   orders  failed to add bandwidth usage   {"Process": "storagenode", "satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "action": "GET", "amount": 15378688, "error": "bandwidthdb: database is locked", "errorVerbose": "bandwidthdb: database is locked\n\tstorj.io/storj/storagenode/storagenodedb.(*bandwidthDB).Add:76\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:249\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-06-10T01:50:04+02:00       ERROR   orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       rpc client when sending new orders settlements  {"Process": "storagenode", "error": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:59694->34.94.153.46:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:59694->34.94.153.46:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "request": {"limit":{"serial_number":"AAAAAADGM5Y6PWMZVGEJS42UIA","satellite_id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","uplink_public_key":"KQYNU3MJ32QZMMKPKHYOFN32BVJL5UACFUIX755EO76OJWZ7RSNQ","storage_node_id":"129hMUXjWfcnXEm4sjywAdM7YnUfS83wLsCVyuxDMCCWFy61GAa","piece_id":"N72B3IYIZTAOPX27NS4QO7RBROCRI3E3KTVVTI7VSJWV46ZRHL3Q","limit":4203520,"action":1,"piece_expiration":"2024-07-09T21:36:39.942874Z","order_expiration":"2024-06-10T21:36:39.95061302Z","order_creation":"2024-06-09T21:36:39.95061302Z","encrypted_metadata_key_id":"QDs91VL1vug=","encrypted_metadata":"mRQEHoOoXHwzsaBVouiG3zdrNFGX7GONN29N8XGNgj4tbA3rw42Y5XLUadGjTNM=","satellite_signature":"MEYCIQDMRLFGmHdOc7XBoSddjmTP1Fp5/XW2rKTTEF8F6eI6OgIhAKR+dt13qX+1Ic3yDlkRdjXQi7EVe5ZYRlKCq4UHgYlr"},"order":{"serial_number":"AAAAAADGM5Y6PWMZVGEJS42UIA","amount":262144,"uplink_signature":"L0llsVBQhHaz6JAOFnHHrJht9REQZEiwTe/hlO+F1DNdYvLHWV3TPT/USDXOGhjFep0a18G7+k+VLK+89ie+DQ=="}}}
2024-06-10T01:50:04+02:00       ERROR   orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:59694->34.94.153.46:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 10.66.66.2:59694->34.94.153.46:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
4
Node 3 
2024-05-17T03:32:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: signal: killed", "errorVerbose": "lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S"}
2024-05-17T03:32:03+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "error": "context canceled"}
2024-05-17T03:32:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs"}
2024-05-17T03:32:03+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "context canceled"}
2024-05-17T03:32:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6"}
2024-05-17T03:32:03+02:00       ERROR   collector       error during collecting pieces:         {"Process": "storagenode", "error": "pieces error: context canceled", "errorVerbose": "pieces error: context canceled\n\tstorj.io/storj/storagenode/pieces.(*Store).GetExpired:574\n\tstorj.io/storj/storagenode/collector.(*Service).Collect:83\n\tstorj.io/storj/storagenode/collector.(*Service).Run.func1:56\n\tstorj.io/common/sync2.(*Cycle).Run:99\n\tstorj.io/storj/storagenode/collector.(*Service).Run:52\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
2024-05-17T03:32:03+02:00       ERROR   lazyfilewalker.used-space-filewalker    failed to start subprocess      {"Process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "context canceled"}
2024-05-17T03:32:03+02:00       ERROR   pieces  failed to lazywalk space used by satellite      {"Process": "storagenode", "error": "lazyfilewalker: context canceled", "errorVerbose": "lazyfilewalker: context canceled\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:73\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkAndComputeSpaceUsedBySatellite:130\n\tstorj.io/storj/storagenode/pieces.(*Store).SpaceUsedTotalAndBySatellite:704\n\tstorj.io/storj/storagenode/pieces.(*CacheService).Run:58\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2.1:87\n\truntime/pprof.Do:51\n\tstorj.io/storj/private/lifecycle.(*Group).Run.func2:86\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2024-05-17T03:32:03+02:00       ERROR   pieces:trash    emptying trash failed   {"Process": "storagenode", "error": "pieces error: lazyfilewalker: signal: killed", "errorVerbose": "pieces error: lazyfilewalker: signal: killed\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*process).run:85\n\tstorj.io/storj/storagenode/pieces/lazyfilewalker.(*Supervisor).WalkCleanupTrash:187\n\tstorj.io/storj/storagenode/pieces.(*Store).EmptyTrash:419\n\tstorj.io/storj/storagenode/pieces.(*TrashChore).Run.func1.1:84\n\tstorj.io/common/sync2.(*Workplace).Start.func1:89"}
2024-06-03T23:32:22+02:00       ERROR   orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       rpc client when sending new orders settlements  {"Process": "storagenode", "error": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78", "request": {"limit":{"serial_number":"AAAAAADGL5SXAZ3URYDH7LSH2E","satellite_id":"1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE","uplink_public_key":"QGJATLLFZP57C7CGKX5RAPYTK45LKM2A3JDCXHPXNWDQXSZN4LIA","storage_node_id":"1CPsVCepUBZgo81W5vk5mhnoj6yA3E4qjMDAkjakXxEKq3u9WZ","piece_id":"XZZX7P4GH5HWSMBVDT3PYHBPIDOFO6GV7KTE2DGGNGGZJCFDL2WQ","limit":4203520,"action":1,"piece_expiration":"2024-07-03T19:05:20.52287Z","order_expiration":"2024-06-04T19:05:20.566908198Z","order_creation":"2024-06-03T19:05:20.566908198Z","encrypted_metadata_key_id":"QDs91VL1vug=","encrypted_metadata":"eS7pDVDgV27ZMUR4N1btQBimpHOyOTw1JcW81CvX4Gh3Dp5HxpGTP5mRaIMQB+U=","satellite_signature":"MEUCIG7pbo7zg1rSwPVUBRqFS7jyp/qhBsgAt4cyynOqtjyiAiEAha9sj+ORpfGWtGmSPgf2jKD+PneWDlBp54ZipBq1QIY="},"order":{"serial_number":"AAAAAADGL5SXAZ3URYDH7LSH2E","amount":262144,"uplink_signature":"qnCRh4WAVWcS9uxPZNiDyBgofgbRioinmF/fwOWzH19gwFBwHZjgrG44DY2nDBJCKPs62spAb5vOxdr6uE2KDg=="}}}
2024-06-03T23:32:22+02:00       ERROR   orders.1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE       failed to settle orders for satellite   {"Process": "storagenode", "satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "error": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer", "errorVerbose": "order: sending settlement agreements returned an error: write tcp 172.24.0.2:44764->34.94.153.46:7777: write: connection reset by peer\n\tstorj.io/storj/storagenode/orders.(*Service).settleWindow:299\n\tstorj.io/storj/storagenode/orders.(*Service).SendOrders.func2:231\n\tgolang.org/x/sync/errgroup.(*Group).Go.func1:78"}
5

seems your disk cannot keep-up. Perhaps you need to move databases to SSD.

Do you have FATAL errors in the log?
Because these errors look like the node is in the stop stage. When the node receives a kill signal, it stopping all processes, they would complain to the logs.

6

The db are on a ssd, on the hdd are only storj data. sda is the ssd
No fatal error the node has restarted after the update and the three nodes are now running for 400 - 580 hours
Since the tests are running I have this utilization, before it was about 20%. I followed the recommendation of a Node pro HDD and even without recommendation I moved all db to the ssd and still have this utilization.
the 3 hdds are ST18000NM000J-2TV103

grafik
grafik948×452 91.4 KB

7

Yes, you would have utilization if they are not full.
What’s filesystem on these disks?

8

The filesystem is ext4 on all disks