Hi
It’s something went wrong with ap1
satellite - I’m getting degradation in ONLINE
and no obvious reasons why.
I’m seeing this on an all 5 of my nodes under the same ISP.
Looks like that for now:
Stopped the docker, reload the server - not helped.
Current version - v1.136.4
Whats going on?
Please help
logs (with docker logs storagenodeX 2>&1 | grep "error" | tail
)
1st node
2025-09-05T22:08:28Z ERROR Error updating service. {"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "context canceled", "errorVerbose": "context canceled\n\tmain.downloadBinary:62\n\tmain.update:56\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.2:388\n\tstorj.io/common/process.cleanup.func1:406\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tmain.main:22\n\truntime.main:283"}
2nd node
2025-09-04T21:37:20Z ERROR Error updating service. {"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "context canceled", "errorVerbose": "context canceled\n\tmain.downloadBinary:62\n\tmain.update:56\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.2:388\n\tstorj.io/common/process.cleanup.func1:406\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tmain.main:22\n\truntime.main:283"}
3rd node
2025-09-13T16:19:33Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.5:28967->79.127.226.98:55886: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.5:28967->79.127.226.98:55886: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2025-09-13T16:31:37Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.5:28967->79.127.205.232:37118: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.5:28967->79.127.205.232:37118: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2025-09-13T18:15:40Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.5:28967->109.61.92.70:41302: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.5:28967->109.61.92.70:41302: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2025-09-14T07:45:02Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.5:28967->79.127.205.239:35518: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.5:28967->79.127.205.239:35518: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
4th node
2025-09-10T08:22:35Z ERROR Error updating service. {"Process": "storagenode-updater", "Service": "storagenode-updater", "error": "context canceled", "errorVerbose": "context canceled\n\tmain.downloadBinary:62\n\tmain.update:56\n\tmain.loopFunc:31\n\tstorj.io/common/sync2.(*Cycle).Run:163\n\tmain.cmdRun:139\n\tstorj.io/common/process.cleanup.func1.2:388\n\tstorj.io/common/process.cleanup.func1:406\n\tgithub.com/spf13/cobra.(*Command).execute:985\n\tgithub.com/spf13/cobra.(*Command).ExecuteC:1117\n\tgithub.com/spf13/cobra.(*Command).Execute:1041\n\tstorj.io/common/process.ExecWithCustomOptions:115\n\tstorj.io/common/process.ExecWithCustomConfigAndLogger:80\n\tmain.main:22\n\truntime.main:283"}
5th node
2025-09-13T05:03:21Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.7:28967->79.127.205.228:58902: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.7:28967->79.127.205.228:58902: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2025-09-13T10:08:00Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "AOBVSZZHIY5BQGFQJPE7VYPNNYA7G7E7PYYVPNZE5OE5OHR47GYA", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Action": "GET", "Offset": 0, "Size": 1024, "Remote Address": "172.105.174.159:35012", "error": "manager closed: read tcp 172.17.0.7:28967->172.105.174.159:35012: read: connection timed out", "errorVerbose": "manager closed: read tcp 172.17.0.7:28967->172.105.174.159:35012: read: connection timed out\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2025-09-13T10:17:28Z ERROR piecestore download failed {"Process": "storagenode", "Piece ID": "VQPIT3YKCBX3SXH74HDU2JER7IQJO2PAFSJQNMUHBRQZHO35CLQQ", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET_REPAIR", "Offset": 0, "Size": 2174464, "Remote Address": "45.137.206.42:3246", "error": "manager closed: read tcp 172.17.0.7:28967->45.137.206.42:3246: read: connection timed out", "errorVerbose": "manager closed: read tcp 172.17.0.7:28967->45.137.206.42:3246: read: connection timed out\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:235"}
2025-09-14T02:28:08Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.7:28967->79.127.226.98:45852: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.7:28967->79.127.226.98:45852: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2025-09-14T03:14:41Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.7:28967->109.61.92.76:45004: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.7:28967->109.61.92.76:45004: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}
2025-09-14T05:10:38Z ERROR piecestore upload internal error {"Process": "storagenode", "error": "manager closed: closed: read tcp 172.17.0.7:28967->79.127.205.236:44628: read: connection reset by peer", "errorVerbose": "manager closed: closed: read tcp 172.17.0.7:28967->79.127.205.236:44628: read: connection reset by peer\n\tgithub.com/jtolio/noiseconn.(*Conn).readMsg:211\n\tgithub.com/jtolio/noiseconn.(*Conn).Read:171\n\tstorj.io/drpc/drpcwire.(*Reader).read:68\n\tstorj.io/drpc/drpcwire.(*Reader).ReadPacketUsing:113\n\tstorj.io/drpc/drpcmanager.(*Manager).manageReader:230"}