SO i have started graceful exit and the percentage in all satellites is 0 after about 2 hours.
Now i checked my logs and i see all these errors.
Is there anything i can do?
2023-07-08T12:02:27.893Z ERROR gracefulexit:chore.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:02:35.103Z ERROR gracefulexit:chore.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:02:35.422Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:02:37.533Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:02:56.793Z ERROR gracefulexit:chore.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:02:57.968Z ERROR gracefulexit:chore.12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB@europe-north-1.tardigrade.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "error": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:60168->35.228.31.57:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:06.038Z ERROR piecetransfer failed to put piece {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "Piece ID": "HJBYGX4ESJZZSIN5YD6MU6NZUHWCDHYSTKMCPL4CXXW56SUYZTTQ", "Storagenode ID": "1a2beRVJZDXs3GSQ8U2inz1kgEgrc3bFRPcyrAG8hNjqXHhqAB", "error": "ecclient: upload failed (node:1a2beRVJZDXs3GSQ8U2inz1kgEgrc3bFRPcyrAG8hNjqXHhqAB, address:86.52.176.37:28967): protocol: expected piece hash; context deadline exceeded; EOF", "errorVerbose": "ecclient: upload failed (node:1a2beRVJZDXs3GSQ8U2inz1kgEgrc3bFRPcyrAG8hNjqXHhqAB, address:86.52.176.37:28967): protocol: expected piece hash; context deadline exceeded; EOF\n\tstorj.io/uplink/private/ecclient.(*ecClient).PutPiece:244\n\tstorj.io/storj/storagenode/piecetransfer.(*service).TransferPiece:148\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:100\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:06.038Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:06.713Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:06.916Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:07.993Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:08.938Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:18.751Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:19.171Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:22.839Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
2023-07-08T12:03:31.662Z ERROR gracefulexit:chore.121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777 failed to send notification about piece transfer. {"process": "storagenode", "Satellite ID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "error": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer", "errorVerbose": "write tcp 172.17.0.3:42020->34.80.215.116:7777: write: connection reset by peer\n\tstorj.io/drpc/drpcstream.(*Stream).rawFlushLocked:401\n\tstorj.io/drpc/drpcstream.(*Stream).MsgSend:462\n\tstorj.io/common/pb.(*drpcSatelliteGracefulExit_ProcessClient).Send:83\n\tstorj.io/storj/storagenode/gracefulexit.(*Worker).Run.func3:101\n\tstorj.io/common/sync2.(*Limiter).Go.func1:49"}
Very strange though at the same time i have some logs showing that everything is ok and for some reason i am still downloading data…
2023-07-08T12:10:16.201Z INFO piecestore download started {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 145152, "Remote Address": "216.66.40.82:60876"}
2023-07-08T12:10:17.044Z INFO piecestore downloaded {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 0, "Size": 145152, "Remote Address": "216.66.40.82:60876"}
2023-07-08T12:10:19.118Z INFO piecestore download started {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 507136, "Size": 181504, "Remote Address": "216.66.40.82:24726"}
2023-07-08T12:10:19.749Z INFO piecestore downloaded {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 507136, "Size": 181504, "Remote Address": "216.66.40.82:24726"}
2023-07-08T12:10:20.935Z INFO piecestore download started {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 869632, "Size": 181248, "Remote Address": "216.66.40.82:23520"}
2023-07-08T12:10:21.533Z INFO piecestore downloaded {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 869632, "Size": 181248, "Remote Address": "216.66.40.82:23520"}
2023-07-08T12:10:22.607Z INFO piecestore download started {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 1231872, "Size": 181504, "Remote Address": "216.66.40.82:23520"}
2023-07-08T12:10:23.454Z INFO piecestore downloaded {"process": "storagenode", "Piece ID": "S7XVS53LMGA5ATOCYNRCFTOFGUWBSBKUIPPOANZF7QNQCAROYFXA", "Satellite ID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "Action": "GET", "Offset": 1231872, "Size": 181504, "Remote Address": "216.66.40.82:23520"}
And some normal transfers to other nodes as well:
2023-07-08T12:09:57.084Z INFO piecetransfer piece transferred to new storagenode {"process": "storagenode", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Piece ID": "LXI6JC4YFISTROGZOIE54JCCGAK527OVDBRRSML3S34LANHSPN2Q", "Storagenode ID": "1Fgu7ESXYYhkQ15ceCdmAbb7QWh8QM3rcNZw8Dfs84E9Jhai4Y"}