Graceful exit at 0% after > 48 hours

Over 2 days ago I started graceful exit on one of my nodes.

~$ docker exec -it storagenode2 /app/bin/storagenode exit-status --config-dir /app/config

Domain Name                  Node ID                                              Percent Complete  Successful  Completion Receipt
ap1.storj.io:7777            121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6  0.00%             N           N/A  
eu1.storj.io:7777            12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs  0.00%             N           N/A  
saltlake.tardigrade.io:7777  1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE   0.00%             N           N/A  

Is this normal that nothing happens after 2 days? Should there be something in the logs? I only have some log entries when there was some brief DNS problem.

~# grep "grace" /mnt/storagenode2/node/node.log
2025-01-17T19:48:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:48:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:48:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:49:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:49:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:49:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:50:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:50:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:50:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:51:40Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:51:40Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-17T19:51:40Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T20:55:35Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T20:55:35Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T20:55:35Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:04:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:04:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:04:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:05:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:05:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:05:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:06:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:06:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:06:30Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: connection refused\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:07:31Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:07:31Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}
2025-01-18T23:07:31Z    ERROR   gracefulexit:chore      worker failed   {"Process": "storagenode", "error": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host", "errorVerbose": "rpc: tcp connector failed: rpc: dial tcp 192.168.100.3:7777: connect: no route to host\n\tstorj.io/common/rpc.HybridConnector.DialContext.func1:190"}

Satellites should not be resolving to private addresses. Check your dns.

Yes, it’s normal after the GE has been changed:

The node should be online the next 30 days, then it would be either finished or the node would be disqualified if it would manage to have an online score lower than 80% during this time.

1 Like

TL;DR “The “Percent Complete” field is a relic of the old graceful exit and is no longer meaningful. It will likely be removed at some point.”

I know and I found the cause. Whenever there was a DNS problem, it added the “search domain” to the domain name it tried to resolve and there was a wildcard entry for that zone configured, which resolved to that host in my DMZ.

1 Like