Disqualified on Stefan satellite during Graceful Exit

[Alexey]

The update on the problem so far:

@littleskunk

Graceful exit disqualification
Most of the storage nodes can finish graceful exit just fine but there are a few edge cases and some storage nodes are getting disqualified for bugs. Here are the bugs that we currently know (order by likelyhood):

1. The storage node has problems with corrupted pieces. The storage node should identify the corruption, report it back to the satellite and continue. As long as the failure rate is low graceful exit should still be succesful. For some reason the storage node identifies the corrupted piece but something is messing up the entire batch. The storage node doesn’t continue as expected. We are working on a fix.
2. Graceful exit is transferring the pieces in a specific order. It starts with pieces that are close to the repair threshold. These are most likely older pieces and they have a higher likelyhood getting corrupted over time. This means a storage node with a overall failure rate of 10% will get most of these failures at the beginning of graceful exit. The satellite is judging after each batch. The storage node might get disqualifed early and has no chance to show the low failure rate at the end of graceful exit. We are working on a fix.
3. After each successful batch the storage node reports the results back to the satellite. We have a connection timeout in place to prevent storage nodes from getting stuck but no retry. This means the results are getting lost and a retry will be triggered. Fix is incoming.

Conclusion:
Lets say there is a 50% chance that you might get disqualified when you execute graceful exit. Would you leave the network anyway or would you stay and wait for a fix? If you would leave anyway you can risk it. The overall success rate is high and most likely it will work. If you see any kind of audit errors in your logs (corrupted or missing pieces) I would recommend to wait for the fix. My expectation is that the combination of issue 1 and 2 is what is causing problems at the moment.