Is there a way to speed up the graceful exit process?

Yes, you can tune some parameters of Graceful Exit, however it’s often results in failed GE if your network/hardware would be not able to keep up.
You should track the number of failed transfers in your log. Each transfer after the failure will be repeated 4 more times, then it will be considered unsuccessful. If the failure rate is greater than 10%, the node will be disqualified.