You need to check your logs for errors, related to a filewalkers (search for walk|retain|trash and failed|error|ERROR, also for FATAL errors).
If you have any - you likely need to figure out, why your disk is so slow.
As a last resort you may disable a lazy mode: