xfs is much slower unfortunately, it’s kind of a known issue.
used-space was run (and completed) last week when I updated the node.
I think it’s a much deeper issue. Looks like a race condition to me. Used space comes along and counts the file size of a piece, adding it to the database. GC runs and moves that size from used to trash in database (along with the piece on disk). Trash-cleanup then removes trash size from the database and the piece.
The thing with lazy filewalkers is that they can only run when there are available IO for them. If used counts the file, gc trashes it immediately while trash-clean tries to update the database at the same time, I don’t see how they can run in a logical linear scenario.
I think we should move to my suggestion of staggered filewalkers:
- You restart the node. All filewalker dot-files (more on this down) are removed. A random timer between 2-10 minutes starts counting (random each time).
- Trash-cleanup starts and creates a .trash-cleanup-running file. Other filewalkers should check for this file and wait until it is gone. A notice is displayed on the node’s dashboard that “Filewalkers are running, space may not be accurate” (the notice can check for this dot-files as well). There is no point in running anything else other than trash-cleanup because the priority is to free up space so that the node can get incoming data, and there is no reason for used-space to count something that is going to be deleted anyway (resources are saved).
- Trash cleanup finishes, and removes its .trash-cleanup-running file.
- GC starts working on saved blooms, since the .trash-cleanup-running file isn’t there. It creates its own .gc-running file (see above for dashboard notice). Trash can’t run if .gc-running file is there because there is no point in going over files that didn’t get expired yet (expired=time to delete from trash, not TTL). Used-space can’t run as well.
- When GC is done (and every file is where it’s supposed to be), the .gc-running file is removed.
- Used-space starts and creates its own .used-space-running file. Trash-cleanup and GC are now blocked until used-space finishes. Again, see above for dashboard notice.
- Used-space finishes and removes its .used-space-running file. There is no notice on the dashboard anymore.
Notes: if the node is restarted mid FW processing, then the dot-files are useless. Trash still needs to run first, if there are any blooms saved, process them, then used-space (if enabled). This is the reason why the node needs to clean any related dot-files when restarting.
I’m not sure if I’m missing any other filewalker, but they can be wiggled in a logical linear check as well.
4 Likes
No need to use files for that, an in-memory data structure internal to the main storage node process would do.
3 Likes
Yes, I shared your suggestion with the team when you posted it in a first time here:
I just finished moving this node from xfs to ext4. It has 1.75TB data and now only 37GB trash. Trash might have been bigger when I started the move. I did 2 used-space filewalkers before the last rsync on xfs. It took ~23h each time. After the move finished the used space filewalker took only 21 minutes on ext4.
Ok, it could be that the hard disks are different in speed, but I don’t think it would account for a 60 times faster filewalker.
The copy process likely aligned your metadata in a way it’s now fast to iterate. This kind of improvement is exactly what I’d expect. As such, it will get worse with time, as files are removed and new uploaded.