As the majority of people here I’m using watchtower to manage updates and subsequent container restarts. I have a RAID 6 array with multiple disks and 3 nodes, and have noticed (as many others) the filewalker task has considerable impact on the disks if used across all nodes at the same time. One solution would be to sequentially update and restart each storage node when the filewalker task finished or after some time.
We almost have something that can be used to accomplished this in watchtower: rolling restart and lifecycle hooks.
From watchtower documentation:
Rolling restart
Restart one image at time instead of stopping and starting all at once. Useful in conjunction with lifecycle hooks to implement zero-downtime deploy.
Lifecycle hooks
Executing commands before and after updating
It is possible to execute pre/post-check and pre/post-update commands inside every container updated by watchtower.
- The pre-check command is executed for each container prior to every update cycle.
- The pre-update command is executed before stopping the container when an update is about to start.
- The post-update command is executed after restarting the updated container
- The post-check command is executed for each container for the post every update cycle.
(…)
Timeouts
The timeout for all lifecycle commands is 60 seconds. After that, a timeout will occur, forcing Watchtower to continue the update loop.
With the above we could have a script that would get a time delay variable from the environment or periodically check if the filewalker task has finished. I have created the following ticket in watchtower repo: Rolling restart delay · Issue #675 · containrrr/watchtower · GitHub.
If the above gets approved we could have a script in storagenode containner that would periodically check if the filewalker task had finished, if so it would finish the post-update script with exit 0
allowing watchtower to update the next container.
Would like to hear your suggestions. Thanks in advance !
EDIT: English is not my first language, will gladly fix any grammar error/typo. Let me know if that’s the case.