storage controllers have long iowait’s so they will stall many operations, in most cases it’s at worst the blink of an eye, but if the disk is in the wrong spectrum of trouble then it can throw these multiple times a minute.
the disk doesn’t have to produce data errors to do so, and it’s most visible on live tasks, like when using the console, live data routing like VoIP or live streaming.
the performance loss is usually minimal, but it is annoying as hell and can be very difficult to get rid of without the right tools.
if you are a linux user i would recommend something like netdata, which will usually be able to show you disk latencies / backlog.
just keep in mind the backlog is a approximate length of the entire backlog and not the actual latency.
i seem to always have a drive that is acting up, if i get rid of one then another less visible one seems to rear its head, it’s a really annoying issue that i haven’t found a good solution for, and which i have seen in almost all storage solutions.
this i think is down to that if a process is running and losses its connection to the storage media, it will break stuff, or make inconsistencies, and thus to mitigate this issue, storage systems basically just halt’s everything on loss of a connection in hope that it can be restored in reasonable time.
hdd’s will usually make a tick sound when having these kinds of issues, and it will create high backlog spikes in netdata.
often these issues also seem to be exaggerated by higher workloads and similarly disk errors are also much more likely to come up when heavily loaded, so it can be a good idea for storage systems to run lightly loaded, as to not over work devices.
ideally i like to split them up in two classes, there are my lighter workload disks which can have issues and i use to store less important stuff, often in mirrors.
and then the near flawless performing drives which runs everything else.
ideally one might want a second host for the poorly performing drives to avoid their latency, ofc if not heavily loaded this is much less of an issue and drive can often run under light loads for years without issues, even if they continually fail under heavy loads.
ofc you can’t trust bad drives, which is why i usually end up throwing them in mirrors.