After each watchtower update of the storagenodes, my prometheus exporters somehow connect to random other nodes. I always need to restart all exporters to fix this. My Grafana dashboard is also showing a weird mix of information when this happens, e.g. it is showing the wrong node name for the metrics…
Yes, weird. Because all nodes are identified by their instance and I could not see how it can display wrong data.
Here is how a graph of mine looks like. We see up and downs due to wrong disk usage numbers because it doesn’t show the correct data from the correct nodes:
Do you use the --link for each exporters in docker?