Hi I am running a pi4 node and 3 other pi4 nodes . On one is working with grafana, and prometheus. Since about 2 weeks Prometheus freezes my PI4 the memory I 100% (4gb) full without that I have done anything. How could I set prometheus to not use so much mem. Does anyone have a tip?
Hi.I dont want to bother the community with exact same problem.Exactly after release of version 1.72.2 prometheus started to give same behavior as yours.Maybe it is not related to relese.
In the same time the ingress started to rise into new highs, so probably also that caused the problem.The log exporter lately have millions of lines to read, and push that to prometheus.
There is a docker command to limit memory usage. --memory=“2048m”
But this resulting the grafana to not show all details in combined boom table
I was forced to config storagenodes to only do error logs.This way it stay operable, but ofc not shows some of data in grafana, like sucessrates, piece information
How did they configure it?
You should edit the storagenode config.yaml
And change the log level to log.level: error
Then delete the node.log file and restart the node.
Probably it is better to remove prometheus folder with previous data, and start from begining, bcs that big database will make you problems again if you load older times, when that massive data was used.Just leave your prometheus config file intact.
You may also configure a data retention period on Prometheus and it will remove an older data automatically
And how ? Could you help me further?