I have my node up and running and I’m using the built in /metrics
endpoint. The issue I’m seeing is duplicate metrics. I’m scraping from my k8s cluster running the prometheus operator. I see the following
ts=2024-08-16T21:54:02.811Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=storagenode-metrics target=http://redacted:6001/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=119```
indeed when I curl the /metrics endpoint I do see duplicates
curl -s http://redacted:6001/metrics | sort | awk 'BEGIN { FS="[ ]" } ; { print $1 }'| grep -v "#" |uniq -D
results in the following (just a sample)
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="pricing",field="OpenConnections"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="pricing",field="OpenConnections"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="reputation",field="MaxLifetimeClosed"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="reputation",field="MaxLifetimeClosed"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="secret",field="MaxOpenConnections"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="secret",field="MaxOpenConnections"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Total"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Total"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Used"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Used"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Used"}
disk{scope="github_com_jtolds_monkit_hw_v2",device="_dev_mapper_ubuntu__vg_ubuntu__lv",field="Total"}
disk{scope="github_com_jtolds_monkit_hw_v2",device="_dev_mapper_ubuntu__vg_ubuntu__lv",field="Total"}
Wanted to see is anyone else sees this before I submit a bug.