Error on ingesting samples with different value but same timestamp

Node Operators troubleshooting
,
1

I have my node up and running and I’m using the built in /metrics endpoint. The issue I’m seeing is duplicate metrics. I’m scraping from my k8s cluster running the prometheus operator. I see the following

ts=2024-08-16T21:54:02.811Z caller=scrape.go:1754 level=warn component="scrape manager" scrape_pool=storagenode-metrics target=http://redacted:6001/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=119```

indeed when I curl the /metrics endpoint I do see duplicates

curl -s http://redacted:6001/metrics | sort | awk 'BEGIN { FS="[ ]" } ; { print $1 }'| grep -v "#" |uniq -D

results in the following (just a sample)

db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="pricing",field="OpenConnections"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="pricing",field="OpenConnections"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="reputation",field="MaxLifetimeClosed"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="reputation",field="MaxLifetimeClosed"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="secret",field="MaxOpenConnections"}
db_stats{scope="storj_io_storj_storagenode_storagenodedb",db_name="secret",field="MaxOpenConnections"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Avail"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Files"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Free"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="FreeFiles"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Total"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Total"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Used"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Used"}
disk{device="_dev_mapper_ubuntu__vg_ubuntu__lv",scope="github_com_jtolds_monkit_hw_v2",field="Used"}
disk{scope="github_com_jtolds_monkit_hw_v2",device="_dev_mapper_ubuntu__vg_ubuntu__lv",field="Total"}
disk{scope="github_com_jtolds_monkit_hw_v2",device="_dev_mapper_ubuntu__vg_ubuntu__lv",field="Total"}

Wanted to see is anyone else sees this before I submit a bug.

2

I don’t ever use the metric endpoint(don’t even know what it is…), but on my system, it seems to be a different port and the output looks very different from yours. Am I even doing this right? I tried it with that sort awk grep filter and it returned nothing, but without it I get:

curl -s http://localhost:5999/metrics
# TYPE function gauge
function{name="root",scope="storj_io_common_process",field="current"} 1
function{name="root",scope="storj_io_common_process",field="highwater"} 1
function{name="root",scope="storj_io_common_process",field="successes"} 0
function{name="root",scope="storj_io_common_process",field="errors"} 0
function{name="root",scope="storj_io_common_process",field="panics"} 0
function{name="root",scope="storj_io_common_process",field="failures"} 0
function{name="root",scope="storj_io_common_process",field="total"} 0
function{name="root",scope="storj_io_common_process",field="delta"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="current"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="highwater"} 1
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="successes"} 2
function{scope="storj_io_storj_private_version_checker",name="__Client__All",field="errors"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="panics"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="failures"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="total"} 2
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="delta"} 0
# TYPE function_times gauge
function_times{name="root",kind="success",scope="storj_io_common_process",field="count"} 0
function_times{name="root",kind="failure",scope="storj_io_common_process",field="count"} 0
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="count"} 2
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="sum"} 0.872506345
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="min"} 0.270716462
function_times{scope="storj_io_storj_private_version_checker",name="__Client__All",kind="success",field="max"} 0.601789883
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="rmin"} 0.270716448
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="ravg"} 0.436253184
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="r10"} 0.303823792
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="r50"} 0.436253168
function_times{scope="storj_io_storj_private_version_checker",name="__Client__All",kind="success",field="r90"} 0.568682544
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="r99"} 0.598479153
function_times{kind="success",scope="storj_io_storj_private_version_checker",name="__Client__All",field="rmax"} 0.601789888
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="recent"} 0.270716462
function_times{name="__Client__All",kind="failure",scope="storj_io_storj_private_version_checker",field="count"} 0
3

Thats the same endpoint, port numbers are just how you installed it. It’s a very long list and I only included part of it. Yours is definitly in the same format. If the command had no results then you had no duplicates. Thats good for you, bad for me. Thanks for the feedback

4

Hello @opswhisperer ,
Welcome to the forum!

I would confirm duplicates on my node too.

5
1 Like
6

I went ahead and filed a bug, please comment there if you have the same issue.

1 Like