Node API missing bandwithDaily

I am running https://github.com/anclrii/Storj-Exporter/ to export API data to prometheus and noticed that one node fails because one satelite doesn’t return the bandwithDaily although it does return those values for my 2 other nodes. See sat 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6 at the bottom, the others are just for reference.

Sat: 12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S
Data: {‘id’: ‘12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S’, ‘storageDaily’: [{‘atRestTotal’: 3510643742.6813974, ‘intervalStart’: ‘2019-10-01T00:00:00Z’}, {‘atRestTotal’: 3516282592.1555266, ‘intervalStart’: ‘2019-10-02T00:00:00Z’}, {‘atRestTotal’: 3471019717.4029627, ‘intervalStart’: ‘2019-10-03T00:00:00Z’}, {‘atRestTotal’: 3555821222.981349, ‘intervalStart’: ‘2019-10-04T00:00:00Z’}, {‘atRestTotal’: 2222392099.332451, ‘intervalStart’: ‘2019-10-05T00:00:00Z’}], ‘bandwidthDaily’: [{‘egress’: {‘repair’: 0, ‘audit’: 35072, ‘usage’: 0}, ‘ingress’: {‘repair’: 0, ‘usage’: 60416}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-01T00:00:00Z’}, {‘egress’: {‘repair’: 0, ‘audit’: 34048, ‘usage’: 1882624}, ‘ingress’: {‘repair’: 0, ‘usage’: 7303424}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-02T00:00:00Z’}, {‘egress’: {‘repair’: 0, ‘audit’: 30208, ‘usage’: 0}, ‘ingress’: {‘repair’: 0, ‘usage’: 0}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-03T00:00:00Z’}, {‘egress’: {‘repair’: 0, ‘audit’: 34816, ‘usage’: 0}, ‘ingress’: {‘repair’: 0, ‘usage’: 0}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-04T00:00:00Z’}, {‘egress’: {‘repair’: 0, ‘audit’: 26368, ‘usage’: 0}, ‘ingress’: {‘repair’: 0, ‘usage’: 0}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-05T00:00:00Z’}], ‘storageSummary’: 16276159374.553688, ‘bandwidthSummary’: 9406976, ‘audit’: {‘totalCount’: 6925, ‘successCount’: 6925, ‘alpha’: 19.99999999999995, ‘beta’: 0, ‘score’: 1}, ‘uptime’: {‘totalCount’: 26157, ‘successCount’: 25737, ‘alpha’: 99.9999999999992, ‘beta’: 1.0401162164177334e-23, ‘score’: 1}}

Sat: 118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW
Data: {‘id’: ‘118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW’, ‘storageDaily’: [{‘atRestTotal’: 16423270691746.77, ‘intervalStart’: ‘2019-10-01T00:00:00Z’}, {‘atRestTotal’: 17234235726123.418, ‘intervalStart’: ‘2019-10-02T00:00:00Z’}, {‘atRestTotal’: 16711585118726.262, ‘intervalStart’: ‘2019-10-03T00:00:00Z’}, {‘atRestTotal’: 1640730812402.883, ‘intervalStart’: ‘2019-10-04T22:00:00Z’}], ‘bandwidthDaily’: [{‘egress’: {‘repair’: 0, ‘audit’: 53248, ‘usage’: 4641195520}, ‘ingress’: {‘repair’: 0, ‘usage’: 967790592}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-01T00:00:00Z’}, {‘egress’: {‘repair’: 0, ‘audit’: 40960, ‘usage’: 8734249984}, ‘ingress’: {‘repair’: 0, ‘usage’: 1366952192}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-02T00:00:00Z’}, {‘egress’: {‘repair’: 0, ‘audit’: 26112, ‘usage’: 9193306624}, ‘ingress’: {‘repair’: 0, ‘usage’: 1479552768}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-03T00:00:00Z’}, {‘egress’: {‘repair’: 0, ‘audit’: 27648, ‘usage’: 8597012992}, ‘ingress’: {‘repair’: 0, ‘usage’: 1338600192}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-04T00:00:00Z’}, {‘egress’: {‘repair’: 2317056, ‘audit’: 23296, ‘usage’: 4366911488}, ‘ingress’: {‘repair’: 0, ‘usage’: 574359040}, ‘delete’: 0, ‘intervalStart’: ‘2019-10-05T00:00:00Z’}], ‘storageSummary’: 52009822348999.336, ‘bandwidthSummary’: 41262419712, ‘audit’: {‘totalCount’: 6338, ‘successCount’: 6329, ‘alpha’: 19.999990014763284, ‘beta’: 9.985236700172143e-06, ‘score’: 0.9999995007381649}, ‘uptime’: {‘totalCount’: 26153, ‘successCount’: 25777, ‘alpha’: 99.9999999999992, ‘beta’: 5.413740671932986e-15, ‘score’: 1}}

Sat: 121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
Data: {‘id’: ‘121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6’, ‘storageDaily’: [{‘atRestTotal’: 0, ‘intervalStart’: ‘2019-10-03T00:00:00Z’}], ‘bandwidthDaily’: None, ‘storageSummary’: 0, ‘bandwidthSummary’: 0, ‘audit’: {‘totalCount’: 0, ‘successCount’: 0, ‘alpha’: 1, ‘beta’: 0, ‘score’: 1}, ‘uptime’: {‘totalCount’: 12374, ‘successCount’: 12118, ‘alpha’: 99.9999981972427, ‘beta’: 1.8027572139194712e-06, ‘score’: 0.9999999819724279}}

This satellite doesn’t used your node yet, so, there is no usage of bandwidth or space and you do not have audits too.
It’s pretty normal. All satellites are independent.

This node has been operating for over a month and it worked hours ago. Even my more recent node works on all satellites.

However it is still possible that no clients for your node from that satellite.

Then it should still return the missing values as 0 and not have them omitted completely.
So this is what is expected:
‘bandwidthDaily’: [{‘egress’: {‘repair’: 0, ‘audit’: 0, ‘usage’: 0}, …
instead of omitting it.

Also how can it be that this only fails today but not the last 4 days in October? The satelite could not just have reset the stored data about my node in the in the middle of the month.

This is the output of localhost:14002/api/satellite/121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6
{“data”:{“id”:“121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6”,“storageDaily”:[{“atRestTotal”:0,“intervalStart”:“2019-10-03T00:00:00Z”}],“bandwidthDaily”:null,“storageSummary”:0,“bandwidthSummary”:0,“audit”:{“totalCount”:0,“successCount”:0,“alpha”:1,“beta”:0,“score”:1},“uptime”:{“totalCount”:12374,“successCount”:12118,“alpha”:99.9999981972427,“beta”:0.0000018027572139194712,“score”:0.9999999819724279}},“error”:""}

There should at least be one entry in bandwithDaily with intervalStart “2019-10-01T00:00:00Z” even if there was no bandwithUsed.

I just want to make sure this output is expected and not a bug. Because it definitely breaks the expected behavior when looking at it and exporting to prometheus-

Hi kevink,
We’re looking into your issue. Thank you for your patience

Would you mind telling us the node ID that is having the problem?

Thank you. The nodeid is 1cYRm7virx4Z7xXvgG9SxMBfJqkEA2XDoy6bWXfxkf2vLsmWjV

Hi @kevink - I haven’t seen this with my node yet, when you say node fails do you mean the whole exporter fails or just this metric? I can probably add a workaround to expose 0 for related metrics if api returns None (I will need to add some validation and capture exceptions anyway)

BTW I released v0.2.0 yesterday and added some missing metrics, check it out. I updated this post with details here https://forum.storj.io/t/prometheus-storj-exporter/ but spamfilter flagged it for review :thinking: hopefully will show up shortly.

The whole exporter fails with this exception:

Traceback (most recent call last):

File “./storj-exporter.py”, line 127, in

REGISTRY.register(StorjCollector())

File “/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py”, line 24, in register

names = self._get_names(collector)

File “/usr/local/lib/python3.7/site-packages/prometheus_client/registry.py”, line 64, in _get_names

for metric in desc_func():

File “./storj-exporter.py”, line 106, in collect

self.add_iterable_day_sum_metrics(['repair','audit','usage'], self.sat_data[sat]['bandwidthDaily'], "egress", storj_sat_month_egress, [sat])

File “./storj-exporter.py”, line 49, in add_iterable_day_sum_metrics

for day in data:

TypeError: ‘NoneType’ object is not iterable

I am using your latest version without realizing you did an update :smiley: Thanks. I use the docker container so watchtower updated them automatically.

kevink,
Thanks for your patience. It appears to me that a NULL field may in fact be normal. I’m testing this myself with a test network and on a node with no data uploaded to it, I see
{"data":{"id":"12dLVjPzCdLjWCFkXm7HcUY7i8eZoT8E7giqnEwweMzsidYTyCN","storageDaily":null,"bandwidthDaily":null,"storageSummary":0,"bandwidthSummary":0,"audit":{"totalCount":0,"successCount":0,"alpha":1,"beta":0,"score":1},"uptime":{"totalCount":3,"successCount":3,"alpha":4.910698,"beta":0,"score":1}},"error":""}
However, I’m still not sure why your storageDaily would not be NULL while bandwidthDaily is. I’m still poking around at it, but wanted to get back to you.

Thanks for your response. Didn’t expect one on Sunday :slight_smile:

We will have to account for that in the prometheus exporter if that is the normal behaviour.

1 Like

@kevink I’ve added a fix for Null valies in api in v0.2.3, let me know if you still see this issue:

1 Like

Answered this question in your other thread :smiley:
Thanks for the fix. Won’t be able to confirm since that satelite has now data for my node.