Satellite specific usage information post v1.122.1

[ItsHass]

Hi all,

I wonder if anyone has found any API’s that allow to find the values being held in storage by each satellite as it used to be in the API ?

[Alexey]

Storage node dashboard API (v0.19.0)

With v0.19.0 you can now query the storage node dashboard API. By default it is running on port 14002. You have to add -p 127.0.0.1:14002:14002 to the docker command. From outside the container you should now be able to call the following API endpoints: curl -s 127.0.0.1:14002/api/dashboard | jq .data.satellites [ "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "118UWpMCHzs6CvSgWd9BfFVjw5K9pZbJjkfZJexMtSkmKxvvAW", "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "12L9ZFwhzV…

Script for Audits stat by satellites

Updated for Docker version (bash) for sat in `docker exec -i storagenode wget -qO - localhost:14002/api/sno | jq .satellites[].id -r`; do docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/$sat | jq .id,.audits; done Docker version (Powershell) (docker exec -i storagenode wget -qO - localhost:14002/api/sno | ConvertFrom-Json).satellites.id | %{"$_"; (docker exec -i storagenode wget -qO - localhost:14002/api/sno/satellite/$_ | ConvertFrom-Json).audits} Dashboard is port m…

example:

# wget -O - http://localhost:14002/api/sno/satellite/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs | jq '.storageSummary'
--2025-02-20 02:08:40--  http://localhost:14002/api/sno/satellite/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:14002... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: 'STDOUT'

-                                 [ <=>                                              ]  10.71K  --.-KB/s    in 0s

2025-02-20 02:08:40 (318 MB/s) - written to stdout [10969]

948294511753518.6

You may see other info

wget -O - http://localhost:14002/api/sno/satellite/12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs | jq

To get a list of satellites:

wget -O - http://localhost:14002/api/sno | jq '.satellites[].id' -r

[ItsHass]

I’m looking for this value :

currentStorageUsed

[elek]

Looks to be removed by this patch: https://review.dev.storj.io/c/storj/storj/+/16040

Commit message explains the challenges, why it is removed.

Looks like piecestore (and also hashstore) share the information via the prometheus compatible metrics endpoint:

http://localhost:30009/metrics (30009 is my debug port)

blobs_usage{satellite="_12whfK1EDvHJtajBiAUeajQLYcWqxcQmdYQU5zX5cCf6bAxfgu4",scope="storj_io_storj_storagenode_pieces",field="total_size"} 5.791104e+07
blobs_usage{satellite="_12whfK1EDvHJtajBiAUeajQLYcWqxcQmdYQU5zX5cCf6bAxfgu4",scope="storj_io_storj_storagenode_pieces",field="content_size"} 5.788288e+07

[ItsHass]

Where is the debug port located by default?

[elek]

Random port, I think:

$ storagenode --help --defaults=release| grep debug                                                                                                                      0 [11:10:09]
      --debug.addr string                 address to listen on for debug endpoints (default "127.0.0.1:0")

It’s better to add explicit configuration.

I have this one:

debug.addr: localhost:30009

[Alexey]

You may find it by netstat -l

[ItsHass]

hmm

$ curl localhost:34267
404: Page Not Found

Think i found it … - but i can’t seem to get data out of it

[elek]

metrics are under /metrics

Normal root context should return with a list of available endpoints.

I am not sure if you found the debug port…

[ItsHass]

Sorry, - i mis-understood i think… i thought it was publishing a port by default locally - however doesnt seem to be the case then

[ItsHass]

Got it ! obviously forgot also to open the port from docker

… I shall have a play

[ItsHass]

@elek

@elek

blobs_usage{satellite="_12whfK1EDvHJtajBiAUeajQLYcWqxcQmdYQU5zX5cCf6bAxfgu4",scope="storj_io_storj_storagenode_pieces",field="total_size"} 5.791104e+07
blobs_usage{satellite="_12whfK1EDvHJtajBiAUeajQLYcWqxcQmdYQU5zX5cCf6bAxfgu4",scope="storj_io_storj_storagenode_pieces",field="content_size"} 5.788288e+07

I dont seem to find this

[zip]

There is no blobs_usage in recent versions (at least with dedicated disk and no filewalkers on), but there is satellite_usage, which shows what sat thinks you have.
The reports are however hit and miss lately.

# TYPE satellite_usage gauge
satellite_usage{satellite="_1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",scope="storj_io_storj_storagenode_nodestats",field="used_bytes"} 2.3688123136e+10
satellite_usage{satellite="_121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",scope="storj_io_storj_storagenode_nodestats",field="used_bytes"} -0
satellite_usage{scope="storj_io_storj_storagenode_nodestats",satellite="_12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",field="used_bytes"} 6.455309131776e+12
satellite_usage{satellite="_12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",scope="storj_io_storj_storagenode_nodestats",field="used_bytes"} 4.0745021765128145e+12

[ItsHass]

is that on the /metrics - endpoint?

I just have this under that endpoint

# TYPE function gauge
function{name="root",scope="storj_io_common_process",field="current"} 1
function{name="root",scope="storj_io_common_process",field="highwater"} 1
function{name="root",scope="storj_io_common_process",field="successes"} 0
function{name="root",scope="storj_io_common_process",field="errors"} 0
function{name="root",scope="storj_io_common_process",field="panics"} 0
function{name="root",scope="storj_io_common_process",field="failures"} 0
function{name="root",scope="storj_io_common_process",field="total"} 0
function{name="root",scope="storj_io_common_process",field="delta"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="current"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="highwater"} 1
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="successes"} 5
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="errors"} 0
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="panics"} 0
function{scope="storj_io_storj_private_version_checker",name="__Client__All",field="failures"} 0
function{scope="storj_io_storj_private_version_checker",name="__Client__All",field="total"} 5
function{name="__Client__All",scope="storj_io_storj_private_version_checker",field="delta"} 4
# TYPE function_times gauge
function_times{name="root",kind="success",scope="storj_io_common_process",field="count"} 0
function_times{name="root",kind="failure",scope="storj_io_common_process",field="count"} 0
function_times{scope="storj_io_storj_private_version_checker",name="__Client__All",kind="success",field="count"} 5
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="sum"} 2.129042113
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="min"} 0.380473731
function_times{scope="storj_io_storj_private_version_checker",name="__Client__All",kind="success",field="max"} 0.513622254
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="rmin"} 0.380473728
function_times{scope="storj_io_storj_private_version_checker",name="__Client__All",kind="success",field="ravg"} 0.425808448
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="r10"} 0.382409497
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="r50"} 0.42080992
function_times{kind="success",name="__Client__All",scope="storj_io_storj_private_version_checker",field="r90"} 0.479702572
function_times{scope="storj_io_storj_private_version_checker",name="__Client__All",kind="success",field="r99"} 0.510230273
function_times{name="__Client__All",kind="success",scope="storj_io_storj_private_version_checker",field="rmax"} 0.51362224
function_times{scope="storj_io_storj_private_version_checker",name="__Client__All",kind="success",field="recent"} 0.513622254
function_times{name="__Client__All",kind="failure",scope="storj_io_storj_private_version_checker",field="count"} 0

[zip]

Correct, you can load it in web browser and see what is in there. Here for example it is on IP 192.168.10.12 and port 9651, so you can just do http://192.168.10.12:9651/metrics.

[zip]

Here it is over 7000 lines long, yours looks like incomplete.
Try restarting the node. Also after restart it takes a few minutes to show all the metrics.

[ItsHass]

ooooo

Ok, I will try it on another node

[nerdatwork]

@zip

There is no blobs_usage in recent versions

Under 1.122.1

[ItsHass]

I can see this

# TYPE blobs_usage gauge
blobs_usage{satellite="_1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",scope="storj_io_storj_storagenode_pieces",field="total_size"} 1.4573824e+07
blobs_usage{satellite="_1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",scope="storj_io_storj_storagenode_pieces",field="content_size"} 1.4565632e+07
blobs_usage{satellite="_12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",scope="storj_io_storj_storagenode_pieces",field="total_size"} 1.35677974528e+11
blobs_usage{satellite="_12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs",scope="storj_io_storj_storagenode_pieces",field="content_size"} 1.35480663552e+11
blobs_usage{satellite="_121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",scope="storj_io_storj_storagenode_pieces",field="total_size"} 3.4527639296e+10
blobs_usage{scope="storj_io_storj_storagenode_pieces",satellite="_121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6",field="content_size"} 3.4381476608e+10
blobs_usage{satellite="_12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",scope="storj_io_storj_storagenode_pieces",field="total_size"} 2.9018338202e+11
blobs_usage{satellite="_12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S",scope="storj_io_storj_storagenode_pieces",field="content_size"} 2.89510744068e+11

However… - What does the value actually mean?

And this also doesn’t look like JSON so what coding/format would this be , just so i can attempt to collect data from it?

Cheers

[nerdatwork]

@ItsHass

What does the value actually mean?

@ItsHass

1.4573824e+07

is scientific notation. The e+07 part means you need to move the decimal 7 places to the right. So the number becomes 14573824. These are bytes of data.

@ItsHass

what coding/format would this be

@elek

share the information via the prometheus compatible metrics

                                                            

Prometheus Storj-Exporter

I’m mostly using Prometheus to monitor stuff and was in need for storj exporter to monitor things like audit counts and uptimes per satellite, data as reported by storagenode etc… So I made one: [image] Checkout the README to setup Storj-Exporter Update exporter to latest version: Use watchtower or update manually using the following commands: docker pull anclrii/storj-exporter:latest ; \ docker stop storj-exporter ; \ docker rm storj-export…