Some decadency in graphs

Node Operators
39

I’m running 5 nodes since about a month. Every time I see the dashboard, I happen to think: that’s a big difference between ingress and increase in storage. So, in the end I wrote a script in order to confirm/refute my suspicion and I’m hoping you can help me with your thoughts and insights.

So this is my dashboard:

Storj-efficiency
Storj-efficiency1317×1427 102 KB

Essentially, the ingress is about 34-48GB a day. However, the increase in storage is about 16GB per day (about a one third to a half of the ingress).

So I wrote this jq-script in bash (clean version at: echo "$( wget -qO - localhost:14002/api/sno | jq -r .satellites[].id | while rea - Pastebin.com ):
echo “$( wget -qO - localhost:14002/api/sno | jq -r .satellites.id | while read -r sNode; do echo ‘,’; wget -qO - localhost:14002/api/sno/satellite/$sNode; done )]” | sed -z ‘s/^,/[/’ |

jq -r ’
"Satellite per day: “,
(
. |
(
" - " + .id + " (” + .audits.satelliteName + “)”,
.bandwidthDaily as $bw |
(.storageDaily | map(select(.intervalStart == $bw.intervalStart)) | .[0].atRestTotalBytes) as $space |
(.storageDaily | map(select(.intervalStart > $bw.intervalStart)) | .[0]?.atRestTotalBytes) as $spaceTomorrow |
(($spaceTomorrow // $space) - $space) as $spaceInc |
($bw.ingress | .repair + .usage) as $ing |
(
" * Date: " + $bw.intervalStart,
" # IN: " + ($ing / 100000 | round | . / 10 | tostring) + “MB”,
" # Increase used space: " + ($spaceInc / 100000 | round | . / 10 | tostring) + “MB”,
" # Efficiency: " + ($spaceInc / $ing * 100 | round | tostring) + “%”
)
)
),
“”,
"Per day: ",
(
[..bandwidthDaily] as $bw |
[..storageDaily] as $stor |
(
$bw | group_by(.intervalStart) | (
.[0].intervalStart as $today |
([$stor.intervalStart] | map(select(. > $today)) | min) as $tomorrow |
($stor | map(select(.intervalStart == $today).atRestTotalBytes) | add) as $space |
($stor | map(select(.intervalStart == $tomorrow).atRestTotalBytes) | add) as $spaceTomorrow |
( [..ingress | (.repair + .usage)] | add ) as $ing |
(($spaceTomorrow // $space) - $space) as $spaceInc |
" - Date: " + $today,
" * IN: " + ($ing / 100000 | round | . / 10 | tostring) + “MB”,
" * Increase used space: " + ($spaceInc / 100000 | round | . / 10 | tostring) + “MB”,
" * Efficiency: " + ($spaceInc / $ing * 100 | round | tostring) + “%”
)
),
“”,
"Total: ",
(
( [$bw.ingress | (.repair + .usage)] | add ) as $ing |
( [$stor.intervalStart] | max) as $maxdate |
( [$stor.intervalStart] | min) as $mindate |
(($stor | map(select(.intervalStart == $maxdate).atRestTotalBytes) | add) - ($stor | map(select(.intervalStart == $mindate).atRestTotalBytes) | add)) as $space |
" - IN: " + ($ing / 100000 | round | . / 10 | tostring) + “MB”,
" - Total used space: " + ($space / 100000 | round | . / 10 | tostring) + “MB”,
" - Efficiency: " + ($space / $ing * 100 | round | tostring) + “%”
)
)’

Showing this output (full version: Showing this output:> Satellite per day:> - 12tRQrMTWUWwzwGh18i7Fqs67kmdhH9 - Pastebin.com ; because whole layout was distorted…)

Per day:

  • Date: 2023-05-01T00:00:00Z
  • IN: 42113.5MB
  • Increase used space: 21750.4MB
  • Efficiency: 52%
  • Date: 2023-05-02T00:00:00Z
  • IN: 39941MB
  • Increase used space: 4858.4MB
  • Efficiency: 12%
  • Date: 2023-05-03T00:00:00Z
  • IN: 38560.3MB
  • Increase used space: 17694.3MB
  • Efficiency: 46%
  • Date: 2023-05-04T00:00:00Z
  • IN: 37709.4MB
  • Increase used space: 21885.2MB
  • Efficiency: 58%
  • Date: 2023-05-05T00:00:00Z
  • IN: 35734MB
  • Increase used space: 10888.4MB
  • Efficiency: 30%
  • Date: 2023-05-06T00:00:00Z
  • IN: 41103.6MB
  • Increase used space: 28519.2MB
  • Efficiency: 69%
  • Date: 2023-05-07T00:00:00Z
  • IN: 54274.2MB
  • Increase used space: -4460.8MB
  • Efficiency: -8%
  • Date: 2023-05-08T00:00:00Z
  • IN: 32035.5MB
  • Increase used space: 14108.8MB
  • Efficiency: 44%
  • Date: 2023-05-09T00:00:00Z
  • IN: 24599.2MB
  • Increase used space: 9432.8MB
  • Efficiency: 38%
  • Date: 2023-05-10T00:00:00Z
  • IN: 51574.6MB
  • Increase used space: 12455.4MB
  • Efficiency: 24%
  • Date: 2023-05-11T00:00:00Z
  • IN: 40622.4MB
  • Increase used space: 20890.4MB
  • Efficiency: 51%
  • Date: 2023-05-12T00:00:00Z
  • IN: 49170.3MB
  • Increase used space: 76446.4MB
  • Efficiency: 155%
  • Date: 2023-05-13T00:00:00Z
  • IN: 34594.7MB
  • Increase used space: 0MB
  • Efficiency: 0%

Total:

  • IN: 249129.4MB
  • Total used space: 92275.6MB
  • Efficiency: 37%

Indeed showing an efficiency of 37% in total.
Interestingly to see, is that sometimes there seems to be an efficiency of >100% (restorations? Other cut-off time of the bandwith than storage-at-rest-time?

Indeed showing an efficiency of 37% in total.
Interestingly to see, is that sometimes there seems to be an efficiency of >100% (restorations? Other cut-off time of the bandwith than storage-at-rest-time?

I thought, maybe a lot of the ingress-traffic can be explained by not winning the competitions with other nodes:
So…

root@Storj-node4:~# docker logs storagenode 2>&1 | grep -c “upload started”
239037
root@Storj-node4:~# docker logs storagenode 2>&1 | grep -c “uploaded”
238371

So, I seem to win 99.7% of the races. Not so bad, so to say, but doesn’t seem to explain the gap between ingress and storage.

The fact trash is also 17GB and the node is the newest (so less than 30 days), also not explaining this part I would say.

So, any thoughts on the matter? Explanations why this protocol seems to be so inefficient? And if so, are there tuning options or developer initiatives to improve this?

Post-script: I was doubting whether this shouldn’t be in the developer section. But because I’m not a developer and it’s an interesting question for storagenode-owners, I decided to post it here.