Earnings calculator (Update 2022-04-14: v11.1.0 - Detailed earnings info and health status of your node, including vetting progress)

So it’s only been a week but so far my vetting seems to continue linearly rather than exponentially. Ingess traffic has grown a bit from 1GB/day to around 2GB/day, about 1/5 of my ingress traffic is quickly deleted again and moved into trash, and vetting continues at around 1/day for the two nodes with the most traffic, eu1 and us1. Concerning egress repair I don’t have any so far.





1 Like

This is great info, thanks. It does seem that some adjustment may be warranted. It would be nice to see progress in about a week again. No need to post daily though.

What makes it slightly more difficult to determine this in a reasonably reliable way is that generally audits scale with data stored, but there is a system that prioritizes audits for new nodes. The smaller the share of data on a new node compared to total stored, the more linear the effect will be. And it seems to be slowly moving towards there. So the question is, what part of this process is now linear. It used to be almost negligible, but seems to be more significant now.

It seems vetting depends entirely on some different and external factor. I have had a lot of vetting for a few days and now next to no change since 7th of April.

1 Like

It’s at the core still a random process, which is what makes it so hard to create good predictions. But I feel I now have enough data to show what I’ve been doing with it, though not enough yet to draw definitive conclusions. I’ll walk you through some of my early findings though.

I plotted your numbers on a time line based on weekly reports. Your previous post didn’t include timestamps for each image, so I just plotted the first and last one. More detail would likely just add more noise anyway.

Here’s what I found


Dashed lines are the predicted progress.

As you can see the actual number of audits does start to show the exponential growth I’m trying to compensate for, but it is less pronounced than I was expecting and less pronounced than it used to be as well. As a result you see the percentage display overcorrects for it.

Based on the numbers you provided I’ve been tuning the formula to better reflect the actual ratio between linearly generated audits per unvetted node and per segment audits which are influenced by amount of data stored. I came up with the following preliminary graph.

With these adjustments the lines look pretty linear, which is what is the intention. With the exception of ap1. But with that being a smaller satellite it is more susceptible to random fluctuations.

It’s not enough for a final update just yet, I fear I may have gone too far in the other direction now when the process moves further, so if you could report again about once a week that would be awesome. If anyone else wants to contribute roughly weekly progress updates on their vetting progress that would be greatly appreciated as well.

4 Likes

I will report back in another week. On another note though, my ingress traffic has been growing from 1GB/day to 2-3GB/day. Is vetting not a binary status anymore but a linear progress, or is this just random and attributed to growing network usage? I know repair traffic has been higher than usual due to Storj increasing the minimal amount of pieces that need to be healthy.

2 Likes

It’s that mostly.
Thanks again for your help!

1 Like

I’ll try to post mine every week as well. Here we are today:

1 Like

Here’s my vetting status. Hope it helps!

1 Like

@BrightSilence it should be possible to export audit stats from Prometheus grabbers I have running for Grafana stats. Are you interested in data like that?
Edit:
This is the example of one node, one satellite (us1), note that when number decreased, grabber was restarted:

I don’t get what this is supposed to contain, it just says a time and then some number of “Successful”. Doesn’t that just show how often your node was successfully tested as online by Prometheus? If so, I don’t think it can be of much use here. Are you maybe confused because Prometheus also calls it “Audit”? Or am I blind and maybe not seeing something here?

That is the counter of succesfull audits from given satellite, so to my understanding, what @BrightSilence asked for. The source for Prometheus is storj-log-exporter watching storagenode log.

Thanks for this, but I’ll need a bit more info before I can judge whether this is useful.

  • Are these audits across all satellites?
  • When did this node start?
  • Is this node already vetted on any of the satellites?

Example data is one satellite (us1). I fixed the counter resets:

This is the node report. Note that I have started log exporter on 2022-02-13 and node on 2022-02-03, which is the reason log exporter contains 32 from 37 audits. Csv counter should start at 5.

Awesome, yeah that does look useful, thanks!
If it’s not too much trouble it would be great to get that data for the other satellites as well. I’ll make another one of those graphs and see if I can tune further.

Would also be awesome if you could provide an update after getting is done on several satellites.

Sure, I just asked if it is worth the time first :slightly_smiling_face:
Here I provide the “prometheus logs” and “earnings calculator output” for two nodes. Node0 is vetted on two satellites (us1, eu1). Logs are missing even more of node lifetime, since node was started sooner than node1. I wish I can be more precise, but only I can tell, node0 was vetted around 28.2. on us1 and few (3?) days later on eu1.
Hope it helps.

Yeah, this is awesome, thanks! I’ll have to spend a bit of time on it, but I can definitely work with that. Across 2 nodes you’ve provided data from earlier as well as later in the process. It’s ok that it’s not entirely complete. Kind of used to working with incomplete data in my day to day to be honest. :stuck_out_tongue:

I’ll report back on my findings when I have some info. But thanks for this!

1 Like

If it’s of any help :
I’ve a text log file for my node going back to April 2021 (I run the earnings report every 24 hours) It was a new node when I started it.

Just need to get it uploaded somwewhere…

It’s been another week so here’s my output again, audits have been slow, traffic was steady, about 2-3GB/day but quite a lot of that (1-2GB) is deleted again within a few days. Sorry for the weird format I’m on my phone.

My weekly report:

Its been another 10 days and vetting still looks rather linear to me, I’ve been seeing 2-3 vets per day on eu1, less on all the others.