Earnings calculator (Update 2023-12-05: v13.1.0 - Now with support for different payouts per satellite - Detailed earnings info and health status of your node, including vetting progress)

I have some discrepancies between the earnings calculator and the web dashboard regarding the age of some nodes. In all cases your script reports first contact joined in the last one or two days of the month whereas the dashboard puts in in the next month.

Is the GUI not taking the data from the same database/table like you do?

I’ve seen something like this on my own node as well. I can only guess as to what the cause is. I’m pulling the join date from reputation.db. It’s reported back by the satellites. I then simply calculate the number of months between now(UTC) and that date. I can only speculate on what happens on the dashboard, but I think they may be calculating the number of paystubs in heldamount.db instead. This will probably be confirmed if we don’t see that month number on the web dashboard go up on June 1st, but only after payout is done. I think that was the case this month as well, but I’m not entirely sure.

It does result in the earnings calculator saying us-central-1 is in month 16 right now, while I haven’t received my held amount return on that sat. I’m not sure how to work around that without displaying the wrong month number for almost half a month. And honestly I think it’s mostly an edge case when a node has made contact with a satellite but not received any data until the next month.

With the upcoming updates on the SNOboard, this will also become visible as they plan on showing the join date and month number right next to each other.
image

1 Like

Hrmm, actually, the web dashboard does something even weirder.

There are so many things weird/wrong here, I’m not entirely sure where to begin.

const now = new Date();

Not entirely sure, but I think this would use local date, not UTC.

const secondsInMonthApproximately = 2628000;

An average number of seconds in a month is used… why? This will lead to rounding errors.

return Math.ceil(differenceInSeconds / secondsInMonthApproximately);

Number of months is calculated as the number of “approximate” months between join and now rounded up. Which is simply wrong as it shouldn’t be counting a time difference, but the number of calendar months.

If you join on January 31st and it is now February 1st, it’s still your second month, but that one day difference rounded up would be counted as 1 month with this method.
The strange part is that it’s so simple to do it right:
(utcnowYear - joinYear) * 12 + (utcnowMonth - joinMonth) + 1
No need to bother with number of seconds in a month.

So I’m left to conclude the calculation on the web dashboard is currently not correct. That doesn’t really solve the problem entirely though. As I joined us-central-1 on 2019-02-28, I’m technically in month 16 now and so I should have received half my held amount back with the latest payout, but I didn’t. I don’t really feel like digging through more code right now to find out how the satellite determines this month number, especially since I don’t have access to satellite db’s to check my data there anyway. But needless to say, this may need some future fine tuning.

@littleskunk: I’m not entirely sure who to point this discrepancy in calculation out to. Perhaps you can confirm my findings and/or send them on to the correct person?

6 Likes

we’re passing this along to the internal team.
dealing with Date() is one thing I really dont miss from my coding days lol.
thanks for raising this point!

3 Likes

Small bug fix today. In a previous release I switched the calculation of hours per month to a static 730 hours which Storj uses to determine payout per month. However, this variable was also used to calculate how far along we are in the month. Since we’ve passed the 730 hours this month (May has 31 days and is longer than the average month), the estimate by end of month was displayed as lower than actual earnings so far. That obviously can’t be correct. This version switches back to using the actual calculation for calculation of month progress, so we can never be more than 100% done with the month.

Changelog

v9.2.2 - Fix time range estimate end of month

  • Fixes the calculation for month progress to use actual number of hours in month instead of 730
2 Likes

These are great findings. Please open a github issue or maybe even a PR with your code change.

This is the correct behavior. Lets say you join on the 31th and you get some download traffic. At the end you might get a 1 cent payout. → We count that month even if you receive 0 payout.

For your held amount you might have received it in month 12 already. In the first version of our new payout calculation was a bug that we fixed quickly but it “affected” about 50 storage nodes that joined very early. Have you check the last few month in the storage node dashboard API? Watch for the disposed value. That should tell you when it was payed and how much.

I think we’re in agreement. The current calculation on the web dashboard would still say you’re in month 1 on February 1st, but it should be month 2 at that point. My initial explanation perhaps wasn’t clear.

I didn’t even have to look. I’ve stared at that heldamount.db for waay more time than I care to admit to get all the features built in to the earnings calculator. Despite that I looked again just to be sure, but there I don’t have any disposed values other than 0. I’m inclined to think the joined_at date for this satellite is not correct and is more likely to be the issue. The reason is that all other satellites have an exact timestamp, while this one has 2019-02-28 00:00:00+00:00. I also remember I didn’t get a payout in that first month, which could be the reason it is not counted by the satellite. At this point I don’t think it’s important enough to look into it further before the next payout happens. I’m pretty sure it will be included in that one anyway.

I’ll create an issue as I don’t currently have a development environment set up to quickly test the changes. That and I haven’t worked with typescript before. It’s always easier to read a new language than to write in it. :wink:

Edit: Done Incorrect calculation of month number on SNOboard · Issue #3902 · storj/storj · GitHub

1 Like

We wiped us central at some point. To make sure the node age for the held amount calculation is correct we used the payout list from a previous month. In that payout list we had only the day without a timestamp. Your node age should be correct.

Let’s wait for the current payout and if you still don’t see it I can dive into the data we have on the satellite side.

1 Like

What does that mean?

Where does it show the surge payouts? Perhaps I didn’t get any?

Instead of underlining text I get ANSI codes, like this:

e[4m

Payout and held amount by satellite:e[0m

Thanks, very useful script and I agree, it should be included in the dashboard.

They are reasons for payout to be withheld. Sanctioned countries refers to countries which are sanctioned by the US government and there are trade limitations that prevent Storj from being allowed to pay anyone in that country.

The 1099 form is a tax form that is required for Storj to pay a SNO more than… I believe something around $600. But I don’t remember the exact amount. This is only applicable to US residents.

Disclaimer: I am not a lawyer and don’t live in the US so I’m not familiar with the applicable details. This is just my understanding.

I’m aware that this doesn’t work correctly in powershell or cmd. Try the new windows terminal. It should work with that. And… it’s a pretty nice improvement on powershell or cmd anyway and lets you use both as well as WSL from within a single app.

2 Likes

This update has taken a little longer than I would like. The reason is that my nodes are on the docker release and the delayed update roll out made it take longer until my nodes had the required data for this update. Most important added feature is the suspension score. Over the past few weeks many nodes have been suspended and at least for now, the score this is based on isn’t displayed anywhere. The calculator will now show this score so you know exactly how close to suspension you are or how close you are to recovering from suspension.
This score has been added as part of a larger rework of the scoring display. The new scores work different from before. The new scores display how close you are to either disqualification or suspension. 0.0 is a perfect score. 100% means you are suspended or disqualified. This makes it easier to know how close to trouble you are without having to know what the threshold is for disqualification or suspension. The downside is that it deviates from what the web dashboard shows.
Another new feature is that the calculator now has a WARNING status when any of the percentages for disqualification or suspension go over 5%. This indicates your node may need some attention as it’s failing audits.
Let me know what you think about this new system!

Note: This update requires storagenode version v1.6.3+

image

Changelog

v9.3.0 - Status reworked, suspension added

  • Reworked the status display
  • Suspension scores (unknown audit scores) added
  • Warning when there might be an audit issue
3 Likes

I would like to recommend a space of 1 line after every sat’s rows to make it more readable.

us-central-1      18
    Status: Suspended @

europe-west-1    15
    Status: OK

europe-north-1

I have a bit of a selfish reason why I don’t want to do that. The calculator won’t fit in my tmux pane anymore if I do that. But this screenshot is messy because it has examples for every situation. It’s much easier to read if your node is performing normally. I also updated the screenshot in the top post with a normal situation screenshot. I think that looks fine, but let me know what you think.

2 Likes

Note I’ve already posted this error to GitHub…

Getting error when running the latest version of the calculator:

Traceback (most recent call last):
  File "/home/bill/source/storj_earnings/earnings.py", line 283, in <module>
    for data in con.execute(query):
sqlite3.OperationalError: no such column: audit_unknown_reputation_score

This error is occurs when the V9.2.2 V9.3.0 earnings.py is run against the pre- V1.6.4 databases.

You need to wait until your node is updated to the latest version.


EDIT: didn’t actually check the version, just used the title. Sorry for any confusion.

1 Like

OK, Just need to wait a while then… :smiley:

Whoops, my bad for not updating the topic title. Should be correct now.

Responded to this on Github already. But just to be sure.

3 Likes

Looks great! As always, very appreciative of your contributions!

1 Like

Hi, good afternoon.

I’m getting an error in running the program. I’ll leave you two captures below. The first one shows the files that you have to copy to another folder, and I think that there is some file missing that you have to copy, because when I make the copy manually of ALL the files that are inside the node, the problem disappears.

Which is the file that I have to incorporate to give us the error?

The paystubs table is in the heldamount.db file. So you’re not missing a file. But if you copy the files, it’s best to do that with the node stopped. Otherwise you could be missing data that’s in the .db-wal files. So stop the node first, copy, then start the node again and run the script on the copy. That will likely fix it, but please report back if it doesn’t.

Edit: Btw, if you’re running a windows GUI node it’s safe to run the script on the live db’s without copying. But definitely don’t do that if you’re using docker as it could corrupt the databases on docker nodes on windows or macos.