Earnings calculator (Update 2024-04-14: v13.4.0 - Additional info on uncollected garbage and unpaid data - Detailed earnings info and health status of your node, including vetting progress)

LinuxNet · April 18, 2020, 10:32pm

will probably be the new north-east-1? just a guess.

https://forum.storj.io/t/new-tardigrade-satellite

Edit: Mean europe-north-1 sorry

BrightSilence · April 18, 2020, 10:34pm

europe-north-1 it looks like. I’ll add the name in the next release. Unfortunately I still have to add them manually.

demeo · April 18, 2020, 10:35pm

Grep or query names from www dashboard

BrightSilence · April 18, 2020, 10:49pm

Yeah, I’m definitely not going to do that. That would mean users would also have to deal with telling the calculator the ip/domain and port for the web dashboard. Sats are added rarely enough, that it’s a super easy manual fix. Preferably I’d pull the name from one of the databases, but as of now none seem to store the satellite name. Anyway, here you go. New version is up on github linked in the top post.

Changelog

v8.2.1 - Added europe-north-1 satellite

Added europe-north-1 satellite

BrightSilence · May 1, 2020, 9:21pm

Development update

Usually when a new node version drops, I take a look at the new goodies, implement new features and the first thing you hear from me is an update with a changelog. This time, I’m a little bit at a loss as to what direction to take.

New data available

The 1.3.3 update adds an important table with new information, called paystubs. This table is filled by the satellite at the moment that payouts go out. It contains information regarding the payout calculation like at rest storage, ingress, egress, held back amounts, payouts of that held back amount, surge percentage and other stuff. This data is used by the web dashboard to show payout information for previous months.

So that sounds great, right?

It’s very good to have this new information available. There is some stuff in there that is currently missing from the earnings calculator, like surge percentage, held back amount totals and payout of that held back amount. Additionally the number of paystubs shows exactly how many months your node has existed, so there is no longer a requirement for the less accurate first contact variable. Based on that output can be cleaned up to only show the correct held back percentage instead of all of them, held back amount can be added to the list at the top and included in totals calculation.

So what’s the problem?

The problem is that this table is missing data until the payouts for the previous month are done. This is why you’re seeing no data for April on the web dashboard right now. It’s also why my dashboard claims I’m in month 14 for stefan-benten’s satellite, while I’m actually in month 15. So all this great new information is basically incorrect/incomplete for half of the month.

Dilemma

I’m a bit at a loss on what to do. The earnings calculator has always based previous months information based on actual usage tracked by the node itself. This new table has a lot of that same information, but based on the paystub calculated by the satellite. I personally like the idea of keeping the earnings calculator based on the nodes local data, so it can function as a sort of check on the satellites calculations. But I would like to hear your input on this.
Additionally I was planning on redesigning the output of the payout and held back break down per satellite, based on the exact number of months that had passed.
Something like this:

SATELLITE       HELD TOTAL  MONTH   HELD%   EARNED       HELD         PAID        SURGE%
us-central-1    14.20 USD   8       25%     0.0164 USD   0.0041 USD   0.0123 USD  100%    
Status:OK (Audit score:1000)

But since that month number seems to be unreliable as well, I’m back in the same boat not being sure what month your node is actually in on each satellite, which makes the rest of the calculation impossible. Additionally surge % only becomes known when the paystub data is filled as well. (It doesn’t help that the table has 250 when the surge payout was 250%, but 0 when there is no surge, which means just a normal 100% payout, but I can work around that part.)

Feedback

I’d love to hear specifically your feedback on the following questions:

Do you think the earnings calculator should display previous months earnings based on paystub data like the web dashboard does or actual node accounting to function as a sort of check on the satellite calculation?
Would you rather I implement the new interface as suggested, knowing that for half of the month it may display the held back percentage for the wrong month or stick with the current setup?

I’m gonna sleep on it for a while. This update will take a little longer than you’re used to as a result of this dilemma. Luckily the current version still works just fine after the update. You’ll just have to wait a bit for the new goodies.

kevink · May 2, 2020, 5:23am

I think it is a valuable tool to get an earnings estimation from the data that the storagenode itself gathered and not use the paystub data.

Of course you could make it a comparison tool that also reads the paystub and then compares if the results are equal If you have too much time… (or fun).

If there was no payout in paystub for the last month but first contact was not within the last month, then you can assume that the node is one month older than first contact. If there was a payout, then the age is from first contact to the month queried.
Did that explanation make any sense to you?

BrightSilence · May 2, 2020, 11:50am

What a difference a good night sleep makes. I found a solution to the node age problem and was able to implement everything I wanted to. The solution is similar to what @kevink suggested, so thanks for thinking with me there! There is lots of new stuff and I’ve changed quite a bit of code. Please let me know if you run into any issues.

Changelog

v9.0.0 - Surge, held amount, redesign sat breakdown

Requires storagenode version 1.3.3 (Data will be incorrect on older versions)
Requires heldamount.db, if you copy db’s prior to running the script, this one needs to be included now
Added surge payout information
Added held amount totals so far
Accurate display of month number for your node on each sat
Redesigned breakdown per satellite
Display only relevant held back % and amount and paid amount
Added totals for satellite breakdown

peem · May 2, 2020, 12:01pm

thank you for your job well done

BrightSilence · May 2, 2020, 12:01pm

When you change a lot of code… sometimes you break things. Quick bugfix release.

Changelog

v9.0.1 - Fix disk current

Previous version introduced a miscalculation of current disk usage. This version fixes it.

stuberman · May 3, 2020, 12:12am

9.0.1 looks great! Thanks for the update!

fonzmeister · May 3, 2020, 8:54am

Looks great! I agree with @kevink btw. This is a valuable tool if you use the data that the storagenode gathered for the estimation. In this way we can verify storj if the payments are correct. (do not trust, verify )

Anyway, I see the info if the satellite is OK or is being vetted / DQ is gone?

BrightSilence · May 3, 2020, 8:58am

It’s not. There is no historic information for that, so it has always only been displayed when you’re looking at the current month like in the first screenshot. Since the surge information is by definition only available after payout, so never for the current month, I used that second line to display surge information for historic months if applicable.

BrightSilence · May 3, 2020, 8:20pm

Minor bugfix. The surge payout line was not displayed in the top overview for months prior to 2019-09.

Changelog

v9.0.2 - Fix surge payout prior 2019-09

Bugfix: Surge payout was not displayed for months with missing disk usage stats (prior to 2019-09)

punyelroo · May 4, 2020, 3:02am

@BrightSilence - I have 9.0.2 and v1.3.3 running

I see the following

Specifically the 13 month and 12 month on a couple of the satellites … the oldest one I have is from Aug 19 so could not possibly be 12/13 months - I’d love it if it was but something seems wrong to me
Here’s a grab of the same node on the 30th April

Am I missing something here?

BrightSilence · May 4, 2020, 7:26am

Thanks for the feedback, I’d love to look into what’s going on there. But I can’t really without being able to look at your heldamount.db. The numbers for my node seem to be correct, but I have only one node to test on. Would you mind sending me your heldamount.db file? (In a private message)

I don’t want you to share anything you’re not comfortable with though, so for your information, this db contains one empty table and a paystubs table with the following information on all previous payouts.

CREATE TABLE IF NOT EXISTS “paystubs” (
“period” text NOT NULL,
“satellite_id” bytea NOT NULL,
“created_at” timestamp NOT NULL,
“codes” text NOT NULL,
“usage_at_rest” double precision NOT NULL,
“usage_get” bigint NOT NULL,
“usage_put” bigint NOT NULL,
“usage_get_repair” bigint NOT NULL,
“usage_put_repair” bigint NOT NULL,
“usage_get_audit” bigint NOT NULL,
“comp_at_rest” bigint NOT NULL,
“comp_get” bigint NOT NULL,
“comp_put” bigint NOT NULL,
“comp_get_repair” bigint NOT NULL,
“comp_put_repair” bigint NOT NULL,
“comp_get_audit” bigint NOT NULL,
“surge_percent” bigint NOT NULL,
“held” bigint NOT NULL,
“owed” bigint NOT NULL,
“disposed” bigint NOT NULL,
“paid” bigint NOT NULL,
PRIMARY KEY(“period”,“satellite_id”)
);

As you can see there is no personal identifiable information in there, but if you consider bandwidth usage numbers, disk usage numbers and corresponding payouts and held amount private. Please don’t share this with me. Unfortunately without the db it will be hard for me to debug.

andrew2.hart · May 4, 2020, 11:13am

This appeared just once :

TOTAL				7.24 USD	0.3912 USD		0.1956 USD	0.1955 USD
				SURGE (100%)	0.3912 USD		0.1956 USD	0.1955 USD

I guess someone applied surge to May instead of April (lol)

BrightSilence · May 4, 2020, 11:22am

Thanks for the feedback. I saw that once too. I think it’s a rounding error in calculation. For the totals I compare total incl surge to total without surge to determine whether surge is applied. This is mostly to account for the possibility of some satellites having surge, but others not. I’ll add in a small margin to prevent these rounding errors from happening in the next version.

Edit: Actually, going to use a better method that doesn’t need a margin.

Nope, as you can see the numbers are actually the same and it says 100%(meaning no more than normal). So the calculation ends up with the exact same amounts as pre-surge. That’s because the rounding error is way more decimal points beyond what is actually displayed.

andrew2.hart · May 4, 2020, 11:56am

And thanks for the new version too.

BrightSilence · May 4, 2020, 12:02pm

I have a new version waiting with more new features and this fix. Still working out the last details and testing. But here’s a sneak preview.

This version will now calculate the correct held back amount, also taking into account amounts that have been paid back in the past. It will also show additional lines for held amount paid back due to either graceful exit or the month 15 return of half the held back amount. Thanks @kevink for sharing his db with graceful exit data, so I could incorporate these options! My node never received any returned amount so far, so I needed a bit of outside help.

I still want to look into @punyelroo’s issue though. I just received the db but with incomplete info. My bad, I should have specified to stop the node prior to copying the db. Just waiting on that and hoping to release this new version today or tomorrow.

bdurrer · May 4, 2020, 1:41pm

I get the following error when running the script (Linux with storj in docker):

python earnings.py /share/external/DEV0201_1/data/storage
Traceback (most recent call last):
File “earnings.py”, line 225, in
con.execute(‘ATTACH DATABASE ? AS su;’,tSU)
sqlite3.OperationalError: file is encrypted or is not a database

Do I need python 3 or something? I tried python 2.7.5 so far