Earnings calculator (Update 2021-11-03: v10.3.0 - Detailed earnings info and health status of your node, including vetting progress)

When I run this (substituting my data path, which is /mnt/storj), I get:

`ERROR: bandwidth.db not found at: "/mnt/storj/storage" or "/mnt/storj".`

@BrightSilence, you pointed out this is a permissions problem; I did try running it with “sudo” and that was successful (and I appreciate the security caution you raised).

I’d like to figure out how to do this properly - that is, how to grant the appropriate permissions to my user account, so I don’t need to use sudo. But, I’m confused on a couple points.

  1. The error message seems a little off, it seems like the script should be able to find the data, it just can’t access it. OK, so the error message is just not perfect, no big deal there.
  2. But the file in question is already globally readable! It has permissions -rw-r–r–.

So, do I have to make every file in that directory readable by my linux user? Or are there a finite number of key files I need to chmod to make it work?

In my case, I have the original db files owned by my linux user while keeping the root group (root permissions bypass the write restrictions that would’ve been forced anyways while still keeping the assigned user intact). I don’t have all the current files owned by my linux user and the script still works (same permissions as yours). I’m on Kubuntu 18.04 (it’s still primarily headless for me though) so maybe it’s something like an SELinux issue? Personally, I’m tempted to just do chown myuser:root -R /opt/storj/data/storage (I might be fudging the backticks, but we’ll see once I post this reply) even though it works just so I never have to worry about it.

EDIT:
Upon further looking, that chown would be a bad idea… Maybe if I were doing ./.db (Discourse, quit using ASTERISKS FOR ITALICS YOU PIECE OF CRAP!) but otherwise, that’s a bad idea.

I just use a python function that checks whether a path points to a file, which apparently says it doesn’t if you don’t have the correct permissions.

It may be that the storage folder doesn’t have the correct permissions set on the folder level.

That’s why we have code blocks!

```
code here

```
You’d get this

code here
1 Like

Ah, thank you. My two issues were related, turns out.

  1. The script couldn’t find the file because it couldn’t access the directory it was in.
  2. By running sudo chmod 711 /mnt/storj/storage I made the directory globally executable, which solved the problem!

Satisfying, thanks for the tips.

I’m sure there’s a more elegant way to do it - e.g., using permissions 701 and adding my linux user to the appropriate group. But for now, I’m happy with this (and I think my machine is secure enough that I’m not worried about anyone accessing it to be able to cause harm in this directory).

1 Like

Do code blocks allow asterisks to pass through though? Only one way to find out I guess.

chown myuser:root -R ./*.db*

According to the preview it works, but we’ll see I guess.
EDIT: I HAVE LEARNED SOMETHING TODAY! Test complete, please ignore XD

6 Likes

I have a small update today to implement a more realistic vetting progress measurement. Since the number of audits you get depends on how much data you have, this process is by definition non-linear. The first audit takes much longer than the next one and so on. This update accounts for that and instead of showing just how many audits you had, it now shows a calculated linear progress percentage (in addition to absolute audit numbers). Let me know if this is more or less accurate in your experience. It should be a lot more accurate, but I’d love to get some feedback as I don’t regularly go through the vetting process myself.

image

Edit: More details on this update can be found here: Intro to the Web Dashboard - #6 by BrightSilence

Changelog

v10.2.0 - Non-linear vetting progress

  • Implemented a new algorithm to account for the non-linear progress of vetting, which should give a better indication of how long is left in vetting
5 Likes

Excellent! I’ll paste my numbers below from the previous script, and the present one.

OLD

e[4mJune 2021 (Version: 10.1.2) [snapshot: 2021-06-09 02:22:16Z]e[0m
TYPE PRICE DISK BANDWIDTH PAYOUT
Upload Ingress -not paid- 4.18 GB
Upload Repair Ingress -not paid- 29.47 GB
Download Egress $ 20.00 / TB 9.96 GB $ 0.20
Download Repair Egress $ 10.00 / TB 111.99 MB $ 0.00
Download Audit Egress $ 10.00 / TB 21.25 KB $ 0.00
Disk Current Storage -not paid- 150.44 GB
Disk Average Month Storage $ 1.50 / TBm 32.23 GBm $ 0.05
Disk Usage Storage -not paid- 23.21 TBh
________________________________________________________________________________________________________+
Total 32.23 GBm 43.72 GB $ 0.25
Estimated total by end of month 119.39 GBm 161.95 GB $ 0.92
e[4m
Payout and held amount by satellite:e[0m
NODE AGE HELD AMOUNT REPUTATION PAYOUT THIS MONTH
SATELLITE Joined Month Perc Total Disq Susp Down Earned Held Payout
us1.storj.io | 2021-04-30 3 | 75% $ 0.08 | 0.00% 0.00% 0.00% | $ 0.0293 $ 0.0219 $ 0.0073
Status: 31% Vetted
us2.storj.io | 2021-04-30 3 | 75% $ 0.00 | 0.00% 0.00% 0.00% | $ 0.0004 $ 0.0003 $ 0.0001
Status: 35% Vetted
eu1.storj.io | 2021-04-30 3 | 75% $ 0.02 | 0.00% 0.00% 0.00% | $ 0.0291 $ 0.0219 $ 0.0073
Status: 34% Vetted
ap1.storj.io | 2021-04-30 3 | 75% $ 0.00 | 0.00% 0.00% 0.00% | $ 0.0010 $ 0.0008 $ 0.0003
Status: 6% Vetted
europe-north-1 | 2021-04-30 3 | 75% $ 0.00 | 0.00% 0.00% 14.29% | $ 0.0008 $ 0.0006 $ 0.0002
Status: 7% Vetted
saltlake | 2021-04-30 3 | 75% $ 0.11 | 0.00% 0.00% 0.00% | $ 0.1881 $ 0.1411 $ 0.0470
Status: OK
_____________________________________________________________________________________________________________________+
TOTAL $ 0.22 $ 0.2487 $ 0.1865 $ 0.0622

								  POSTPONED PAYOUT PREVIOUS MONTHS $  0.0729

NEW

e[4mJune 2021 (Version: 10.2.0) [snapshot: 2021-06-09 02:24:23Z]e[0m
TYPE PRICE DISK BANDWIDTH PAYOUT
Upload Ingress -not paid- 4.18 GB
Upload Repair Ingress -not paid- 29.47 GB
Download Egress $ 20.00 / TB 9.96 GB $ 0.20
Download Repair Egress $ 10.00 / TB 111.99 MB $ 0.00
Download Audit Egress $ 10.00 / TB 21.25 KB $ 0.00
Disk Current Storage -not paid- 150.44 GB
Disk Average Month Storage $ 1.50 / TBm 32.23 GBm $ 0.05
Disk Usage Storage -not paid- 23.21 TBh
________________________________________________________________________________________________________+
Total 32.23 GBm 43.73 GB $ 0.25
Estimated total by end of month 119.37 GBm 161.95 GB $ 0.92
e[4m
Payout and held amount by satellite:e[0m
NODE AGE HELD AMOUNT REPUTATION PAYOUT THIS MONTH
SATELLITE Joined Month Perc Total Disq Susp Down Earned Held Payout
us1.storj.io | 2021-04-30 3 | 75% $ 0.08 | 0.00% 0.00% 0.00% | $ 0.0293 $ 0.0219 $ 0.0073
Status: 75% Vetting progress (31 / 100 Audits)
us2.storj.io | 2021-04-30 3 | 75% $ 0.00 | 0.00% 0.00% 0.00% | $ 0.0004 $ 0.0003 $ 0.0001
Status: 77% Vetting progress (35 / 100 Audits)
eu1.storj.io | 2021-04-30 3 | 75% $ 0.02 | 0.00% 0.00% 0.00% | $ 0.0291 $ 0.0219 $ 0.0073
Status: 77% Vetting progress (34 / 100 Audits)
ap1.storj.io | 2021-04-30 3 | 75% $ 0.00 | 0.00% 0.00% 0.00% | $ 0.0010 $ 0.0008 $ 0.0003
Status: 42% Vetting progress ( 6 / 100 Audits)
europe-north-1 | 2021-04-30 3 | 75% $ 0.00 | 0.00% 0.00% 14.29% | $ 0.0008 $ 0.0006 $ 0.0002
Status: 45% Vetting progress ( 7 / 100 Audits)
saltlake | 2021-04-30 3 | 75% $ 0.11 | 0.00% 0.00% 0.00% | $ 0.1881 $ 0.1411 $ 0.0470
Status: OK
_____________________________________________________________________________________________________________________+
TOTAL $ 0.22 $ 0.2488 $ 0.1866 $ 0.0622

								  POSTPONED PAYOUT PREVIOUS MONTHS $  0.0729
1 Like

Tiny update today, not really a need to update if you don’t feel like it. I just tweaked some conditional statements to ensure lines that aren’t relevant don’t show up with a value of 0. In most cases this didn’t happen to begin with, but due to the use of floats, which are an inexact datatype, the values were slightly over 0. I’ve corrected for this by implementing a margin for the conditional statements.

Changelog

v10.2.1 - Fix: 0 lines showing for rounding errors

  • Implements a small fix for lines with 0 showing up in case of float rounding errors

What % audit disqualification (Disq column) as printed by this script is equal to my node being perrmanently disqualified?

This is output from a node I rescued (ddrescue) from a failing 1TB disk onto a replacement 6TB disk. In the process I lost around 15MB because of bad blocks, hence the failing audits. Downtime is around 3 days it took for the rescue operation.

I know some of you may look down on running this node because I lost any data. But bad blocks happen and the network is fault tolerant. If I fail enough audits the node is toast.

June 2021 (Version: 10.2.1)                                             [snapshot: 2021-06-20 00:16:23Z]
                        TYPE            PRICE                   DISK       BANDWIDTH             PAYOUT
Upload                  Ingress         -not paid-                          41.14 GB
Upload Repair           Ingress         -not paid-                          65.97 GB
Download                Egress          $ 20.00 / TB                        65.33 GB            $  1.31
Download Repair         Egress          $ 10.00 / TB                        26.24 GB            $  0.26
Download Audit          Egress          $ 10.00 / TB                       318.46 KB            $  0.00
Disk Current            Storage         -not paid-         990.13 GB
Disk Average Month      Storage         $  1.50 / TBm      555.53 GBm                           $  0.83
Disk Usage              Storage         -not paid-         399.98 TBh
________________________________________________________________________________________________________+
Total                                                      555.53 GBm      198.68 GB            $  2.40
Estimated total by end of month                            876.63 GBm      313.52 GB            $  3.79

Payout and held amount by satellite:
                        NODE AGE         HELD AMOUNT            REPUTATION                 PAYOUT THIS MONTH
SATELLITE          Joined     Month    Perc     Total       Disq   Susp   Down        Earned        Held      Payout
us1.storj.io    |  2020-07-17    12  |   0%  $   0.37  |   0.00%  0.00%  7.10%  |  $  0.1174   $  0.0000   $  0.1174
    Status: WARNING: Downtime high
us2.storj.io    |  2021-01-07     6  |  50%  $   0.00  |   0.00%  0.00%  8.97%  |  $  0.0022   $  0.0011   $  0.0011
    Status: WARNING: Downtime high
eu1.storj.io    |  2020-07-17    12  |   0%  $   0.40  |   0.00%  0.00% 11.44%  |  $  0.1821   $  0.0000   $  0.1821
    Status: WARNING: Downtime high
ap1.storj.io    |  2020-07-17    12  |   0%  $   0.31  |  10.18%  0.00%  8.84%  |  $  0.0772   $  0.0000   $  0.0772
    Status: WARNING: Audits failing
europe-north-1  |  2020-07-17    12  |   0%  $   1.49  |  11.28%  0.00%  5.36%  |  $  0.3733   $  0.0000   $  0.3733
    Status: WARNING: Audits failing
saltlake        |  2020-07-17    12  |   0%  $   7.22  |   0.00%  0.00% 10.06%  |  $  1.6503   $  0.0000   $  1.6503
    Status: WARNING: Downtime high
_____________________________________________________________________________________________________________________+
TOTAL                                        $   9.79                              $  2.4024   $  0.0011   $  2.4013

Nah that is fine. Just keep it running. 15MB isn’t much, it should recover if there were some bigger pieces in there.

100% is where you get disqualified.
And yeah, I agree with @kevink, your node should be fine if only 15mb was lost. Just keep an eye on the scores, but I think you’ll see them recover a little while your node gets more data. For now though, they may be quite erratic. That has to do with how the scoring is currently implemented. It’ll jump up and down and that doesn’t really mean your node is doing better or worse. But if data loss is below 10%, you should be fine. Long term you want it below 2% at least though, as changes to the scoring mechanism are planned that will become more strict. Those changes should also stabilize the score though.

My Storjnode run on a Synology in a docker container. When I run your script can I do this when my Node is online? Or is it better to shutdown copy the data and then run it?

Your first post says only do that on a Win or Mac.

That’s exactly the same setup as I use and I never stop my node. As long as you run the script on the Synology itself (over ssh) it should be fine.

1 Like

@BrightSilence I seem to remember telling you I’d give you updated numbers in six weeks, about six weeks ago. But I can’t find the thread where that was. I’ve certainly started to fall substantially behind your earnings calculator (I’m about 11 weeks in and have accumulated only 240 GB of data, and only 3 satellites have vetted me - one only just yesterday) but I would guess that just has to do with reduced demand and/or a surge in SNOs. Anyway, here’s the data from your script, as of now:

July 2021 (Version: 10.2.0)						[snapshot: 2021-07-12 20:17:07Z]
			TYPE		PRICE			DISK	   BANDWIDTH		 PAYOUT
Upload			Ingress		-not paid-			    20.56 GB
Upload Repair		Ingress		-not paid-			    30.15 GB
Download		Egress		$ 20.00 / TB			    23.39 GB		$  0.47
Download Repair		Egress		$ 10.00 / TB			   137.33 MB		$  0.00
Download Audit		Egress		$ 10.00 / TB			    64.51 KB		$  0.00
Disk Current		Storage		-not paid-	   240.14 GB
Disk Average Month	Storage		$  1.50 / TBm	    82.09 GBm		$  0.12
Disk Usage		Storage		-not paid-	    59.11 TBh
________________________________________________________________________________________________________+
Total							    82.09 GBm	    74.24 GB		$  0.59
Estimated total by end of month				   214.85 GBm	   194.30 GB		$  1.55

Payout and held amount by satellite:
                        NODE AGE         HELD AMOUNT            REPUTATION                 PAYOUT THIS MONTH
SATELLITE          Joined     Month    Perc     Total       Disq   Susp   Down        Earned        Held      Payout
us1.storj.io	|  2021-04-30     4  |  50%  $   0.19  |   0.00%  0.00%  0.00%  |  $  0.0858   $  0.0429   $  0.0429
    Status: OK
us2.storj.io	|  2021-04-30     4  |  50%  $   0.00  |   0.00%  0.00%  0.00%  |  $  0.0005   $  0.0003   $  0.0003
    Status: OK
eu1.storj.io	|  2021-04-30     4  |  50%  $   0.09  |   0.00%  0.00%  0.00%  |  $  0.0337   $  0.0168   $  0.0168
    Status: 94% Vetting progress (78 / 100 Audits)
ap1.storj.io	|  2021-04-30     4  |  50%  $   0.01  |   0.00%  0.00%  0.00%  |  $  0.0028   $  0.0014   $  0.0014
    Status: 77% Vetting progress (34 / 100 Audits)
europe-north-1	|  2021-04-30     4  |  50%  $   0.01  |   0.00%  0.00%  0.00%  |  $  0.0030   $  0.0015   $  0.0015
    Status: 64% Vetting progress (19 / 100 Audits)
saltlake	|  2021-04-30     4  |  50%  $   0.71  |   0.00%  0.00%  0.00%  |  $  0.4665   $  0.2332   $  0.2332
    Status: OK
_____________________________________________________________________________________________________________________+
TOTAL					     $   1.00                              $  0.5923   $  0.2961   $  0.2961

									  POSTPONED PAYOUT PREVIOUS MONTHS $  0.3329

The script still assumes constant ingress patterns, but unfortunately ingress has dropped quite a bit since June. So I think it’s doing exactly what it’s supposed to do. Thanks for reporting back though, this confirms my calculations are actually matching real world behavior (when corrected for changes in ingress). If only I had a crystal ball I would correct for future ingress patterns too. :wink: But for now this will have to do.

I think the topic you referring to is the one I also linked in the changelog post: Intro to the Web Dashboard - #6 by BrightSilence

1 Like

Hey There :slight_smile:
I am also like 4 Month old. And thats my status. I am also still behind your predictions :wink:

Well the same response applies. Though I’m curious what predictions you are referring to as I only made some predictions in the linked topic about @o1eal’s node specifically. But yeah, ingress has slowed down, that’s always going to lead to slower vetting. Nothing I can do about that.

Ok, maybe predictions wasn’t the wrong word. I was reffering to:

You created a great tools!

@padso Interesting (maybe not surprising, but interesting) to see how similar our numbers are. Considering you do indeed have a head start on me, your total amount from each satellite is slightly higher. And our vetting status is the same, except you have been fully vetted by eu1, and I have not…though I have the most audits completed on that of the three remaining satellites (84).

@o1eal i startet on 10-04-21