Earnings calculator (Update 2024-07-28: v14.1.0 - Now shows garbage collection progress! - Detailed earnings info and health status of your node, including vetting progress and garbage collection status)

Node Operators
,
1

I made this calculator to provide payout information and be used as an extension of the dashboard functionality already provided by Storj.

Storj Earnings Calculator

Earnings calculation script for Storj V3 storagenodes

Prerequisites

Python is required to run this script.

This script was tested on Windows 10 and Linux, with Python 3.7. Other OS’s and versions will likely also work.

Warning

If you are running Docker on Windows or MacOS, stop the node, copy the satellites.db, bandwidth.db, storage_usage.db, piece_spaced_used.db, reputation.db and heldamount.db to a different location and run the script from there. Running the script on docker nodes on these OS’s while the node is using it could lead to corruption of the database, which will kill the node.

Usage

Earnings for current month:

python earnings.py /path/to/storj/data

Note: If you omit the path it will look in the current working directory.

Earnings for previous months:

python earnings.py /path/to/storj/data 2023-03

Screenshots

image
image836×515 13.8 KB

image
image837×918 30.8 KB

image
Note: Screenshot is from an older version, layout has changed, but the warnings remain similar
As this last screenshot shows, the calculator keeps track of detailed status. If one of the displayed scores goes over 100%, you will be disqualified or suspended, depending on which score. Vetting progress is also displayed. If any of the displayed scores go over 5%, the status will change to WARN, signaling that your node needs attention. Under healthy conditions you want to see 0.0% for all scores, which is a perfect score.
Note: This deviates from how scores are displayed on the web dashboard.

Get latest version

Direct link (right click > save as):

https://raw.githubusercontent.com/ReneSmeekes/storj_earnings/master/earnings.py

From CLI:

wget https://raw.githubusercontent.com/ReneSmeekes/storj_earnings/master/earnings.py

Changelog

v14.1.0 - Garbage collection progress

  • Add feature to show garbage collection progress

v14.0.0 - Compatibility with storagenode v1.104 and up

  • Breaks compatibility with node versions v1.103 and below.
  • Compatible with v1.104 and up
  • Fix an issue with including empty records in satellite stats
  • Add a note during first days of the month or when deviations are detected that satellite stats may be temporarily unreliable
Older versions

v12.3.1 - Fix division by 0 for storage usage not available

  • Fixes an issue where the script caused a division by 0 error if any satellite hasn’t yet reported any storage usage within the given month

v12.3.0 - Stabilized disk average so far and end of month estimate

  • Use byte hour timeframes to determine exact time over which disk usage has been reported so far to stabilize disk usage reporting and end of month estimates
  • Includes changes from unreleased versions below

v12.2.0 - Replace disk usage in Bh with average disk usage so far (unreleased)

  • Disk usage displayed in Bh has been dropped. As this is no longer displayed on the official dashboard it no longer serves a useful purpose
  • Disk average so far has been added to display the average amount stored on the node over the course of the month

v12.0.1 - Fix suspension threshold (unreleased)

  • Fix has been implemented to correct erroneous change to new suspension threshold. The change was only valid for audit scores

v12.0.0 - Compatibility with node version 1.62+

  • Fixes error on new node version > this version is only compatible with node version 1.62+
  • Implements new scoring threshold into audit and suspension scores (As before, a score of 0% means you’re good, 100% means you’re suspended or disqualified)

v11.2.0 - Fallback satellite names (unreleased)

  • Adds fallback for satellite names for new nodes (It takes up to 24 hours before satellite names are available in the node db’s)

v11.1.0 - Adjusted linear audit progress percentage

  • Audit progress was adjusted based on community feedback

v11.0.0 - Redesign + dynamic satellite names

  • Pull satellite name from database
  • Redesign payouts table
  • Note: Satellites.db is now required to run the script

v10.4.0 - Add zkSync bonus

  • zkSync bonus has been added for nodes using zkSync for payouts for October 2021 and later

v10.3.0 - QA satellite added

  • QA satellite added

v13.4.0 - Uncollected Garbage & Total Unpaid Data

  • Added Uncollected Garbage, which shows an estimate of how much data is stored in blobs, but hasn’t been collected by garbage collection. Uses Last Disk Report and Disk Current Blobs to calculate.
  • Added Total Unpaid Data, which is the sum of Trash and Uncollected Garbage.

v13.3.0 - Disk size last reported by satellites & reported by source

  • Added Disk Size Last Report, which shows a disk size calculated on the most recent disk usage report sent by the satellites. This is the most accurate representation of how much the satellite currently thinks you store and pays you for.
  • Added REPORTED BY column to show which values are calculated using local node data and which are calculated using reports from the satellites.

v13.2.1 - Fix wrong calculation for Trash

  • Fixes a wrong calculation for trash, introduced in v13.2.0

v13.2.0 - Add trash details & replace deprecated utcnow() function

  • Adds additional details for Disk Current, splitting out Trash and Blobs, for easier comparison with satellite reported Disk Average So Far
  • Replaced the now deprecated function utcnow() with functions compatible with the latest python versions

v13.1.0 - Change zkSync bonus & fix forgotten satellites

  • zkSync bonus has been adjusted to 3% to match recent change. (Note this will apply to older months as well)
  • Worked around missing data for satellites which have been forgotten using the forget-satellite function so you can still view data from old months, using fallback names for satellites

v13.0.3 - Work around erroneous storage usage reports

  • Implements workaround for erroneous storage usage reports by satellites to limit impact on storage usage and end of month estimate numbers

v13.0.2 - Fix inconsistent end of month estimation

  • Fixes an inconsistency of how end of month payouts are calculated in the top and bottom overview
  • Improves accuracy of end of month estimated payout for storage

v13.0.1 - Fix issue with vetting status

  • Fix wrong display of vetting status introduced in v13.0.0

v13.0.0 - Refactor for different payout schemes per satellite

  • Calculate weighted average payout per TB for top overview
  • Add payout scheme per satellite
  • Split out payouts for storage, egress and repair/audit
  • Add estimated total by end of month to per satellite overview
  • Change layout to keep output compact with newly data added
  • Source payout data from pricing.db instead of hard coded (note: new db required to use the script)

v10.2.1 - Fix: 0 lines showing for rounding errors

  • Implements a small fix for lines with 0 showing up in case of float rounding errors

v10.2.0 - Non-linear vetting progress

  • Implemented a new algorithm to account for the non-linear progress of vetting, which should give a better indication of how long is left in vetting

v10.1.2 - New storj.io satellite names

  • Updated to use the new satellite names

v10.1.1 - Fix payout previous month

  • Payout previous month didn’t take into account what was paid out in the month you are looking at; this has been corrected

v10.1.0 - zkSync transaction link support

  • Added support for displaying zkSync transaction links
  • Removed warning about stefan-benten payouts being reported as “postponed” as storagenode update v1.24.4 will correct the misreporting
  • Added shebang so the script can be run without explicitly calling python3 on supported systems

v10.0.0 - Detailed payout information

  • Added amounts of actual payouts
  • Added difference between calculated payout and actual payout (helps to identify possible payout issues)
  • Added postponed payout information
  • Added transaction link
  • Several tweaks and additional information related to the above

v9.5.0 - Cleaned up layout

  • Moved more to a single line layout to improve readability
  • Created column groups with descriptive headers
  • Uptime changed to down time to align with 0% = perfect, like the other scores

v9.4.2 - Satellite us2 added

  • Added the us2 satellite name

v9.4.1 - Uptime rounding fix

  • Fixes an issue where uptime percentages were rounded down to a whole percentage

v9.4.0 - Uptime is back

  • Fix for changes to databases with node release v1.12
  • Uptime scores are now included
  • Status display for uptime suspension and under review
  • More verbose warnings display what should be looked at
  • Layout changes for readability

v9.3.1 - Change hours per month to 720

  • Storj has started to use 720 hours per month in calculating payout, this update changes the earnings calculator to use the same

v9.3.0 - Status reworked, suspension added

  • Reworked the status display
  • Suspension scores (unknown audit scores) added
  • Warning when there might be an audit issue

v9.2.2 - Fix time range estimate end of month

  • Fixes the calculation for month progress to use actual number of hours in month instead of 730

v9.2.1 - Minor bug fixes

  • Fix in how percentages are displayed which could cause layout issues on some systems
  • Fixed an issue where the script checked for the wrong filename of heldamount.db leading to the script throwing a non-user friendly error message if this db is not found (Thanks @karaliux for the PR)
  • Fixed an instance where local time was used instead of UTC

v9.2.0 - Additional status information

  • Accurate joined date added
  • Month number calculation decoupled from paystubs; calculation is now based on the accurate join date
  • Current month: Suspension status added
  • Current month: More verbose status display including time of suspension/disqualification
  • Previous months: Payout notes added for nodes that have been disqualified, graceful exit, sanctioned countries and missing 1099 form

v9.1.1 - Bugfix and dependency removed

  • Fixed a bug that caused incorrect calculation of total held amount returned
  • Worked around dependency of statistics module

v9.1.0 - Held amount payouts

  • Held amounts are corrected for payouts in the past
  • Payouts of held back amount in the displayed month are shown and included in totals
  • Fixed a bug where the surge line would sometimes show up in the current month for no reason
  • Small changes to layout to improve readability

v9.0.2 - Fix surge payout prior 2019-09

  • Bugfix: Surge payout was not displayed for months with missing disk usage stats (prior to 2019-09)

v9.0.1 - Fix disk current

  • Previous version introduced a miscalculation of current disk usage. This version fixes it.

v9.0.0 - Surge, held amount, redesign sat breakdown

  • Requires storagenode version 1.3.3 (Data will be incorrect on older versions)
  • Requires heldamount.db, if you copy db’s prior to running the script, this one needs to be included now
  • Added surge payout information
  • Added held amount totals so far
  • Accurate display of month number for your node on each sat
  • Redesigned breakdown per satellite
  • Display only relevant held back % and amount and paid amount
  • Added totals for satellite breakdown

v8.2.1 - Added europe-north-1 satellite

  • Added europe-north-1 satellite

v8.2.0 - Price and month estimate

  • Adds price per TB for download and storage
  • Adds an estimate for total earnings for the current month

v8.1.3 - Terminology

  • Additional correction in terminology

v8.1.2 - Remove uptime score and align terminology

  • Removed uptime scores which always displayed 0 (May return when new uptime system is implemented)
  • Aligned some terminology with Storj documentation

v8.1.1 - Saltlake satellite added

  • Added the new saltlake satellite to the known list to get rid of the -UNKNOWN- satellite

v8.1.0 - First contact

  • Added first contact date to display when the node first dealt with each satellite
  • Made use of additional space to display slightly more descriptive uptime and audit scores
  • Optional note when first contact date is prior to May 2019. As nodes didn’t keep bandwidth history prior to some time in April, dates before May are not reliable. This is only displayed when dates before May 2019 are found.

v8.0.1 - Layout fix

  • Fixed a layout issue when displaying vetting progress on some python versions (Thanks @schnutzie helping me test)

v8.0.0 - Reputation update

  • Added status to show whether your node is OK or DQ (disqualified)
  • Status displays progress of vetting while still in vetting phase
  • Added uptime and audit scores (0-1000)
  • Code cleanup merged (Thanks @smurfix )

v7.0.1 - Cleanup output

  • Remove debug print lines
  • Made the script compatible with the split up sqlite3 databases
  • Changed readme to reflect the 3 databases required for the script to work

v7.0.0 - Compatibility with split database

  • Made the script compatible with the split up sqlite3 databases
  • Changed readme to reflect the 3 databases required for the script to work

v6.1.0 - Historic storage data

  • Storage information for historic months is now also available starting with September 2019
  • Small change in terminology for used storage to align with planned changes on the SNOboard (GBm instead of GB/Month, which was actually mathematically the wrong unit anyway)
  • USD amount is now shown on the Disk Average Month line

v6.0.0 - Big refactor with several updates

  • No more estimates for disk usage, it now uses the nodes own numbers
  • Switched Disk Average to usage so far based on Bh (Bytes*hour)
  • Added Bh line and switched the total line to Bh as well
  • USD calculation for disk usage now doesn’t assume usage will stay the same for the rest of the month and displays USD based on usage until now. Just like for bandwidth only USD made until now is displayed. This results in this version displaying a lower number than the previous version as future income is not counted yet.
  • Known satellites are now mentioned by name
  • Visual tweaks
40 Likes
2

This version runs FAST!

3 Likes
3

Yep, since they introduced bandwidth rollups in the db there is much less data to go through. I wish I could claim that improvement, but that’s all storjlabs. I just changed the script to use the new tables.

5 Likes
4

Thank you for this helpful tool!

Just a thought: it would be even easier if the tool handled itself all the preparation:

  • stop the node
  • copy the info.db in a temporary directory
  • run the calculator algorithm
  • delete the file
  • restart the node

Why do you think?

1 Like
5

To be honest, I think it’s probably safe to run it on a live db on linux. That’s how I’ve been using it for months now. I just felt really bad when people using my script ended up with a corrupted database, so I didn’t want people to take the risk. On windows and macos you really shouldn’t do that though. I could make the script to do everything you mentioned, but it’s kind of trivial to make a script for it yourself and everyone is using different paths and container names.

For linux systems it’d be something like this. This may even work as a powershell script as well, since powershell adopted a lot of the linux commands. So yeah, adapt these lines, put it in a .sh or .ps1 and you’re done. But please test first… as I just wrote this untested.

docker stop -t 300 storagenode
cp /path/to/info.db /path/to/infodbcopy
docker start storagenode
python earnings.py /path/to/infodbcopy
rm /path/to/infodbcopy/info.db
2 Likes
6

Thanks. I got this to work. Very helpful.

Like others I suspect, my actual and projected payouts are much lower than the Storj V3 earnings calculator suggested - even though I have a fast Internet connection. But that’s another discussion for elsewhere.

2 Likes
7

Thank’s for the good work and bring to us a tool that should be included in the dashboard.

There is a reason for start the node after the earnings script? I usualy start the node after the database copy for keep the downtime as low as possibile.

1 Like
8

Starting after copy is fine too. The time difference is minimal though. But I changed it in my previous post.

1 Like
9

I have a nice little update today. I noticed that the info.db actually stores storage data from previous months now. Last month it looked like it was there for one month only, but that likely was just because this way of storing the data was started that month. Long story short, starting with September 2019, historic data is now available, meaning that your historic totals should now reflect actual payouts!

Today’s update incorporates changes to actually display that storage information and adds a warning about the missing information for months prior to September 2019. There are some other tweaks as well. Enjoy!

v6.1.0 - Historic storage data

  • Storage information for historic months is now also available starting with September 2019
  • Small change in terminology for used storage to align with planned changes on the SNOboard (GBm instead of GB/Month, which was actually mathematically the wrong unit anyway)
  • USD amount is now shown on the Disk Average Month line
4 Likes
10

It works like a charm!
Thank you so much for this very usefull tool! It would be great if Storj teams decided to inculde it natively :slight_smile:

1 Like
11

Already suggested that and more on the Node Dashboard UI/UX Feedback topic :smiley:

1 Like
12

Our Storj friends love to throw me curve balls with updates to the database structure for storagenodes. But they’re not getting me that easily! :laughing:

v7.0.0 - Compatibility with split database

  • Made the script compatible with the split up sqlite3 databases
  • Changed readme to reflect the 3 databases required for the script to work
1 Like
13

@Alexey you may want to update the reference to info.db at the bottom of this page as well.


It now requires bandwidth.db, storage_usage.db and piece_spaced_used.db to be copied.

btw… is that a typo in piece_spaced_used.db? That’s actually what the file is called.

2 Likes
14

Love this latest version, it rocks!

1 Like
15

Maybe a stupid question: Why do we have some “NOT PAID” lines?

2 Likes
16

Because ingress traffic isn’t paid and storage is paid only once. I have multiple metrics for storage in there that basically descrive the same thing in different units. The currently stored amount is just what’s stored at this moment. And then there is the average over the entire months and GBh. Since payment is based on the average over the entire months, that line now lists the actual amount.

1 Like
17

I have these 2 lines since version 7.0.0 at the top of the output when I run the script.

<sqlite3.Cursor object at 0x7fe9cea99570>
<sqlite3.Cursor object at 0x7fe9cea99570>

Are these error messages or what else do they mean?

2 Likes
18

No, they’re me being an idiot and leaving in some debug code in the version I put on github. Thanks for pointing it out. Fixed it with a minor release.

v7.0.1 - Cleanup output

  • Remove debug print lines
4 Likes
19

Do the estimated earnings take into account the surge payout (x4 for July, August, September, October)?

Thanks!

1 Like
20

No, This is before surge. The surge payouts are different depending on how old your node is. I get 5x because my node existed prior to surge going into effect. I pondered including it but eventually decided against it since this is only temporary.

2 Likes