Actually, there are two. Ford and Dwight D. Eisenhower so it appears you didn’t get that right either.
https://www.reuters.com/world/us-aircraft-carriers-what-they-bring-middle-east-2023-10-15/
3 Likes
This is absolutely correct. We have situations in Australia where one side of the street may get fast internet and the other side may be stuck on fixed wireless. It sucks.
2 Likes
Well, playing devil’s advocate here…. It would be nice for us to know but why do you think you’re entitled to?
2 Likes
i don’t know? i only noticed the problem of whole point to even share those information in general, if it in fact, carries no information, just to cast some unnecessary shadows of, some node, somehere, sometime, causing confusion and pointless followups like this one … but glad You acKNOWLEDGEd the devil.
Well, if you don’t need to know and knowing makes really no difference to any of us… just let it rest, man 
1 Like
i do not know that, maybe it would make a difference, if i would know it. And since that ifnromation been placed, now i do want to know too, what nodes are consider large, how often STORJ ask those large nodes, mayby i want to be a part of those invited to discusion too! i will buy some HDDs just to have a voice maybe! 
1 Like
Hahahaha!
I think you and I would have to spend many times our yearly earnings in HDDs to make it to the “league of big boys”…. 
1 Like
Hello storage node community,
We really appreciate the lively discussion the pricing announcement yesterday has engendered. We’re especially appreciative of the replies that understand what we’re trying to achieve in the long term, but we also understand that the new pricing simply doesn’t work for some storage node operators. In addition to the forum discussions, we’ve been meeting with node operators (both public network operators as well as commercial node program participants) on an ongoing basis to get feedback and share perspectives.
We wanted to share a bit more on two fronts: i) the data and details that led to our decisions, and 2) what options and steps are available to node operators.
One challenge the conversation has is a bit of asymmetrical data knowledge. While we don’t share everything about our business expenses and revenue, we do aim to be transparent (see any of our recent town halls). Above all, we aim to be reasonable and responsive, and have had multiple discussions with many storage node operators in advance of the pricing announcement we made yesterday. In addition to the already public information about the costs and scale associated with running the network, we are sharing additional data below that will provide context to the pricing announcement.
First, market conditions, and our economic situation have driven our payout rate adjustments. Storj doesn’t operate in a vacuum. While market prices for cloud storage have not changed dramatically in recent years, the market price for bandwidth and egress has dropped significantly. Wasabi, Backblaze, and Cloudflare have all introduced pricing models where a significant amount of egress is free. And, the large hyperscalers and CDNs are dropping prices as well. In order to stay competitive, we need to be able to drop prices too.
Furthermore, payouts needed to be adjusted to accommodate our current and growing customer base. As you may have gathered from our recent town halls, Storj is attracting larger and larger customers. This is a good thing overall for the health of the network. But, as you can imagine, large customers demand both significant discounts and significant guarantees around performance and security. So, large customers are generally not paying the quoted list prices.
Additionally, we are increasingly working with channel partners to sell our product. This is also a good thing overall – it means that there are more people selling Storj and offering a complete solution. However, the channel partners also need to make money to offset their costs. So, they take a portion of the list price, leaving less for Storj to use to cover expenses.
Not the least, we value our team members and need to compensate them fairly for the work that they do building and maintaining a great network. Our team members work full time to write the code, sell and market Storj, and keep the network running. They need to be paid, and paid at rates that reflect current economic reality.
Finally, we strive to give back to the communities where our customers and team members live and work, and to show by example how a company can contribute for the greater good. As noted above, we do provide a good deal of free or reduced priced data for non-profit organizations.
The bottom line for all of this is that – to maintain a healthy network, we need to adjust payouts. We did so after significant deliberation and input from the node community. We are responding to market signals and the needs of multiple stakeholders, storage node operators included. We understand that, for node operators in some markets and circumstances, this means that it may no longer be profitable to operate nodes, to provide large amounts of bandwidth, or to add new nodes.
Storj Network Data
We also want to share some internal data to show how our payout costs have gone up.
For instance, here are the totals for the amount we have spent on storage node payments since mid 2022. Taking into account minimum payment threshold, we paid the following real dollar amounts to storage node operators, listed by the month of service earnings. All of these are verifiable on the Ethereum, Polygon, zkSync Lite, and zkSync Era chains.
| Month | Amount |
|---|---|
| 2022-01 | $49,580.83 |
| 2022-02 | $158,818.23 |
| 2022-03 | $78,281.42 |
| 2022-04 | $182,561.60 |
| 2022-05 | $50,529.21 |
| 2022-06 | $129,058.82 |
| 2022-07 | $121,382.26 |
| 2022-08 | $105,435.62 |
| 2022-09 | $135,788.54 |
| 2022-10 | $141,526.23 |
| 2022-11 | $135,372.38 |
| 2022-12 | $149,968.03 |
| 2023-01 | $145,930.23 |
| 2023-02 | $156,195.96 |
| 2023-03 | $185,784.37 |
| 2023-04 | $138,213.39 |
| 2023-05 | $173,766.86 |
| 2023-06 | $144,336.49 |
| 2023-07 | $142,058.61 |
| 2023-08 | $136,346.83 |
| 2023-09 | $106,183.86 |
| Average | $131,767.61 |
As you can see, the average payout since the beginning of 2022 has been around $130K per month. This is the total amount we are targeting per month to ensure there is sufficient network utilization to make it economically viable to run a storage node (as first announced here), considering the size of the network we are trying to target. In advance of customer usage, we store synthetic data and use synthetic egress to maintain that level of payout. We plan to continue to use these tools to keep this average payout rate near these levels until the growth of customer data eliminates the need for the synthetic load.
There has been some discussion regarding our costs in this thread. For a previous discussion on costs you can check out this post. But if we analyze our costs just for storing data (not including bandwidth) if a customer uploads 1TB of data to our network, we don’t only store 1TB. Because there is a risk some nodes could leave or fail, we actually store closer to 2.7 TB (80/29 = 2.758…). What this means is that for each TB of data we store, we pay node operators $1.50 x 2.7 = $4.05. Our current list price per TB is $4 so we are currently paying SNOs over 100% of our revenue for data stored. Then, in addition to the SNO costs, we have costs for running the satellites as well as customer acquisition and customer support. Originally we had anticipated changing the payout rate for storage to between $0.75 - $1.00, but after listening to feedback from the community (including here, here, and here) we determined that it was better for the network to keep the storage payout rate stable, even though it meant going above our desired $1.00, and instead work on becoming profitable by more aggressively doing other things including lowering the Reed Solomon expansion factor (still to be done), lower satellite costs (work has begun, see post regarding shutdown of EU North and US 2, but more work to be done), and bringing down our heavily subsidized egress payouts.
On the egress side of things, again originally we had targeted bringing the SNO costs somewhere in the range of $1.50 - $5.00 per TB. With this current announcement we are bringing that down to $2.00 and don’t anticipate further changes in the foreseeable future. There was a lot of analysis and work that went into determining the right balance on this incentive. While competitors like Wasabi and Backblaze have raised prices, both offer some level of free egress. We frequently need to price more aggressively or bundle some egress to win against those competitors.
Our new egress pricing is admittedly on the lower end of the above range in part because of the decision to keep storage payout high, but also due to the fact that we still have significant costs for our edge services and, as noted, customer acquisition and customer support costs that need to be covered by the $5 per TB difference between the $7 we charge customers and the $2 we will be paying SNOs. With the revenue we receive from customers, we also need to cover all of our other operating and engineering expenses to run the company and improve the product.
While we aim to be supportive of our public network and want to pay rates that are sustainable, it is worth pointing out the broader environment we are in. Other projects that are similar to Storj in structure but with much larger networks, i.e. Filecoin and Chia, are currently paying lower rates for data stored with zero remuneration for egress. Those other projects have significantly higher requirements for more expensive to operate hardware than is required for a storage node in our network. In light of that, we don’t believe that the rates that will take effect in December are unreasonable in this market. We provide these data points to share where the market for storage seems to be going (which is evidenced by the continued growth of Storj nodes).
Steps available to node operators
If you are one of the many node operators who we have either talked to directly, or has expressed support for what we are doing in the forum or elsewhere, thank you! We see you!
If the new prices don’t work for you, you have three options available to you:
- Abrupt halt of your node for any length of time, which is destructive to our overall network health (see below for details).
- Graceful Exit, which has recently been improved and revamped and does not have the challenges past graceful exit incarnations had. Please see the details here.
- Continue to engage in dialogue via the forum or by reaching out to us directly.
The first choice is fairly destructive to our overall network health. While the incentive design and rules of the network are optimized to encourage nodes to choose the second choice, a Graceful Exit, we have designed our network to handle abrupt network exit by failing nodes. Due to the war in Eastern Europe we have already adjusted our durability characteristics such that we could lose a substantial number of nodes and not lose a single object. However, this type of durability does result in higher amounts of repair and extra pieces, which reduces the amount of egress per piece and thus the total per TB earnings for node operators. If a large-scale abrupt exit were to happen, remaining nodes will get much more traffic, and nodes that we allow to return will come back to find less data. To underscore that last point, nodes and node operators that break our terms of service by engaging in intentional disruption of the network are explicitly subject to direct disqualification.
On the other hand, the second option is helpful to the network overall, as it provides us a strong signal, with a decent lead time. We see the number of nodes that are queued for Graceful Exit, which allows us to keep the network healthy while having a dialogue with you. This is also the only way to recover your held amount. It’s worth pointing out that Graceful Exit was designed with node operators in mind, both so that we could keep the expansion factor lower (thus increasing the ratio of egress per piece), and so that node operators had control of the (potentially) strong signals they send to satellite operators, without jeopardizing the network altogether.
As you know, we recently refactored Graceful Exit. Thanks to this new implementation, we can now offer a support-ticket driven ability to cancel Graceful Exit. If you have started a Graceful Exit and then later decide you want to remain on the network, during the next few months you can file a support ticket to have this Graceful Exit canceled. Note that this will only affect data that has yet to be transferred off your node, and all data that gets transferred off before you cancel will remain transferred. We will evaluate if adding general support for this is feasible for the long term for functionality like reducing the amount of data any storage node shares with the network. But most importantly, a partial Graceful Exit will not risk disqualification, like abrupt disconnect would.
Summary
With this new announcement, we still are passing through almost all storage margin to storage node operators, and are still paying reasonable egress rates we have carefully vetted against the market. We want to reiterate that we avoid frequent rate changes. Our previous set of price changes to egress was nearly four years ago. And, based on our read of the market, we don’t anticipate needing to reduce prices further in the foreseeable future.
Again, we want to thank you all for your patience and understanding as we go through this effort to bring our business and the network into the best possible health.
The Storj Team
17 Likes
This is not something I would use to brag.
Could you please also give us the amount of money you spent monthly on “others”?
But I agree on everything else you said. STORJ is to expensive compared to the competition the december pay rates seem fair to me.
3 Likes
As you can see, according Storj token balances and flows report, the “other” expenses several times higher than the net network operations.
Just a 5-10% saving in that “other” expenses would allow to maintain the SNO rates.
Storj Labs have any mitigation plan designed for a scenary without token sale reserves?? What will be the Storage Node Operators rates in 6 months?
My opinion is that they are selling STORJ from token sale done by Storj Labs (British Virgin Islands) to fund the private company Storj Labs Inc. When the reserves are gone, they will migrate to a regular company offering datacenter storage with encryption and redundancy in a “global network” done by commercial nodes, without SNOs at all.
Check the ingress per day of all your nodes, is it simmilar? 99% “synthetic”
6 Likes
I don’t get one thing… Why do we, as SNOs, have to support the burden of free or reduced prices for non-profit organisations, Ukraine and other sponsorships that Storj Inc decides to do whitout asking for our opinion? Nobody gives me free stuff, and I don’t want to give free stuff to nobody. If some SNOs want to sponsor some unselfish causes, organisations etc. than there should be an option to give some free space/bandwith for that, or an wallet for donnations. Not to impose it to the entire network. Because even though there is no direct link between payouts to SNOs and these free gems, there is the obvious cause and effect that in the end conduct to payout cuts (2 in one year) for all SNOs. Storj is burdened with small revenues? Stop the damn free gems that you are giving to others. Nobody asks you to participate in noble causes.
Second: just increase the storage price to 5$. Who wants to use Storj, would not mind to pay one extra dollar per TB used. Each part of the business should support itself: the revenue per stored TB should cover the SNOs payouts and bring a proffit; the egress revenue should cover the SNOs payouts and bring a proffit. Not one of them to bring proffit for both parts.
All these are pretty easily and obvious solutions to be implemented.
Third: reducing the expansion factor will decrease the network security and will reduce the ingress for all SNOs, ingress that is already pretty low. This should be compensated by aquiring new huge clients with more data to be stored, but that’s a slow process.
This is my one and only reply.
6 Likes
Everything in the mathematics described above is correct, and the desire to turn a profit of $7-$6=1$ into $7-$2=$5 is also understandable, this is a business and it should make a profit.
But I still have a question. If we talk not about relative figures and the general economic model, but about absolute costs, at the moment we are talking about total savings of 12% * $130k = $15,600, which is clearly not a significant figure for the company (this is about 1-2 salaries it- employee). Both in the texts of the announcements starting in the spring, and in your letter, you talk about the future, about the situation when this 5 $/TB will be the locomotive that pays for all the company’s expenses, but now this is obviously not the case, since the entire total “profit” of the company fits into $15k. At the same time, the real costs of the company are obviously much higher, which means the tariffs are being reduced not to save $15k in itself, but for the sake of a pure economic model (perhaps for investors or other similar purposes)?
This was a view from the economic side of the company, but from the SNO side (from a financial point of view) there are only two parameters - the total $/TB per month and the % of its nodes filled. For many years, this parameter fluctuated around $3.3/TB and nodes were constantly growing, but now, in a few months, it has fallen to $1.6-1.7/TB and growth has slowed down noticeably, primarily on old nodes. And here you have a more flexible mechanism than tariffs, this is synthetic traffic, synthetic data, and free space on the nodes.
Right now, on the SNO side, two things are happening:
- Two satellites were deleted, one of which took up a lot of space (up to several TB on old nodes), that is, a lot of free (unpaid) space was created.
- Now there is a lot of incoming traffic, and it would seem that the nodes should be better filled, but no - approximately the same volume is immediately sent to the trash. The result is a heavy load on the equipment with a complete lack of volume growth. I think that in this way you “replace” the synthetic data with new client data, and this is a reliable method, but from the point of view of SNO, this option is only normal on a completely full node, and on a node with empty space it only causes resentment.
It seems to me that if you changed this point at the level of the node operation algorithm (add synthetic data, or simply do not delete it until the node is filled), so that the SNO could see the growth of its brainchild, there would be much less disturbance, and payments would grow a little without damage to the economic model and tariffs. And now it turns out that you simultaneously made a one-time reduction in volume and income, and reduced tariffs, and also the nodes are engaged in the (“useless” from the point of view of SNO) work of pumping data “from the satellite to the trash.”
Thank you for your attention, perhaps you can come up with a mechanism for “non-tariff” support for your SNO partners.
P.S. In business terms, I see no reason not to move the payment for the test satellite/traffic/data from part of “COGS” to part “R&D, QA etc”, and at the expense of this part, increase the payment for SNO until the network reaches a normal operating level. Essentially, the entire “synthetic” part of the network is, by definition, not part of the product being sold.
1 Like
I’ve actually registered on this forum for the first time ever just to leave an opinion regarding this subject, despite the fact I’ve been a node operator for quite some time (a work in progress, with ups and downs).
I’ve seen, as well, both announcements and the strike proposal for November 11th (which I won’t enter, btw).
In order to make things easier I’ll simply write my thoughts on this with some bulletpoints (and invite everyone to answer this, or consider this, on one or the other end of the spectrum):
-
I do understand the reason why the proposed payout is being subject to a change. And, truth being said, I also understand the economics that support it. That being said, I personally think this is the wrong way to go, rate-wise.
-
When we decide to put to service an HDD to Storj, we decide to do it under several different aspects, including predicted durability of the hard drive, expected payouts, predictability of the load, size of the networks, and we sustain a whole structure of ours to deal with requirements, including having a good enough Internet (and that is not a guarantee on some parts of the planet!), having a PC turned on 24/7 (with all incurred electricity costs), and etc.
-
I leave outside of it all this being the third time a payout change is announced, since it was expected. But the rates in this regard are meager, very very meager, and combining with the 75/50/25 current structure of held amount (which does what’s intended for), this creates a very discouraging scenario not only for further network operators join Storj, but as well for manteinance of all current ones. I’m not even talking on the reduction from 10$ to 2$ on egress/repair, I’m talking on the final paid monthly amount, which I believe it’s what we’re all here for, regardless of being here for 1 day or 5 years.
-
Storj is currently at a decisive moment. With Sia just around the corner with a product that will technically be as good as Storj from the get-go for node operators, some structures being developed, and with free fluctuation and payments for all nodes, letting several aspects be held by the operators themselves like storage pricing and ingress/egress, and evaluating several aspects that Storj doesn’t do currently like uptime (that has to be at least 60% to avoid suspension/disqualification), speed (that Storj does not consider when having a service, although that is absolutely decisive for an eventual customer as they surely don’t want lower speeds than the big boys), available capacity, and letting a SNO actually choose its main angle of operation under that market, there is a significant risk that graceful exits or sudden disconnections happen, medium to long term, affecting Storj growth.
Yes, I know still isn’t there customer-wise, but it will be - having rates manually selected by node operators will be good enough for many customers themselves, even though they would have to pay ingress/egress on some of them. And customers go from one to the other without any problem - no contractual obligation abides them to you, as Storj, or any other similar project. It could take 6 months or 6 years, but if a wave like that happens, how will Storj defend itself as a project with this centralized, market-maker pricing? It can’t. Just can’t.
-
I don’t even believe Filecoin and Chia are in the same ballpark as Storj; Filecoin is directed at big markets with substantial configurations, mostly China-related, and rates (afaik) are actually higher than Storj (they should be, considering their requirements), all things considered including the bonuses they give, and actually run much closer to your current commercial services for datacenters; Chia is not even remotely the same kind of product as Storj, as it is a mining-a-coin service, not a rent-your-HDD service, and no customer services are provided, it’s simply a question of supply and demand. The economics behind Chia are so bad that I can’t even believe there are people investing in 480TBs worth of HDDs for that, but here we are. As such,
-
Let’s face it: people do not like having to pay to download stuff. Never did, for the last 30 years, from pirated stuff to legal, simply-held-online stuff. They simply aren’t used to it, never will be. The current market makers know that, and offer close-to-unlimited, in-practice-unlimited or truly unlimited ingress, or egress, or both. Storj must be competitive price-wise, but HDDs and SNOs must also be taken in consideration, and I believe a balance is possible, with some adjustments, guaranteeing a lower cost, assuring that those who are trustable and can be trusted stay on the network, and defending yourself against the current market stance.
-
In that regard, I frankly believe, considering your current customers (which you’ve outlined so well on the second post), that this would be the best solution for SNOs:
» paying for egress/ingress should be abolished at all; considering current and future market conditions, it seems impossible to keep doing medium-term, as that’s not the trend and I frankly believe it won’t be.
» in predictability grounds, pricing per held TB should be increased, to somewhere between 2.25$ and 2.50$, also to fare for depreciation of HDDs and servicing/manteinance of the SNO;
» uptime should be taken into consideration on pricing, as it is already on data, to assure network health - for example (it’s just an example, could be any uptime higher than that!), a reasonable 93.33% uptime minimum before penalties would mean that, over a 30-day period, a SNO should be able to have a maximum of 48 hours of downtime, which is perfectly feasible to anyone to take care over a 24h regular day, allows to sort things out in an event something’s closed for the day or it happens overnight, and allows up to two times of failure before payouts start to be affected; uptimes at the end of the month under that minimum would exponentially lose their amounts, from like 50% cut for 90% uptime to 100% cut for 87% uptime and lower. If someone disagreed, it would be simply a question of doing better the next month.
-
Repair could either be 0$ or 1$ for example (depending on storage pricing and Storj’s will to retain this last item on their pricing structure), and it would be an obligation of the SNO.
-
No bandwidth could be limited under the minimum required; if so, payouts would be fully cut (it’s actually easy to implement, quite a simple mathematical comparison with held GB/TBs, uptime [potential storage] and final amount [real held storage] and its deviation). Additionally, further speed improvements would be totally on the SNO responsibility. In a world where the additional cost of providing 10 Gbps, comparing to 1 Gbps, is just 5%, it makes no sense to limit a bandwidth dedicated to a service that provides storage…
-
Improving actual storage held, an additional satellite could be opened for customers in Africa/South America, since you are experiencing demand on that area, improving the matters after closing the US2/European-North satellites). It’s actually feasible to implement if you implement it in Brazil, that has the best connections of all local countries afaik, on one of the biggest cities (Fortaleza - state of Ceará would be preferential). That would contribute to a relevant growth of the Storj network.
-
STORJ coin could be required to be held as collateral to new entrants, with a fixed amount per TB, effectively rate-limiting new operators and constraining space availability from the get-go, and acting as an incentive to add more storage, should the network need it at any time.
-
The Reed-Solomon could be adjusted accordingly with uptime requirements to further improve matters, from 2.7 to 2.0, and that could be an additional penalty factor for earnings.
This would be a gamechanger. I assure you, at a 2.25$ rate, we would be looking at 11.25$ per 5TB for example - and that, in my book, is well paid, regardless of ingress/egress, uptime requirements and external pricing. Storj could as well adjust their market pricing and it would still be different comparing to other services like Sia, Filecoin and Signum, having their space both on the SNO world and on the market itself.
Just an idea.
7 Likes
I have two internet connections - my main one has 600mbps upload, but my backup connection only has 10mbps. If my main connection goes down - should I put the node on the backup connection with the low upload or leave it offline? Right now the average egress from my node is less than 10mbps by the way.
In general I agree with you. The rates would not be so bad if the usage went up. However, the rates go down, usage goes down (two satellites removed) and now usage stays almost constant while the rates are going to go down even more.
My node has almost 9TB of free space. I also have some 8TB hard drives in a box, I could easily add even more space. Nobody needs it though. Same for egress. At this point the egress may as well be unpaid, since last month I got $1.7 for it with the new rates it would be $0.6, well, whatever.
5 Likes
I have the following problem with your payout figures.
-
The majority of payments are still on L1
-
The substantial egress we had in August - paid out in September did not appear to change your payouts.
Instead of looking only at raw payout levels you should be looking at the change in payout commitments to SNO’s. In other words in your calculation you should be including amounts owed but not yet paid. Particularly since the majority of payments still go out on L1.
(Getting SNO’s to change to zksync has to be considered a monumental failure so far - but lets see how era goes in the future.)
I would also be interested to know the number of wallets tied to currently active nodes that had payouts owing and to see if that number was trending up or down over time.
I wouldn’t get my hopes too high. At Storj we have someone at the helm, we have company structures, and that is a good thing. Sia might be great for some idealist preaching 100% decentralisation, but the complexity that brings and no structures behind is a huge deterrent for someone having to store petabytes of data effortlessly and with some guarantees.
Migrations sometimes take days (5 or even 10 when you are moving a big node), or you might be out of town for a week or two with limited remote access or no remote hands. There also might be a problem with supply of spares as waiting 3 or 4 weeks for a server component is a norm recently. Thus I don’t think such a strict requirement would be a good idea.
We are supposed to use what we have. Sometimes I do need that bandwidth. Also, all these companies have FUPs in their contracts and sometimes you have no option but to somewhat limit the bandwidth.
This might be true for home Internet, in that case that FUP rule applies. You also won’t really get a 10Gbps as that access medium is in most cases shared with at least 15 other people. You can go dedicated to truly have 10Gbps just for yourself, that however is 50 times as expensive as home Internet around here. And believe it or not, even top tier datacenters here bill the traffic that leaves the country.
3 Likes
Payouts are still too high. Should be $1 or even much lower for storage, possibly $0.25 per TB per month is the market rate.
Egress numbers seem very reasonable to me
Edit i have 0.5PB doing almost nothing and would take your arm off for $500 a month
Edit and yet after 4 years storj only uses less than 5% of it
4 Likes
A hard drive uses about 10W of power, which, corresponds to 7.2kWh/month. at @0.2/kWh it adds up to $1.44/month. So, anything less than a full 6TB node does not even pay for the power of the hard drive, much less the CPU.
Although, I may be OK with those rates, assuming my node grew to at least 100TB. But 20TB*$0.25 would be $5/month - really not worth it.
4 Likes
it will take several years to grow that big, I am here about 4 years and have only 290TB for now
And as I understand, it’s not just one node? If it’s not a secret, how many nodes do you need to set up to fill 290TB?