Test Data on the Storj Network and Surge Payouts for Storage Node Operators

We appreciate all the feedback and engagement from the storage node operator community. While we’ve been doing a lot of testing as we’ve developed, we’ve only just recently reached production. We’re very excited about what the future holds in terms of customer demand. We’re also really happy with the growth of the storage node network. While the network is really in its infancy today, we expect to see steady growth of the network both in terms of customer demand and capacity.

Since launching into production last month, we’ve had a few community members inquire about our plans around test data, bandwidth utilization, and whether we’d stop uploading test data to the network in the months to come. This post is intended to clarify our test plan and the economics for Storage Node Operators over the next few months.

With last week’s payout, you probably noticed two things:

  1. There was potentially less storage and bandwidth utilization than in prior months
  2. There was an unannounced surge payout of 2.5x

Here’s some clarification on what we’re doing on the Storj side of things to help make sense of what you’re seeing.

Why Do We Test The Network?

We upload and download test data on the network on an ongoing basis for a variety of reasons. These tests include everything from performance and load testing to durability and functional unit testing. The actual amount of data and bandwidth we use during testing far exceeds what we need from a data gathering and analysis perspective. The majority of our test data is used to provide an ongoing incentive and reward for our SNOs to remain on the network as we grow customer demand for cloud storage.

Testing and Satellites

Previously, we used one of our R&D Satellites (satellite.stefan-benten.de) for performance tuning and configuration A/B testing, as well as load and performance testing the network. Recently, we’ve separated the functions out, creating a dedicated Satellite for load and performance testing (saltlake.tardigrade.io), and shifted responsibility for uploading test data to that Satellite.

Ultimately, we will be deleting test data uploaded from satellite.stefan-benten.de and replacing it with data uploaded through saltlake.tardigrade.io, and that data will be cycled regularly. As we cycle through test data, it will continue to be distributed throughout the network, so SNO drives won’t be filled with the same test data indefinitely.

As we transition to using the new performance testing Satellite, we concurrently need to significantly ramp up the amount of test data on the network. We generally try to keep the network sized to anticipate near-term customer demand—that means uploading several PB of test data to the network while reducing the amount of test data on the R&D Satellite. In addition, we try to test egress (download) bandwidth in a way that’s similar to the patterns we expect to see from customers.

What the Storage Node Operator community is seeing in the reduction of data on the network, is the transition as we dial down usage on one Satellite and dial it up on another. One artifact of this transition is that it takes a substantial amount of time to upload several PB of data, and operationalizing those uploads so we can evenly distribute the test egress across the largest number of Storage Nodes.

In terms of the amount of download traffic we generate during testing, we’re targeting a similar usage pattern to what we expect from early customers with early use cases - mainy backup and other similar low frequency retrieval use cases. Our current target is to download approximately 10% of the data stored each month. As we get more customer data on the network, we’ll continue to tweak our test protocols to match customer behavior.

Surge Payouts

We don’t have anywhere near the amount of test data we want on the network today, so we included a surge payment in the March payout as a “thank you” to our loyal Storage Node Operators who’ve been diligently providing storage and bandwidth to date. Until we reach the planned levels of test data storage and bandwidth, we’ll continue to use surge pricing this way, dialing down the surge amount as we increase network utilization. The most important things in this process include both generating valuable test telemetry and feedback as well as ensuring that we continue to reward our loyal storage node operators.

As we’ve mentioned before, we use surge payouts much like a ride-sharing service might. During certain times when Uber or Lyft need more drivers in specific areas (the supply side of their network) they offer a surge payout for drivers to incentivize them to get out and pick up riders. On the Storj Network, we reserve the ability to use surge payouts as a tool to compensate Storage Node Operators (the supply side of our network) who provide their excess capacity and bandwidth. It’s worth noting that on our network, Storj Labs pays the surge payout cost, whereas, on a ride-sharing service, it’s passed on to the rider.

For example, if we foresaw a large amount of incoming data from one of our customers, we might send out a notification that we’d implement surge pricing in the coming month to incentivize more Storage Node Operators to join the network. This past month’s surge payout was to incentivize Node Operators to maintain their Storage Nodes on the network.
We continue to make progress on increasing the demand for the network and look forward to announcing our customers as they’re ready to tell their stories.

Overall, our goal is to make sure the network grows in a healthy way and a big part of that is making sure it is rewarding to be a storage node operator. With the paint barely dry on the platform from our production launch, we have a long road ahead of us. We hope our storage node operators currently on the network remain a part of this platform as we grow. We’re doing what we can to make storage node operation economically viable today, as we build the customer demand that will drive storage node utilization in the future.

45 Likes

Thank you for the clarification. I believe communication here is key and the frequency of communication can control the number of threads on the forum.

It’s now clear that we can’t expect more than 10% of egress on the data stored on SNO nodes, if no other traffic comes into the region the SNO operates. In my case I’m seeing 20% egress traffic, and the ingress is growing with several hundred GBs per day.

5 Likes

Thank you for a thorough post. It’s very well written and summarizes the Storj stance in a clear way! Please allow me to comment on one thing, though.

TBH, I would personally prefer to not do busywork like that. If you want to incentivize SNOs, surge payments are IMHO the way to go. If you want to ensure reserve capacity or you do actually need testing data for testing, then it’s fine. But if you don’t need that much testing data for actual testing or capacity planning, then it feels wrong to just eat resources to have an excuse to pay more—not just from, let say, environmental point of view, but also that this traffic competes with legit customer traffic (even if with the current numbers the effect should be imperceptible).

So I hope that what you mean by the quoted sentence is that your goal is to have reserve capacity ready for, let say, short-term growth.

3 Likes

Hi @john

Could you please tell SNO community more information about the current status of tests?
Also, SNO community would like to know how long upload test traffic will be present?
Many SNO now have out of space, and lack of information about what’s going on and can’t make decisions about what we should do next.
Could you please inform us about the current status and test plan?

8 Likes

Why would that info be of any use? Test data is not different from customer data from the SNO point of view. If you configured your storage node properly, your hard disk should not run out of space. Only your node should get filled. And isn’t this the goal? I don’t understand what I would do differently if I know whether it’s test data or customer data.

8 Likes

This information is required for capacity planning.
Out of space - mean for example 7.9TB is used of 8TB on. SNO not have unlimited drive space :smiley:

If you not require this information it doesn’t mean that other SNO not require it too.

1 Like

Yes, but what is the difference for you knowing whether it’s test or consumer data? You shouldn’t buy new hardware for STORJ anyway…

You spoke for “the SNO community” :stuck_out_tongue_winking_eye: And judging by the likes of our 2 posts, I’m currently in the lead :stuck_out_tongue_winking_eye:

1 Like

For example: we got information that test data will be deleted at the end of this month or test data will be never deleted, are you fell the difference? :smiley:

You shouldn’t buy new hardware for STORJ anyway…

As I said before:

lack of information about what’s going on and can’t make decisions about what we should do next.

It help me and other SNO make OWN decision and anybody can’t recommend me what I should do with my infrastructure and my money :smiley:

Yes, absolutely, because I persist in one of the big SNO community groups, and we discussing and sharing experiences and plans with each other, and I write this post like parliamentary of this group (It not only my question).
I don’t care how many likes your post have, everyone can like anything. I care only about initial questions :wink:
If this information not required and not helpful for you - just ignore it :slightly_smiling_face:

3 Likes

I’m with @Odmin here. I ordered another HDD to expand my node so I don’t run out of space. If the plan is that the test sat is filled to 4TB after which 3TB from the old satellite would be remove, I really didn’t need to do that upgrade yet. Now, I’m buying this HDD with my storj earnings (literally, found a store that accepts bitcoin, traded my storj tokens in and bought it). So I don’t mind perhaps expanding before I absolutely had to. But it would be nice to have some idea on whether this investment was worth it. Preferably before purchase next time. :slight_smile:

3 Likes

“Hey SNOs, I’m Tardigrade customer. I want to backup my three servers today, please, be prepare for that!”
Do you expect something like that from customers too?

In the reality I’ll just push the button

1 Like

I agree with Odmin it would be nice if 90% of the space used wasnt just testing data though, I expanded my node as well I just dont want that node to be filled with testing data to be full before any customers actually store data on it. Because we still all have Stephan satellite data stored on all our nodes and now were getting alot more testing data.

Testing data is also used to pay snos while there is not enough real data.
So if you are worried about having full nodes, just blacklist saltlake on a new node :smiley: Then that node will be exclusively for customer data.

So only saltlake is used for test data and no other satellite ?

I don’t get this. Data is data and traffic is traffic. If Storj wants my node to essentially “dig a hole and fill it up” so it does not sit idle, but pays for it, why should I care?

5 Likes

Because its only ingress and unless you have unlimited storage your not getting paid much. Were supposed to be in production now and my nodes are all full of testing data that is ingress only and stored data that may not ever get downloaded again. Id rather have no data then have my nodes full of just testing data.

I expect, that Storj Team shares information about the current test plan (I hope they have a test plan) and the current status of testing. We all know, that saltlake satellite is using for tests and we see huge activity but no information about how long this test will do and how much data will be uploaded/downloaded (no any information on what exactly Storj Team is testing now?).

I not have any complaints about tardigrade production satellites or customers. I just asking about the current status and future plans for testing.

Stefan-benten satellite is currently generating constant egress on my node. Saltlake will probably generate egress as well at some point.

The “customer data” satellites generate much less traffic than the “test” satellites. If I did not have any data from the test satellites, I’d get much less money and my node would be mostly empty with less than 2TB stored.

This is my earnings.py result for March

Payout and escrow by satellite:
SATELLITE       FIRST CONTACT   TYPE      MONTH 1-3       MONTH 4-6       MONTH 7-9       MONTH 10+
us-central-1    2019-04-05*     Payout   0.3338 USD      0.6675 USD      1.0013 USD      1.3350 USD
                                Escrow   1.0013 USD      0.6675 USD      0.3338 USD      0.0000 USD

europe-west-1   2019-06-01      Payout   0.8125 USD      1.6249 USD      2.4374 USD      3.2499 USD
                                Escrow   2.4374 USD      1.6249 USD      0.8125 USD      0.0000 USD

asia-east-1     2019-06-04      Payout   0.1413 USD      0.2827 USD      0.4240 USD      0.5654 USD
                                Escrow   0.4240 USD      0.2827 USD      0.1413 USD      0.0000 USD

saltlake        2020-02-11      Payout   1.8789 USD      3.7579 USD      5.6368 USD      7.5158 USD
                                Escrow   5.6368 USD      3.7579 USD      1.8789 USD      0.0000 USD

stefan-benten   2019-04-05*     Payout   4.1924 USD      8.3847 USD     12.5771 USD     16.7694 USD
                                Escrow  12.5771 USD      8.3847 USD      4.1924 USD      0.0000 USD

As you can see, most of the money was from the test satellites.

3 Likes

No I get that but what you have to understand is if my nodes are all full from test data there’s no room for actual customer data, I expanded my nodes to hopefully get some customer data but it still seems like the testing data is still coming in super strong. I mean yes its nice to see traffic and yes most of my egress is from stefans sat as well, But this is much more incoming data then I ever expected.

Its kinda funny to complain about getting too much data though…either way im in it for the long haul. But one has to think why are we all getting so much data now is it cause the storj network is saturated with data and most of the nodes are getting full though?

1 Like

It is to be expected that storj will remove data as soon as the network needs more free capacity.

2 Likes