Test Data on the Storj Network and Surge Payouts for Storage Node Operators

I do have an update (Note I tend to use all the deadlines).

For context, I’ve posted a couple of updates and mentioned this in the Town Hall about how we use test data on the network. This post should give you some more visibility into how we use test data and what you can expect over the next few months.

What are we doing with test data?

Text data serves a number of different purposes, some of which may be more obvious than others. The main uses for test data are:

  • Storage Node Incentive - probably the reason you care about most, all test data is paid as if it were real customer data. We don’t distinguish between real and test data since we’re using storage space and bandwidth either way. Until we have PBs of customer data on the network, we want to make sure the early storage nodes earn money as we grow demand.
  • Use Case Validation - as we pursue different opportunities with partners and customers, we use test data to simulate actual customer data and use cases. We do this so we can be confident that the Tardigrade platform will meet the customer’s need and also to simplify onboarding
  • Load Testing - as we test the scalability of the network and make improvements, it’s useful to push high volumes of data through the network to determine the limits of the different network components
  • Performance Testing - we’re continuously testing the network to evaluate the performance, particularly relative to customer expectations for different use cases
  • Functional Testing - test data helps us assess the different functions beyond the client use for upload, and download, including file repair, audit, other durability-related components
  • Competitive Benchmarking - we test performance against other cloud providers to compare performance on some standard functions. The most common benchmark test is put and get on a 10 MB file.
  • App Testing - as we, or partners and customers develop different connectors or app integrations, we do significant testing to determine how those apps will behave in the wild
  • Incentivizing demand - We’ll be using test data to continue to support supply in advance of forecasted customer demand.

How much test data do we plan to push to the network?

The amount of test data we push to the network varies, but there a couple of different factors that impact the amount of data and bandwidth we’re using for testing. Those factors include:

  • Test priority - Depending on where we are in a release cycle, we may want to test different aspects of the platform. When we’re testing concurrency, vs. streaming performance for uploads and downloads, deletes, repairs, etc. the needs change and the test patterns change.
  • Customer and partner POCs - testing larger use cases or proving value to specific customers/partners
  • Coordinating initiatives - spinning up new satellites or testing new features, code or use cases

Over the past few months, we’ve been testing a wide range of different factors on the platform, including significant amounts of egress bandwidth. Over the next few months, we’ll be shifting our test patterns to more closely simulate real customer use cases. While priorities change from time to time, in general, we expect that our test profile will change over the next few months in terms of how much storage and bandwidth we use for testing.

We have a few PB of test data on the network now, but we expect that to grow to around 8 PB (pre expansion factor) over the next few months. In terms of download traffic, we’ve used a higher rate of downloads as a percentage of data stored over the last few months, but we expect that to average out to somewhere around 10% of data stored over the next few months.

While the Salt Lake satellite will be focused on load and performance testing, the other satellites (including our new European satellite) wil be focused on customer growth and testing customer-oriented use cases, including projects like GitBackup and some POCs with prospective customers and partners.

The test data and bandwidth planned over the next few months may increase or decrease depending on what we need at any particular time. One thing we plan on continuing to do is use surge payouts to maintain a relatively stable level of network payout. We know that significant swings in the incentive program for storage nodes has a big impact on the community.

It will take some time to get 8 PB loaded on the network, but we’ve been ramping up the load process across all satellites. More importantly, we’re working even harder on getting real (paid) customer data on the network. A number of long time supporters have published posts that we need time to build a successful cloud storage business on top of this network and with the reception we got in the media, with partners and customers at launch, it’s going to be a wild ride… We will continue to be more transparent and clear when it comes to expectations, especially for new SNOs.

20 Likes

@heunland - thanks for requesting an additional clarification. One thing we should be clear about is that test data is dynamic. It’s intended to mirror real usage so it produces valuable and actionable telemetry. As we move into the post-production era, we’ll also be cycling test data more - both setting a TTL for test data and deleting and refreshing test data. What that means for SNOs is that as we move forward, you’re much less likely to have a drive fill up with data that takes up space that may prevent new data with potentially greater egress from being stored.

We are deliberately evolving our test data strategy, but we have specific objectives we’re also trying to achieve. You’ll see an accelerating ramp up of data, and as we get the volume of data we want to maintain, then we’ll begin cycling that data. We’ve heard the concern that storage node operators worry that their node will be filled with test data early and then that test data will potentially persist indefinitely. That’s not how we intend to proceed. In the near term, you can expect us to load up around 8 PB of data, which, with the expansion factor, will occupy around 22 PB of network capacity. Once we achieve the target network test data volume, we’ll use a combination of TTL and deletes to cycle test data around the network. Egress will be generally fairly broadly distributed.

11 Likes

This is interesting. 22PB distributed over 6000 nodes would mean about 3.5TB per node, though I guess a lot of those nodes are smaller than 3.5TB (or slow) because my node has about 9.5TB stored. So I guess after you load the all 22PB, my node may have more than 16TB (depending on how much you have loaded already). That’s cool to know.

Thank you for the detailed explanation.

Thanks a lot for provided detailed specifications of the test plan!

This information precisely that SNO community is looking for, now SNO can make a decision and also make plans on how to deal with own infrastructure and how to grow in the future. Transparency is very important for us like an opensource code is preferred than proprietary.
Thank you again for this information, I personally really appreciate it.

1 Like

Thanks for the update! I would not expect stefan’s test data to be deleted until 8PB target is reached

Thanks a lot for sharing this! It seems I made the right choice expanding my node (16TB now). I appreciate the effort you are putting in keeping SNOs happy with a fairly constant income.

Just to make sure you’re aware. There are some nuances in play right now. Some SNOs are pretty happy this month and will recognize what you’re saying about higher download percentages. (I’m happy to be among those) But newer nodes don’t have any data from the stefanbenten satellite that’s doing most of the downloading right now and are actually having a bad month.
I think a lot of SNOs would like to know when that downloaded traffic would move to saltlake and when data from stefanbenten would be deleted.

That said, thanks a lot for the transparency. It’s really appreciated!

@john
I have just learned that vetting halts once a node is full. I have asked that here: Do full nodes get vetted?

Could you confirm that. And maybe explain? Despite the fact that test data is getting paid I don’t see any sense to fill up a node with test data blocking it to get vetted for customer data.
I’d like to see either an option to keep being vetted even if the node is full or an option to limit test data so there remains required space for being vetted on the other satellites.

2 Likes

To get vettet your node has to pass audits. To pass audits your node has to get audits. To get audits your node has to have data.

If your node is full and a new satellite comes online. Because your node is full the satellite cannot put any data on your node. No data → no audits → no vetting.

But does it matter? If the node is full, you get paid for it and Storj is planning to download some of that test data to give you egress (which is also paid). Unless you are hoping to get a file from a customer who is going to access the file 5 times per second (extremely unlikely), just keep the test data.

Or expand your node :slight_smile: Or set up a second (third etc) node.

1 Like

I see. So audit data is no “special” kind of data from the satellites, it is based on real upload data. That makes it even sadder if real customer data cannot placed on a node as it is blocked with test data.

Yes. First of all the network exists for customer data not for holding test data. And if data gets deleted from your node or you get disqualified on one satellite it is much better if you are in the position to receive new traffic from other satellites immediately than having to wait up to 30 days to get vetted on other satellites.

Expansion does not really make sense when satellite data cannot be restricted as with current rate of appr. 200 GB test data flowing in it would require at least 6TB to cover 30 days and it is no guarantee that you have been vetted on the other satellites by then.

It would be much better if there was a way to limit the space for test data or to assign space to satellites.

We can complain about this all we want, the question is, how can that problem be solved?
do you see a way around it?
If so, please let us know.
I currently can’t think of a way for a node to get data for audits although being full

However, it has been said that test data will receive deletes and uploads so theoretically there should be data from the new satellite too, although chances might be significantly lower.

Yes, as long as your node has at least one piece from a satellite, it will be vetted eventually.

I like seeing my node full and even more like seeing a lot of egress. If Storj wants to waste their own money by paying me for storing test data, so be it. It’s much better than an empty node.
Why would Storj do that? Well one reason would be to guarantee that there is enough space available (for example if a new client wants to store 1PB of data, Storj can just delete 1PB of test data).

This is actually interesting - assuming there will be non-Tardigrade satellites in the future I may want to distribute the space.

My node exists to hold the customer data. That customer is whoever runs the satellite. Whether my customer stores his own data on my node or resells the space to someone else does not really matter to me as long as I get paid.

But it’s awesome, judging from the Storj posts, the test data will continue to come in for some time. This is great. I already picked what new drives I will get to expand my node once the space runs out.

2 Likes

Not from a technical perspective. It would require some kind of limitation how much a satellite can upload to keep vetting going for the other satellites. For the specific test satellite for SNO it might be helpful to limit the amount of space to occupy. So let’s say if I have a 2TB node I could set 1.5TB to be available also for test data but the rest exclusively for other satellites.
A temporary upload block could also help that SNO can lift when audits from the other satellites flow in. The test satellite is a bit special in a way as it is pushing soooo much data onto a node. I am getting 200GB per day currently and it is simple math to tell when the node will be full. Just full of test data.

Is it possible to estimate how much data must be present on a node for being vetted? Would it be sufficient theoretically if there is a single bit of data from a satellite present to get vetted?

1 Like

Yes, if your node has a single piece from the satellite, it will get vetted eventually, though it may take a longer time.

But if this is the case, wouldn’t be a solution easily be possible? Node knows which satellites exist, node knows from which satellites data is stored and node knows how much space is left. So when a node fills up, for the last remaining 5GB or something, he could report “space full” to all satellites he has already data from and “space available” to only those satellites that are missing or he carries only very few data. Maybe even SNO could set the threshold (vetting space).

With this a SNO could make sure that his node keeps receiving data but also that he receives data from all satellites so he keeps getting vetted on all satellites.

And even if a new satellite pops up, node would recognize that data is missing for this satellite. As soon as space gets available e.g. if customer deletes a few GBs, the node would reserve the vetting space for the new satellite and thus make sure, that the node has a chance to get vetted on the new satellite even if it is full.

How does this sound?

3 Likes

This looks interesting to me - reserving some space for new satellites.

Reversing space does not seem a good idea to me, if you have been running a node for 9+ months you will get 75% of the revenue if a new satellite comes along you will only get 25% of the revenue in the first month. Assuming you have a full or nearly full node what is the incentive to want to get vetted on the new one when your currently stored data is worth more with the older satellite?

See payout structure at https://storj.io/blog/2019/01/sharing-storage-space-for-fun-and-profit

3 Likes

Thinking about it:

  1. If SNO could set the threshold, then it is totally up to him. If he sets it to 0 no space will be reserved.
  2. Space will only be reserved if node does not hold already data from all satellites. If enough data is present, no need to reserve space.
  3. Reserved space might only be small share. If you run a 2 TB node and 5 GBs are reserved space, I don’t see it would hurt your earnings very much.

It is a fallback strategy. Vetting is expensive. It takes time up to 30 days while you receive only 5% of potential traffic from a satellite. As SNO has no control over the data he never can be sure the data he has today he still has tomorrow. If a customer decides to delete data it can be gone any time. If you get disqualified on a satellite, it is gone. In that case you will be happy if you can replace the lost data as quickly as possible with data from other satellites. Also when you expand your node. And the best position for that is if you have been vetted.
So basically the incentive is to be ready to receive data at 100% rate from any satellite at all times.

One thing I’m not sure about and might help the situation is …

How much of the “test” data is there for node Incentive only? and how quickly can it be removed from the network if required?

So when a node is getting full, to say 85 - 95%, It sends an update to the testing satellites who then start to remove some of the “incentive data”, freeing up room for customer data?

1 Like

Looks like stefan-benten satellite has started deleting data:

2 Likes

Mine too…

How low will the baseline go ?

I’m hoping it’s low enough to shuffle my drive space… been wanting to add some new services.