# Updates on Test Data

**URL:** <https://forum.storj.io/t/updates-on-test-data/26034>\
**Category:** Announcements\
**Created:** [April 30, 2024, 3:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034 "2024-04-30T15:36:45Z")\
**Posts on this page:** 20\
**Page:** 92

<div class="post-metadata">

**Author:** ![Knowledge](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/knowledge/32/10681_2.png) [@Knowledge](https://forum.storj.io/u/Knowledge)\
**Post date:** [August 16, 2024, 6:25pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2019 "2024-08-16T18:25:30Z")

</div>

It depends. That data is meant as reserved capacity. If customers are starting to utilize some of that capacity, it will be less on SLC and more on the other Sats. But this is all fluid, as it depends on customer trends. We want the space to be available to the customers and not used up because we are holding it on SLC. So, if a customer is ready to utilize a chunk of that data we have to have it available and thus it gets released. The exact timing of when a customer uses that data isnt up to us. We are just making sure the capacity is there.

And, of course the SLC data could go up if more reserved capacity is asked for. And it can go down as it gets released to be used. Thus, it is fluid and such definitive pins on how much capacity will be in one place or another is not guaranteed.

---

<div class="post-metadata">

**Author:** ![EricTheRed123](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ericthered123/32/1124_2.png) [@EricTheRed123](https://forum.storj.io/u/EricTheRed123)\
**Post date:** [August 16, 2024, 6:49pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2020 "2024-08-16T18:49:38Z")

</div>

> [@littleskunk](#):
>
> Testing is done. We will switch back to SLC now.
> 
> Now to the test results. We can see that the maximum throughput without surge nodes has increased by about 20% in less than a month. We don’t know for sure how much load these customers are going to upload. It is more like a range. The good news is we can cover this range with no surge nodes for the most part. Only on the higher end of that range we would have to mix in surge nodes. And the second good news is even if we go nuts on the possible range and take the most unlikely scenario into account we can make that possible with a few more surge nodes.
> 
> This has been an interesting journey. We started with an almost impossible target and a few releases later with a lot of code changes it turns into a simple math problem that is easy to solve for any customer that needs it. Now we just need to wait to find out if this plan works out. Sorry I still don’t have an ETA or any more details for you. All I can say for now is that we are ready for these customers. And thank you all for your patience and the additional capacity you have added.

When you say “throughput”, are we talking IOPS or bandwidth? I’m wondering whether I’ll need more HD space, bandwidth or IOPS in the future.

---

<div class="post-metadata">

**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/6_449c533f586939b6e5f246632a46421f.png) [@Roxor](https://forum.storj.io/u/Roxor)\
**Post date:** [August 16, 2024, 11:42pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2021 "2024-08-16T23:42:12Z")

</div>

Replying to myself, as the math is confusing me. If [the targets](https://forum.storj.io/t/updates-on-test-data/26034/2018) add up to 30PB of capacity-reservation… then to maintain it SLC would have to be uploading 1PB/day steady-state to keep up with 30-day-TTL data expiring… wouldn’t it?

If a 1Gbps Internet connection can move around 10TB/day… then SLC would have to fill a 100Gbps connection 24x7. I can’t imagine how much Storj is paying for that class of bandwidth.

Or did I add it up wrong? If targets are quarterly… wouldn’t it makes sense to have 90-day TTL… so the required upload rate would be 1/3rd? (You can still force-delete TTL files if you need the space back sooner).

(Edit: Also, I [want a pic](https://forum.storj.io/t/post-pictures-of-your-storagenode-rig-s/1422) of SLC’s upload hardware: it must be _beefy_!)

(Edit #2: Is this worse than I think… because to have space to hold 30PB of _customer_ files… you’d have to upload around 2.7x that amount to the network? So 2.7PB SLC uploads/day?)

(Edit #3: I guess I don’t need to know these answers. I’m content if I see my HDD fill a bit more every month… 🤑 )

---

<div class="post-metadata">

**Author:** ![Aleksman4o](https://storj.bcdn.literatehosting.com/letter_avatar/aleksman4o/32/6_449c533f586939b6e5f246632a46421f.png) [@Aleksman4o](https://forum.storj.io/u/Aleksman4o)\
**Post date:** [August 18, 2024, 1:44am UTC](https://forum.storj.io/t/updates-on-test-data/26034/2022 "2024-08-18T01:44:19Z")

</div>

> [@Roxor](#):
>
> wouldn’t it makes sense to have 90-day TTL

I dont think they can say to clients: Hello, you wanna 10Pb of storage? Ok, it will be available to you after 3 months, because of our test TTL )

---

<div class="post-metadata">

**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\
**Post date:** [August 18, 2024, 3:33am UTC](https://forum.storj.io/t/updates-on-test-data/26034/2023 "2024-08-18T03:33:44Z")

</div>

I think it is still possible to delete data without waiting for TTL to expire.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [August 18, 2024, 3:51am UTC](https://forum.storj.io/t/updates-on-test-data/26034/2024 "2024-08-18T03:51:53Z")

</div>

Yes, of course. It will be collected by the garbage collector, it should take about two weeks.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [August 18, 2024, 3:58am UTC](https://forum.storj.io/t/updates-on-test-data/26034/2025 "2024-08-18T03:58:36Z")

</div>

> [@Roxor](#):
>
> If targets are quarterly… wouldn’t it makes sense to have 90-day TTL… so the required upload rate would be 1/3rd?

There are three points:

1. Test the throughput
2. Keep space used
3. The test data should be autoremoved without a garbage collector and waiting in the trash

Thus 90 days is not better, because it cannot help to test a throughput, if the node would be full. Also these customers have a similar usage pattern as we emulating with Saltlake, include segments size and levels of TTLs (of course, this is an estimate based on the information we have about how they plan to use the Storj network). Therefore, it is important to conduct the most accurate testing possible.

And it’s already allowed us to optimize our code to have more high throughput than before, it’s also helped to reveal several bugs and fix some of them.

---

<div class="post-metadata">

**Author:** ![EricTheRed123](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ericthered123/32/1124_2.png) [@EricTheRed123](https://forum.storj.io/u/EricTheRed123)\
**Post date:** [August 23, 2024, 2:57pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2026 "2024-08-23T14:57:17Z")

</div>

Hello. It’s been almost 24 hours since test data ingress paused. I can see that SLC data stored is slowly going down.  
Any updates?

---

<div class="post-metadata">

**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\
**Post date:** [August 23, 2024, 3:01pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2027 "2024-08-23T15:01:59Z")

</div>

As of right now I don’t have an update. My hope is on Monday I can provide an update.

---

<div class="post-metadata">

**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/6_449c533f586939b6e5f246632a46421f.png) [@Roxor](https://forum.storj.io/u/Roxor)\
**Post date:** [August 23, 2024, 3:17pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2028 "2024-08-23T15:17:14Z")

</div>

> [@EricTheRed123](#):
>
> It’s been almost 24 hours since test data ingress paused. I can see that SLC data stored is slowly going down

That’s actually a good thing: It means TTL deletions are working! 😜

[Some just don’t get that space back](https://forum.storj.io/t/debugging-space-usage-discrepancies/24874)… but help [is on the way](https://forum.storj.io/t/when-will-uncollected-garbage-be-deleted/27327/294)!

---

<div class="post-metadata">

**Author:** ![EasyRhino](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/easyrhino/32/1133_2.png) [@EasyRhino](https://forum.storj.io/u/EasyRhino)\
**Post date:** [August 23, 2024, 3:24pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2029 "2024-08-23T15:24:47Z")

</div>

Not gonna lie, it’s more exciting when the test data is coming in fast and filling up drives and breaking things and causing errors.

---

<div class="post-metadata">

**Author:** ![EricTheRed123](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ericthered123/32/1124_2.png) [@EricTheRed123](https://forum.storj.io/u/EricTheRed123)\
**Post date:** [August 23, 2024, 3:37pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2030 "2024-08-23T15:37:05Z")

</div>

> [@littleskunk](#):
>
> As of right now I don’t have an update. My hope is on Monday I can provide an update.

Thank you. This is all I need. My main goal was to make sure it was not just my nodes on pause. 🙂

---

<div class="post-metadata">

**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/6_449c533f586939b6e5f246632a46421f.png) [@Roxor](https://forum.storj.io/u/Roxor)\
**Post date:** [August 23, 2024, 8:33pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2031 "2024-08-23T20:33:04Z")

</div>

> [@EricTheRed123](#):
>
> My main goal was to make sure it was not just my nodes on pause.

Taking a look at [one of the big SNO](http://www.th3van.dk/)s: it looks like they lose 15-20TB _per day_ when SLC isn’t firing out data. Imagine emptying an entire HDD every day, yikes! 😮

---

<div class="post-metadata">

**Author:** ![mikee027](https://storj.bcdn.literatehosting.com/letter_avatar/mikee027/32/6_449c533f586939b6e5f246632a46421f.png) [@mikee027](https://forum.storj.io/u/mikee027)\
**Post date:** [August 24, 2024, 12:27pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2032 "2024-08-24T12:27:26Z")

</div>

When the ingress drops 97% suddenly I always think theres a problem with my node

---

<div class="post-metadata">

**Author:** ![ACarneiro](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/acarneiro/32/1129_2.png) [@ACarneiro](https://forum.storj.io/u/ACarneiro)\
**Post date:** [August 24, 2024, 12:41pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2033 "2024-08-24T12:41:20Z")

</div>

The good thing about having multiple nodes in multiple countries is that when ingress drops in all of them at the same time you’re a bit more relaxed… 🙂

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [August 24, 2024, 1:20pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2034 "2024-08-24T13:20:06Z")

</div>

Do not panic. This is a usual everyday in the distributed storage with an unpredictable usage…  
Just imagine - our storage is used by real humans. Do you still expect a predictable results? Even for a day…

---

<div class="post-metadata">

**Author:** ![alpharabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alpharabbit/32/1129_2.png) [@alpharabbit](https://forum.storj.io/u/alpharabbit)\
**Post date:** [August 24, 2024, 3:44pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2035 "2024-08-24T15:44:41Z")

</div>

> [@mikee027](#):
>
> When the ingress drops 97% suddenly I always think theres a problem with my node

None of my nodes showing such large drop, it is more like 30-40%. 🤔

---

<div class="post-metadata">

**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/6_449c533f586939b6e5f246632a46421f.png) [@Vadim](https://forum.storj.io/u/Vadim)\
**Post date:** [August 24, 2024, 4:28pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/2036 "2024-08-24T16:28:40Z")

</div>

Looks like BF cleaning unpaid garbage, so nothing special.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [August 25, 2024, 3:08am UTC](https://forum.storj.io/t/updates-on-test-data/26034/2037 "2024-08-25T03:08:51Z")

</div>

By the way, did you check the usage on the dashboard? Does it match the usage reported by the OS? Your node may believe that’s full, if the databases are not updated with the actual usage.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [August 25, 2024, 3:11am UTC](https://forum.storj.io/t/updates-on-test-data/26034/2038 "2024-08-25T03:11:12Z")

</div>

I do not think that they are talked about the used space, they are talking about the ingress.  
Yes, the GC would move data to the trash, but it would not increase a free space (to make the node to be not full again). Only the TTL collector and trash-filewalker can do it.

[Previous page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=91)

[Next page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=93)
