# Updates on Test Data

**URL:** <https://forum.storj.io/t/updates-on-test-data/26034>\
**Category:** Announcements\
**Created:** [April 30, 2024, 3:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034 "2024-04-30T15:36:45Z")\
**Posts on this page:** 20\
**Page:** 35

<div class="post-metadata">

**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vadim](https://forum.storj.io/u/Vadim)\
**Post date:** [June 5, 2024, 3:58am UTC](https://forum.storj.io/t/updates-on-test-data/26034/771 "2024-06-05T03:58:24Z")

</div>

for some reason my cpu usage not consistent across nodes.  
some working on 1-2% cpu and getting traffic 20-30mbit other taking 13% of cpu and get only 13mbit so i really not understand why it so big difference. RAM usage is normal.  
 ![Screenshot 2024-06-05 065924](https://storj-s3.bcdn.literatehosting.net/original/3X/a/e/ae55630f65ef29d65ffb62e2d599701369a956a3.png)

---

<div class="post-metadata">

**Author:** ![MarviBiene](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/marvibiene/32/1124_2.png) [@MarviBiene](https://forum.storj.io/u/MarviBiene)\
**Post date:** [June 5, 2024, 4:39am UTC](https://forum.storj.io/t/updates-on-test-data/26034/772 "2024-06-05T04:39:03Z")

</div>

I love the utilization on my 1 Gbit bandwidth. But I spotted, that my disks are too slow. Time for more disks.

 ![1000052305](https://storj-s3.bcdn.literatehosting.net/original/3X/c/5/c527fb6c9db4c289e50bfe9b934486c43d1dda19.png)

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [June 5, 2024, 5:29am UTC](https://forum.storj.io/t/updates-on-test-data/26034/773 "2024-06-05T05:29:45Z")

</div>

Probably difference of disks? I especially see CPU and RAM usage on the nodes with SMR disks.

 ![1000023865](https://storj-s3.bcdn.literatehosting.net/original/3X/d/4/d43c062c02af6e46f631d8348cb53512b63ac8ee.jpeg)  
I’m not near saturating my bandwidth…

 ![1000023867](https://storj-s3.bcdn.literatehosting.net/original/3X/f/9/f976d0f89312437694bd3cc3a6b3c3c41d792a5c.jpeg)  
But now already almost same ingress as yesterday, with still 18h to go. So I actually like current settings.

Also never seen an increase of 1TB in 5 days.

---

<div class="post-metadata">

**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/5_5575768a8748004e209b776fc1b2916d.png) [@snorkel](https://forum.storj.io/u/snorkel)\
**Post date:** [June 5, 2024, 6:38am UTC](https://forum.storj.io/t/updates-on-test-data/26034/774 "2024-06-05T06:38:01Z")

</div>

Just spread the load…

![IMG_6618](https://storj-s3.bcdn.literatehosting.net/original/3X/d/c/dccc23f8b73df72d55ecfe6ff0df01146325e013.jpeg)

---

<div class="post-metadata">

**Author:** ![peter\_linder](https://storj.bcdn.literatehosting.com/letter_avatar/peter_linder/32/5_5575768a8748004e209b776fc1b2916d.png) [@peter\_linder](https://forum.storj.io/u/peter_linder)\
**Post date:** [June 5, 2024, 7:39am UTC](https://forum.storj.io/t/updates-on-test-data/26034/775 "2024-06-05T07:39:23Z")

</div>

My aggregated bandwidth across all my nodes is now over 3gbit/s total since about midnight UTC+2. Never saw that before during your testing.

There is a large difference between my “good” nodes and my “not so good” nodes (scraped together from whatever was sitting on the shelf since 10+ years with varying performance)

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [June 5, 2024, 7:46am UTC](https://forum.storj.io/t/updates-on-test-data/26034/776 "2024-06-05T07:46:45Z")

</div>

Bandwidth is kind of not important. How much of it turns back to you in increased risk space?

I mean, of 1TB ingress only 650GB yesterday really turned back to me in disk space. Over the days it’s about 250GB per TB (also deduction of trash included).

---

<div class="post-metadata">

**Author:** ![flo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/flo/32/64_2.png) [@flo](https://forum.storj.io/u/flo)\
**Post date:** [June 5, 2024, 7:53am UTC](https://forum.storj.io/t/updates-on-test-data/26034/777 "2024-06-05T07:53:21Z")

</div>

> [@littleskunk](#):
>
> will run it over night on a lower total throughput.

You do realize that running this test impacts your customers?  
My company noticed a significant performance drop, both when storing as well as retrieving data.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 5, 2024, 7:53am UTC](https://forum.storj.io/t/updates-on-test-data/26034/778 "2024-06-05T07:53:44Z")

</div>

This depends on the initial size of the file. Perhaps it was not enough small to be stored as an inline segment.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 5, 2024, 8:00am UTC](https://forum.storj.io/t/updates-on-test-data/26034/779 "2024-06-05T08:00:37Z")

</div>

> [@MarviBiene](#):
>
> the VPN from portmap is the problem.

You spotted it correctly. You are limited by their bandwidth. You may try to use a different VPN provider with a port forwarding, but I believe that it wouldn’t be free unfortunately.  
It would be better to contact your ISP to get a public IP.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 5, 2024, 8:07am UTC](https://forum.storj.io/t/updates-on-test-data/26034/780 "2024-06-05T08:07:32Z")

</div>

Perhaps depends on the used model of the disk?

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 5, 2024, 8:10am UTC](https://forum.storj.io/t/updates-on-test-data/26034/781 "2024-06-05T08:10:17Z")

</div>

I shared your feedback with the team. We didn’t expect, that it may affect customers, sorry about this!  
Could you please file a support ticket to check, why is it happen?

---

<div class="post-metadata">

**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\
**Post date:** [June 5, 2024, 11:08am UTC](https://forum.storj.io/t/updates-on-test-data/26034/783 "2024-06-05T11:08:50Z")

</div>

I’m surprised a customer noticed the testing: it looked like the network had so much spare capacity they wouldn’t notice. But I guess the last 12h have really been pushing it!

I suppose that will also affect their TTL duration decision, for capacity-reservation? Like maybe 1-month TTL would mean they’d have to sustain too-high uploads and could affect paying customers… but a 3-month TTL would allow them to upload slow-and-steady and not be noticed?

---

<div class="post-metadata">

**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\
**Post date:** [June 5, 2024, 11:10am UTC](https://forum.storj.io/t/updates-on-test-data/26034/784 "2024-06-05T11:10:08Z")

</div>

This is interesting:

 ![graph_image-2](https://storj-s3.bcdn.literatehosting.net/original/3X/2/b/2b2c24b66c561025c36f01b038c08aa95d7c35d3.png)  
 ![graph_image-2](https://storj-s3.bcdn.literatehosting.net/original/3X/e/0/e02adc369478394509a8b431d6f005b17dd2c778.png)

 ![image](https://storj-s3.bcdn.literatehosting.net/original/3X/3/1/31a07fee1f3753d128117e23fe19d56888bf3a1f.png)

Some kind of process runs every hour (a type of filewalker probably, looks like the node is deleting a lot of expired pieces at the beginning of each hour), and as it runs, it increases the IO load, lowering success score for a bit. Then the satellite drops my node from the list for a few minutes until probably some kind of timeout expires.  
The success rate does not seem to drop that much though.

> [@Roxor](#):
>
> but a 3-month TTL would allow them to upload slow-and-steady and not be noticed?

They are testing performance, so they probably periodically load the network as much as possible.

---

<div class="post-metadata">

**Author:** ![Mitsos](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/mitsos/32/1125_2.png) [@Mitsos](https://forum.storj.io/u/Mitsos)\
**Post date:** [June 5, 2024, 11:16am UTC](https://forum.storj.io/t/updates-on-test-data/26034/785 "2024-06-05T11:16:46Z")

</div>

From a network guy’s perspective, even if storj keeps pumping test data, I do see a reason for it to affect performance. Disk heads need to be moved to write the test data, and they need to be moved to read customer data. All this while the node ACKs the incoming data, eating into its upload capacity (network overhead + processing).

---

<div class="post-metadata">

**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\
**Post date:** [June 5, 2024, 11:17am UTC](https://forum.storj.io/t/updates-on-test-data/26034/786 "2024-06-05T11:17:39Z")

</div>

> [@Pentium100](#):
>
> lowering success score for a bit. Then the satellite drops my node from the list for a few minutes until probably some kind of timeout expires.

The current success tracker is really good in dialing down the request rate the moment your node is failling a few uploads in a row. This happens so fast that you will have a hard time to measure the success rate on your side. It will still look like almost 100%. Best indicator now is the upload rate hitting your node.

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [June 5, 2024, 11:18am UTC](https://forum.storj.io/t/updates-on-test-data/26034/787 "2024-06-05T11:18:19Z")

</div>

> [@littleskunk](#):
>
> Best indicator now is the upload rate hitting your node.

And compare it to what?

---

<div class="post-metadata">

**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\
**Post date:** [June 5, 2024, 11:28am UTC](https://forum.storj.io/t/updates-on-test-data/26034/788 "2024-06-05T11:28:02Z")

</div>

> [@Toyoo](#):
>
> And compare it to what?

Maybe just to your same node at any other point in time?

Like even if data sizes vary you can still count “upload started” vs. “uploaded” and “upload canceled” log entries to see rate of race wins/losses over 5min or something. I think what littleskunk is saying is that when your node gets slow ingress it may show 99/100 wins… and during performance testing it may see 990/1000 wins. Either way you’re winning 99%…

… and the reason you don’t see 990/10000 (9.9% win) is that the system _backed off the ingress rate_ as soon as it saw you starting to lose a few more races… so it never ramped up from 1000 to 10000. It’s trying not to overwhelm nodes.

(at least that’s my guess 🤔)

---

<div class="post-metadata">

**Author:** ![Th3Van](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/th3van/32/66_2.png) [@Th3Van](https://forum.storj.io/u/Th3Van)\
**Post date:** [June 5, 2024, 11:31am UTC](https://forum.storj.io/t/updates-on-test-data/26034/789 "2024-06-05T11:31:40Z")

</div>

![image](https://storj-s3.bcdn.literatehosting.net/original/3X/1/a/1ada9cef7318ea9bd4434bda68e9e52b44e35928.png)

![image](https://storj-s3.bcdn.literatehosting.net/original/3X/b/6/b661e2aa4875f245bbedf67155e104b9d8ea39cc.png)

```auto
Single node :
tail -f server002-v1.104.5-30002.log |grep --line-buffered "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | grep --line-buffered 'upload started' | pv --line-mode --average-rate > /dev/null
[39,9 /s]

All nodes :
tail -f server* |grep --line-buffered "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE" | grep --line-buffered 'upload started' | pv --line-mode --average-rate > /dev/null
[3,11k/s]

```

Th3Van.dk

---

<div class="post-metadata">

**Author:** ![BrightSilence](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/brightsilence/32/6048_2.png) [@BrightSilence](https://forum.storj.io/u/BrightSilence)\
**Post date:** [June 5, 2024, 11:31am UTC](https://forum.storj.io/t/updates-on-test-data/26034/790 "2024-06-05T11:31:57Z")

</div>

I think @Toyoo’s point was that success rate used to be a decent measure to see if your node had issues. Now if there is little traffic, we don’t know if it’s just quiet on the network or our nodes are having issues.

---

<div class="post-metadata">

**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/5_5575768a8748004e209b776fc1b2916d.png) [@Roxor](https://forum.storj.io/u/Roxor)\
**Post date:** [June 5, 2024, 11:35am UTC](https://forum.storj.io/t/updates-on-test-data/26034/791 "2024-06-05T11:35:38Z")

</div>

> [@Th3Van](#):
>
> ![image](https://storj-s3.bcdn.literatehosting.net/original/3X/3/3/33815631a93769c4fdbdadca80522ea0bc7ff366.png)

Holy crap that’s a lot of ones and zeroes! You’ll make up for losing like half of your stored data in no time! 😉

(but seriously man: I check your report every once in awhile: what happened to your earnings rate was brutal 😢 )

[Previous page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=34)

[Next page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=36)