# Updates on Test Data **URL:** \ **Category:** Announcements\ **Created:** [April 30, 2024, 3:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034 "2024-04-30T15:36:45Z")\ **Posts on this page:** 20\ **Page:** 22
**Author:** ![thelastspark](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/thelastspark/32/1132_2.png) [@thelastspark](https://forum.storj.io/u/thelastspark)\ **Post date:** [May 28, 2024, 4:14pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/499 "2024-05-28T16:14:55Z")
Just wanted to chime in and express my appreciation for the open communication (and excited to hear the results of further tests/onboarding grapevine) ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 30, 2024, 2:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/500 "2024-05-30T14:36:31Z")
Satellite are all running the new choice of 2 node selection. (I used the wrong term “power of 2” earlier. Lets try to call it choice of 2 from now on.) Time to find out the maximum throughput we can reach now. So this time there will be a higher load again. ---
**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\ **Post date:** [May 30, 2024, 2:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/501 "2024-05-30T14:36:45Z")
I’ve added some logging to my nodes and now I see that pieces uploaded with TTL=30 days (I presume: test traffic) have a fixed order size of 2319360 bytes, even if the piece itself is 2048 bytes. This is worrying, as this would make traffic accounting in 1.104.5 nodes way off. This is exactly the part of my patch that I wanted to investigate before making a pull request. At least the patch was reverted later, so up to date nodes will again have the right numbers. ---
**Author:** ![Roxor](https://storj.bcdn.literatehosting.com/letter_avatar/roxor/32/6_22734026db12803ebb3e059622ac3534.png) [@Roxor](https://forum.storj.io/u/Roxor)\ **Post date:** [May 30, 2024, 3:16pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/502 "2024-05-30T15:16:41Z")
> [@littleskunk](#): > > Lets try to call it choice of 2 from now on. I have [a better name](https://www.youtube.com/watch?v=pmRAiUPdRjk) for the new node-selection system. ---
**Author:** ![jtolio](https://storj.bcdn.literatehosting.com/letter_avatar/jtolio/32/6_22734026db12803ebb3e059622ac3534.png) [@jtolio](https://forum.storj.io/u/jtolio)\ **Post date:** [May 30, 2024, 4:24pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/503 "2024-05-30T16:24:32Z")
> [@Toyoo](#): > > Indeed. If I understand correctly, now satellite penalizes you for each lost race with less traffic, regardless of origin of that traffic. So if you operate a node in, let say, South America, if you fail too many connections from Europe, you will get not just less uploads from Europe, but also from South America as well. Just wanted to reply to this - this is only kind of true. It is true that the Satellite is now going to keep track of recent success rates in a system-wide data structure (a speed profile, I guess), but we actually have many of these speed profiles to choose from. We are going to be choosing which speed profile to use based on where the request is coming from. Right now, this functionality isn’t enabled, so yes, your description is true /this week/. Our next step is to have every region’s S3 gateway (where most of our traffic comes from anyway) get their own speed profile (once I can figure out why [https://review.dev.storj.io/c/storj/edge/+/13309](https://review.dev.storj.io/c/storj/edge/+/13309) isn’t passing the build). And then after every gateway region is handled, then we’ll explore what we can do for remaining native integrations. ---
**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/6_22734026db12803ebb3e059622ac3534.png) [@snorkel](https://forum.storj.io/u/snorkel)\ **Post date:** [May 30, 2024, 4:33pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/504 "2024-05-30T16:33:43Z")
> [@littleskunk](#): > > So this time there will be a higher load again. (…with all the risk of beeing banned…) That’s what she said! 🕶 ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [May 30, 2024, 6:25pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/505 "2024-05-30T18:25:19Z")
So I’m watching this and the binary storagenode process, during these tests, is doing a CPU load of over 700% with IOwait% being pretty much insignificant. This is on x86 CPU with plenty of RAM and SSD cache and with logs on ramdisk. It looks like the process is quite CPU heavy, I assume the more the smaller the segments gets. Are there some plans to optimise the storagenode code more, maybe doing binaries for different, more advanced CPU flags if such thing would help? As with such high CPU loads and PPS, even the ping to the local gateway is quite erratic and the node is loosing significant amount of races. I can probably throw more cores at it, but that probably isn’t an option for someone running this on a low powered device or for someone running this at scale. I wonder how Synos and similar devices are coping with this load. ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 30, 2024, 6:51pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/506 "2024-05-30T18:51:36Z")
> [@zip](#): > > Are there some plans to optimise the storagenode code more, maybe doing binaries for different, more advanced CPU flags if such thing would help? We can control the piece size by changing the RS numbers. Last test that ended right now should have been twice the piece size and halve the number of connections. According to our result that increased performance. This isn’t a solution. It is just an observation for now. ---
**Author:** ![Ruskiem](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ruskiem/32/64_2.png) [@Ruskiem](https://forum.storj.io/u/Ruskiem)\ **Post date:** [May 30, 2024, 7:43pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/507 "2024-05-30T19:43:54Z")
i can confirm. Windows GUI. Win 10 pro. 2 cores per node and 1h ago 80-100% CPU usage. Those are 2 cores of a 8/16 CPU (AMD Zen 2, AM4 platform) delegated per VM instance, If someone going to say: Oh its VM fault - i’d say wonder what would happen if that wasn’t in VM. A potential CPU usage leak would eat me all the CPU usage of a whole workstation. With VM, at least i can limit cores per node like that. now (1 node, 1 HDD, 2 CPU cores) Edit: Normally 1 VM storj instance uses 20-40% CPU load, even with quite high traffic like now (ingres now is 30-45% network, that is high, past months normal was like 3-10%) That is 2 core per VM. The high CPU % occurs only when network was tested up to 100% its ability, like yesterday, so not something that occurs often, but rather rare, but still. ---
**Author:** ![ACarneiro](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/acarneiro/32/1129_2.png) [@ACarneiro](https://forum.storj.io/u/ACarneiro)\ **Post date:** [May 30, 2024, 8:36pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/508 "2024-05-30T20:36:07Z")
That’s interesting. Mine had a peak of bandwidth use but CPU and IOWait didn’t really go very high. I wonder if my nodes were selected less often with the new choice of 2 selection criteria… ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/6_22734026db12803ebb3e059622ac3534.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [May 31, 2024, 3:28am UTC](https://forum.storj.io/t/updates-on-test-data/26034/509 "2024-05-31T03:28:46Z")
In my setup, I do not see big cpu usage on nodes itself, but antivirus eat significantly more that usual, and RAM consumption more than usual 300-500 mb, but if there is lot of connections it is OK for me, as my setup in node buffer is 4mb ---
**Author:** ![nerdatwork](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/nerdatwork/32/1135_2.png) [@nerdatwork](https://forum.storj.io/u/nerdatwork)\ **Post date:** [May 31, 2024, 3:38am UTC](https://forum.storj.io/t/updates-on-test-data/26034/510 "2024-05-31T03:38:47Z")
> [@Vadim](#): > > antivirus eat significantly more that usual Didn’t you add node drive to exclusion list ? ---
**Author:** ![Vadim](https://storj.bcdn.literatehosting.com/letter_avatar/vadim/32/6_22734026db12803ebb3e059622ac3534.png) [@Vadim](https://forum.storj.io/u/Vadim)\ **Post date:** [May 31, 2024, 3:49am UTC](https://forum.storj.io/t/updates-on-test-data/26034/511 "2024-05-31T03:49:29Z")
Thank you, I forgot to add around half of HDDs to exclusion. now it more resonable. ---
**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\ **Post date:** [May 31, 2024, 5:08am UTC](https://forum.storj.io/t/updates-on-test-data/26034/512 "2024-05-31T05:08:52Z")
> [@zip](#): > > As with such high CPU loads I suspect a VM is a culprit > [@Ruskiem](#): > > 2 cores per node and 1h ago 80-100% CPU usage. > [@Ruskiem](#): > > Oh its VM fault - i’d say wonder what would happen if that wasn’t in VM. see > [@ACarneiro](#): > > Mine had a peak of bandwidth use but CPU and IOWait didn’t really go very high. > [@Vadim](#): > > In my setup, I do not see big cpu usage There are bare metal nodes as far as I know. My observation is the same: the Windows service node doesn’t consume much of CPU (unfortunately in Windows there is no easy way to get a CPU usage in percentage in CLI), but the task manager shows ~0.12% (8 CPU cores). while docker nodes (they are running in the VM because it’s Docker Desktop for Windows): ```auto CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 29ce6d785933 wireguard 0.17% 24.16MiB / 24.81GiB 0.10% 1.98kB / 183B 0B / 0B 21 3d20fef76e67 storagenode5 6.69% 75.09MiB / 24.81GiB 0.30% 2.63MB / 78.5MB 0B / 0B 54 9fb28e5cf48a storagenode2 12.23% 668.8MiB / 24.81GiB 2.63% 10.3GB / 723MB 0B / 0B 229 ``` And the Docker itself shows about 30% CPU usage (8 CPU cores). ---
**Author:** ![anto294](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/anto294/32/7831_2.png) [@anto294](https://forum.storj.io/u/anto294)\ **Post date:** [May 31, 2024, 9:52am UTC](https://forum.storj.io/t/updates-on-test-data/26034/513 "2024-05-31T09:52:42Z")
![Screenshot 2024-05-31 115018](https://storj-s3.bcdn.literatehosting.net/original/3X/0/1/018fe8059656cf1ad40399b17256add3706dbc59.png) It seems that my node has been selected. Hard disks are frying like eggs 😂 ---
**Author:** ![littleskunk](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/littleskunk/32/63_2.png) [@littleskunk](https://forum.storj.io/u/littleskunk)\ **Post date:** [May 31, 2024, 12:52pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/514 "2024-05-31T12:52:11Z")
Test results are a step closer to our target and this time we can keep it running for hours without any errors on the other satellites. That is good. Our target gets more and more in reach. We are working on a different success tracker. Deployment on a Friday is a bad idea so it will have to wait for Monday. There is a good description in the PR what the differences are: [https://review.dev.storj.io/c/storj/storj/+/13308](https://review.dev.storj.io/c/storj/storj/+/13308) To me it looks like we might have a problem with the number of connections now. If I run a single node I have almost 100% success rate. If I add a second node on the same IP they split the load and still almost 100% success rate. I can continue like that up until 4 nodes. If I add more nodes they start to get long tail canceled. My reading of this is that TCP fast open or connection pooling gives me 100% success rate for a single node but the more nodes I add the higher the chance that a new connection needs to be established and that makes me lose the race. Not much so I could as well just ignore it but it is visible on my grafana dashboard. ---
**Author:** ![pangolin](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pangolin/32/15685_2.png) [@pangolin](https://forum.storj.io/u/pangolin)\ **Post date:** [May 31, 2024, 2:13pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/515 "2024-05-31T14:13:14Z")
My Internet connection is now at its limit. Almost all successful uploads coming from SLC. ---
**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\ **Post date:** [May 31, 2024, 2:51pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/516 "2024-05-31T14:51:56Z")
I do not see a lot of CPU usage on my node. The node runs in a VM, the host has 2x Xeon X5687 CPUs, the node VM has 4 cores. ![graph_image-2](https://storj-s3.bcdn.literatehosting.net/original/3X/4/b/4ba8fef3dd61a2af13d845905c2420128a0727c2.png) It looks like the node process uses about 30% CPU. ---
**Author:** ![Ruskiem](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/ruskiem/32/64_2.png) [@Ruskiem](https://forum.storj.io/u/Ruskiem)\ **Post date:** [May 31, 2024, 3:20pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/517 "2024-05-31T15:20:29Z")
30% CPU of 4 cores, could be 60%+ of 2 cores, so roughly rather high, per one storagenode.exe ---
**Author:** ![zip](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/zip/32/7813_2.png) [@zip](https://forum.storj.io/u/zip)\ **Post date:** [May 31, 2024, 3:35pm UTC](https://forum.storj.io/t/updates-on-test-data/26034/518 "2024-05-31T15:35:50Z")
> [@Alexey](#): > > I suspect a VM is a culprit You are correct. I did move few of the nodes from VMs to a bare metal as a test, and the CPU usage is night and day. In some cases tenfold less usage on bare metal vs the VM. Really hard to believe, as I was used to a norm the overhead is a few percent at most when running workloads on VMs. I suspected the networking is causing this, but that isn’t the case as it was the same when I switched to VFIO. I suspect all the mitigations the Kernel is doing for the guests is the problem, especially on older hardware. This can be turned off, but I haven’t had a chance yet as that requires a reload and probably won’t bother and move out of virtualisation alltogether. If Docker causes similar performance hit, then I guess it would be advisable to move away from the Docker as well and ask SNOs to run this on bare metal if possible. [Previous page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=21) [Next page](https://forum.storj.io/t/updates-on-test-data/26034.md?page=23)