Updates on Test Data

Sorry, I did not check the satellite ID, just noticed some weirdness in my graph and decided to share it.

Congratz, you saturated my connection. :smiley:

Edit: CPU for context.


Itā€™s busy, but acceptable.

2 Likes

Previously I didnt even notice the tests, but I am for sure feeling it now (between this and some other stuff my 1GB seems to be about saturated). Keep it coming!

3 Likes

Ahh here is an update i have 3 sites.

flwstern-fw-external - 10 gbits - maxxing out at 4-5 gbits
fw-bahnhof - 1 gigs burstable but need to do QoS here cause i only pay for 100mbit.
fw-sumpan - here im totally saturated at 500mbit.

3 Likes

How does your upload successrate looks with circuit being saturated?

Better than the last tests during the night. Succeeding most transfers. I donā€™t have exact numbers though.

2 Likes

The last test run was bestofn with n=3. This time we uploaded to all nodes and not just vetted nodes. We hit our goal for the first time.

Now trying n=3.5 and later n=2.5. Also on the list is to make the long tail a bit shorter like 3-5 extra nodes instead of the current 8.

What I donā€™t understand is why the unvetted nodes have such an impact. The fact that you get higher load on your node means this is more a side effect. That is something I will question later.

8 Likes

Nice to hear that! Can you tell us more precisely what the goal is? How much bandwidth is the target?

At least in my case I have nodes that are almost 6 months old and still have not received a single audit from saltlate in any of them.
My guess is that this might be the situation for a lot of others nodes, and if so, there is a significant amount of nodes (and therefore bandwidth) that has not been used in previous tests.

5 Likes

I donā€™t have any unvetted nodes though. I can only speculate that currently my SSD cache isnā€™t saturated, which it was during the last tests during the night.

:clinking_glasses:

2 Likes

Filewalker has started, so it may impact the performance somewhat
graph_image-2

CPU usage by the node process
graph_image-2

Success rate seems OK, at least to me:

Thatā€™s a lot of requests at the same time:
graph_image-2

1 Like

Aaaaah! I did wonder what had happened when my non-vetted nodes went a bit crazy :smile:

You beat me to it.
I have about 5 nodes of different ages (all between 3 and 6 months old) and none of them have received a single audit from Salt Lake either.
I think vetting is broken on that satteliteā€¦

1 Like

Why are there such a sharp fall off cliffs? Was this the actual expected load pattern or was there some bottleneck getting overwhelmed?

Most likely the test was stopped so the traffic went down immediately.

1 Like


The tests appear to be ramped in 8 steps giving that shape.

1 Like

Fresh information from our standup:

  1. Choiceofn node selection to compare it to the bestofn
  2. Remove 20% of the nodes from the node selection and check how that impacts performance. Donā€™t worry we will add them back a few minutes later.
  3. Run the load test for a few hours and not just minutes to test how stable it is.
  4. Verify how good our node selection is. It feels a bit like picking random numbers and measure throughput without understanding how the actual distribution looks like.
9 Likes

With my internet connection at the limit I guess it only depends on average TTL how much storage I can accumulateā€¦ :roll_eyes:

Any ideas how to reject uploads based on TTL? :thinking:

Just do the math. For my node I came to the conclusion that I just need a faster internet connection and my storage node payout is going to cover the extra costs.

Currently testing choiceofn with n=4

It should be less biased and less risky than bestofn. I really hope we can get to the same throughput with this one. It would be my favorit.