Updates on Test Data

pangolin · May 23, 2024, 8:51pm

My combined ingress from all nodes was over 1 TB yesterday while the router reported only 600 GB. And there is a lot of other traffic. So I guess the new fake ingress is 2x to 3x real ingress.

littleskunk · May 23, 2024, 8:52pm

You still don’t get a benefit of running multiple nodes on the same subnet but it will scale up and down the traffic per node not per subnet. So lets say we have 2 nodes sharing the same subnet and one has a better success rate. It will get selected more often by the power of 2 node selection but it would still only reach halve the request rate it could get without the second node in the subnet.

littleskunk · May 23, 2024, 8:56pm

I don’t know what you are talking about. You are still on the wrong party. My node has a 99% success rate. There is no fake traffic if you win all the races. Also how do you expect netdata to show any fake traffic? There is no code change we could do that would let netdata show wrong numbers.

Toyoo · May 23, 2024, 8:57pm

What is the timeframe to recover?

pangolin · May 23, 2024, 8:59pm

I was talking to @ACarneiro. Could it be you are on the wrong party?

littleskunk · May 23, 2024, 9:05pm

I am a mind reader you know. We run a test and all the feedback that we got within minutes must be from tools that track this in minute intervals. So I know exactly what party we all signed up for and I see no room for fake traffic on this party.

ACarneiro · May 23, 2024, 9:07pm

I’m mostly seeing spikes in Ingress, though. Not much change in egress at all.
Are you just testing uploads your end or am I just not being selected?

littleskunk · May 23, 2024, 9:13pm

The part you quote was about the stomping herd effect. The power of 2 node selection shouldn’t overload the node in the first place so no need for a recovery. As long as you still have a reasonable success rate the recovery should take just minutes.

If you manage to get a 0% success rate you might end up with no uploads at all. That makes recovery more difficult. In that case this timer will forget the 0% success rate after some time and your node would get selected one time at random and if you make it you will get selected more frequent. If you don’t make it you will stay at 0% until the timer is over again for a new chance.

github.com

storj/storj/blob/60c8f73fe67d64a54d0e30842095c9695d71513c/satellite/metainfo/config.go#L146


      
          	MaxMetadataSize              memory.Size         `default:"2KiB" help:"maximum segment metadata size"`
          	MaxCommitInterval            time.Duration       `default:"48h" testDefault:"1h" help:"maximum time allowed to pass between creating and committing a segment"`
          	MinPartSize                  memory.Size         `default:"5MiB" testDefault:"0" help:"minimum allowed part size (last part has no minimum size limit)"`
          	MaxNumberOfParts             int                 `default:"10000" help:"maximum number of parts object can contain"`
          	Overlay                      bool                `default:"true" help:"toggle flag if overlay is enabled"`
          	RS                           RSConfig            `releaseDefault:"29/35/80/110-256B" devDefault:"4/6/8/10-256B" help:"redundancy scheme configuration in the format k/m/o/n-sharesize"`
          	RateLimiter                  RateLimiterConfig   `help:"rate limiter configuration"`
          	UploadLimiter                UploadLimiterConfig `help:"object upload limiter configuration"`
          	ProjectLimits                ProjectLimitConfig  `help:"project limit configuration"`
          	SuccessTrackerEnabled        bool                `default:"false" devDefault:"true" help:"enable success tracker based node selection"`
          	SuccessTrackerTickDuration   time.Duration       `default:"10m" help:"how often to bump the generation in the node success tracker"`
          	SuccessTrackerTrustedUplinks []string            `help:"list of trusted uplinks for success tracker"`
          
          	// TODO remove this flag when server-side copy implementation will be finished
          	ServerSideCopy         bool `help:"enable code for server-side copy, deprecated. please leave this to true." default:"true"`
          	ServerSideCopyDisabled bool `help:"disable already enabled server-side copy. this is because once server side copy is enabled, delete code should stay changed, even if you want to disable server side copy" default:"false"`
          	UseListObjectsIterator bool `help:"switch to iterator based implementation." default:"false"`
          
          	UseBucketLevelObjectVersioning bool `help:"enable the use of bucket level object versioning" default:"false"`
          	// flag to simplify testing by enabling bucket level versioning feature only for specific projects
          	UseBucketLevelObjectVersioningProjects []string `help:"list of projects which will have UseBucketLevelObjectVersioning feature flag enabled" default:"" hidden:"true"`

Edit: I think there are 4 generations in total. So it should retry once every 30 minutes? I mean this only applies to the nodes with the lowest success rate. I have a hard time to imagine how such a node would look like.

Toyoo · May 23, 2024, 9:18pm

I’m mostly thinking of a case where due to an unrelated I/O the node started performing badly. So, let say, I run a scrub on a partition where node is hosted, making my node lose 90% races for that time. How long after the scrub finishes my node will start getting the regular traffic again?

Also, would it be possible to publish these statistics, so that

node operators can verify how their nodes compare to others’?
for purposes of comparing performance of network in various geographical regions (an idea I recall being suggested somewhere on the forum as well)?

BTW, happy to see @BrightSilence’s suggestion implemented!

Edit: and one more thing: would this approach penalize more nodes in geographical regions with less customers?

BrightSilence · May 23, 2024, 10:04pm

Also very compute efficient. Nice solution. And since you only have to beat a single node, it’s not like nodes won’t be selected at all. Except for I guess the absolute worst one. But that one probably deserves it. And I see you’ve already taken care of the fact that they will be tried again at some point.

Though I feel like it also leaves some performance on the table. Every time you compare 2 slow nodes, you end up selecting a slow node anyway. Though I don’t know how to solve that efficiently and without causing the stomping herd effect. So I’ll shut up for now and give that some thought first. (Until I think of something more inspired than power of 3! node selection )

What’s the logic of this? Does it look at a specific timeframe? Last x uploads? Or something similar to how scores are updated?

Sometimes I have decent ideas, I guess. I have suggested changing the RS settings many times as well, so I’m a happy camper right now, seeing all these tests happening.

Node selection only deals with the whole node population on upload. Downloads are already limited to the nodes that have the pieces. Currently I believe 39 downloads are initiated for 29 successful pieces. (@littleskunk correct me if I’m wrong or this has changed) Since the success threshold is now 65, they couldn’t select double 39 to use power of 2 node selection. Given that and since it seems uploads are a priority atm, I’m guessing it’s only uploads for now.

Ps. I noticed this line still lists 80 as default success threshold.

github.com

storj/storj/blob/60c8f73fe67d64a54d0e30842095c9695d71513c/satellite/metainfo/config.go#L141


      
          	MaxInlineSegmentSize memory.Size `default:"4KiB" help:"maximum inline segment size"`
          	// we have such default value because max value for ObjectKey is 1024(1 Kib) but EncryptedObjectKey
          	// has encryption overhead 16 bytes. So overall size is 1024 + 16 * 16.
          	MaxEncryptedObjectKeyLength  int                 `default:"4000" help:"maximum encrypted object key length"`
          	MaxSegmentSize               memory.Size         `default:"64MiB" help:"maximum segment size"`
          	MaxMetadataSize              memory.Size         `default:"2KiB" help:"maximum segment metadata size"`
          	MaxCommitInterval            time.Duration       `default:"48h" testDefault:"1h" help:"maximum time allowed to pass between creating and committing a segment"`
          	MinPartSize                  memory.Size         `default:"5MiB" testDefault:"0" help:"minimum allowed part size (last part has no minimum size limit)"`
          	MaxNumberOfParts             int                 `default:"10000" help:"maximum number of parts object can contain"`
          	Overlay                      bool                `default:"true" help:"toggle flag if overlay is enabled"`
          	RS                           RSConfig            `releaseDefault:"29/35/80/110-256B" devDefault:"4/6/8/10-256B" help:"redundancy scheme configuration in the format k/m/o/n-sharesize"`
          	RateLimiter                  RateLimiterConfig   `help:"rate limiter configuration"`
          	UploadLimiter                UploadLimiterConfig `help:"object upload limiter configuration"`
          	ProjectLimits                ProjectLimitConfig  `help:"project limit configuration"`
          	SuccessTrackerEnabled        bool                `default:"false" devDefault:"true" help:"enable success tracker based node selection"`
          	SuccessTrackerTickDuration   time.Duration       `default:"10m" help:"how often to bump the generation in the node success tracker"`
          	SuccessTrackerTrustedUplinks []string            `help:"list of trusted uplinks for success tracker"`
          
          	// TODO remove this flag when server-side copy implementation will be finished
          	ServerSideCopy         bool `help:"enable code for server-side copy, deprecated. please leave this to true." default:"true"`
          	ServerSideCopyDisabled bool `help:"disable already enabled server-side copy. this is because once server side copy is enabled, delete code should stay changed, even if you want to disable server side copy" default:"false"`

littleskunk · May 23, 2024, 10:15pm

10 minutes per generation * 4 generations = 30 minutes history

Th3Van · May 23, 2024, 10:30pm

MRTG

Peaked at 5,48 Gbit/s

Th3Van.dk

Roxor · May 23, 2024, 11:16pm

Somewhere out there is a SNO with several external 2.5" SMR USB drives… plugged into their grandma’s Gateway computer… passed through VMWare Workstation to a Window 2012 VM, and combined with Storage Spaces RAID5 before being presented to their node.

…and they just read about the new power-of-2 node selection… and are thinking:

“Crap.”

Toyoo · May 23, 2024, 11:16pm

Power of 2 choices has an explicit name because it’s a spectacular jump from just picking one item at random. But going from 2 to 3, the difference turns out to be miniscule. This is one of those counterintuitive results from probabilistic algorithms field. See e.g. this PDF for some exposition.

It makes a lot of sense here, it’s great that Storj applied it with success.

BrightSilence · May 23, 2024, 11:31pm

Yeah, that was clearly a joke. You quoted it by skipping the first part of the sentence and omitting the emoji, both of which made it pretty clear that was not a serious suggestion.

pangolin · May 24, 2024, 12:09am

Isn’t it in fact “crap” for the lower 50% of nodes?

mattventura · May 24, 2024, 4:41am

It seems that it will be more percentile-ish. The fastest node out of the bunch will be selected 100% of the time, a median node will be selected 50% of the time, etc.

jammerdan · May 24, 2024, 5:06am

Ok. But all this testing and the optimizations have been announced being necessary for customers in the sales pipeline. I believe you have announced that, I can’t remember. So it would be great to learn if with all the tweaking we are gearing towards what those customers expect because at the end this is what everybody wants I guess, to have the network ready to take on the requirements of them. And if there are signs that the progress brings us closer to the deal and the potential customers are pleased with what we are able to offer, then it would be great to hear about it. Even more of course if a contract has been won.

This sounds very interesting. And if it helps to not to overload the nodes with upload requests they cannot fulfill then it is even better. Are we going to see longer tests with that like over the course of several days?

But does this kind of selection work even more in favor of nodes that are close to the uploader? It is just because for the use case where upload and download location are the same or close, this is perfect.
But Storj advertises kind of worldwide CDN-like file access speeds. And when the use-case is to move files fast, so that they get uploaded in place Toronto but downloaded in Singapore would that new node selection not hurt that experience?

My idea would be to give a customer an option to select which matches their use case best:

Upload in A download in A
Upload in A download in B
Upload in A download everywhere

And while the node selection for upload could always be the ones that are fastest to the uploader which gives him a good uploading experience. In the background the repair workers could distribute the files over time according to the customers use case.
And of course also the S3 gateways could take that profile selection into account as well when uploading and select nodes according to the customers profile.

Maybe at some time in the future maybe even AI could be used to find the best node selection and distribution for each file of a customer based on the upload and download behavior. An AI powered node selection and file distribution based on individual customer behavior, that would be awesome.

jammerdan · May 24, 2024, 6:19am

Oh yes that is an interesting thought.
We have these silly limited lines here all over.
And even fibre is being sold not as symmetric but with limited upload bandwidth.
So a node with good download bandwidth can accumulate, with that change now maybe even more than ever. But on a crappy asymmetric line with limited upload bandwidth he can’t deliver → Downloading customers not happy

Alexey · June 6, 2024, 7:02am

49 posts were split to a new topic: “Unlimited” ISP plans