Helping the network become profitable — what can operators contribute while staying viable? (calculator inside)

[arrogantrabbit]

@Grimm

is not compatible with “wasted space”, and the skill to add and manage several nodes although it’s not that hard, it’s not trivial either if you want a good performance reliable node, specially with current HDD prices, it’s completely bonkers.

I had rams sticks die, PSU died on me, an MB also died, hardware dies, it needs replacement.

What changed though? The price of drives makes it even less desirable to buy extra drives for storj — but that was never encouraged in the first place. How you manage your server for other needs did not change. Buying more space than today’s need is driven by anticipated use going forward, and does not depend on cost of drivers.

But if your ram stick dies or PSU or mb — you would have to replace it regardless of whether storj is running. Otherwise you don’t really need a server and we are back to square one — building hardware for storj.

[littleskunk]

@Grimm

Dude, I did say Repair to zero but increase the storage, a shift not an elimination.

Dare you! I can’t offer repair for free. This is going to kill the network!

Take this with some humor please. I just want to give you some of your own medicin…

Anyway back to topic. We can lower the repair costs to some extend but all the way to 0 means storj will lower the expansion factor. You would end up with at least 3x more repair traffic but at a $0 rate thats fine. The reason storj would want to do that is total disk usage goes down. That will be a lower total payout. So be careful with $0 repair egress offers. Storj is going to take that deal, adjust RS numbers to reduce costs and you still end up with the same lower payout you are trying to kill me for.

@Grimm

The latest statistics I can pull from EU and US satellites of median (not average, very important) healthy pieces is 41 and 28, I might be wrong in understanding what that means but what I read is half of the files have less than 41 and 28 to reconstruct respectively, unless I am wrong, or the data is wrong, if not then repairs need to be made correct?

The number for EU1 sounds about right. I think yesterday the lowest number I have seen in my report was about 40. I don’t know how frequent it gets updated.

On the US1 satellite the lowest number is from storj select. That has a different RS config.

@Grimm

OK but storj advertises to customers how many pieces should be on the network? Close to 80?

Current RS config for the global network is 29/46/54/75

@Grimm

just get repairs done more quickly

There is no need for any quick repairs. Lets say your node goes offline and lets say a few thousand segments fall below the repair threshold. It doesn’t matter if these segments get repaired the next day or the next week. The only requirement is that the repair system at all times repairs enough segments so that we are closer to the repair threshold and stay far away from the minimum threshold. And that seems to be the case as we can see from the repair progress.

[Grimm]

@arrogantrabbit

The price of drives makes it even less desirable to buy extra drives for storj — but that was never encouraged in the first place.

It was, sorry but it was and it was crazy, you can search the forum.

[Grimm]

@littleskunk

Take this with some humor please. I just want to give you some of your own medicin…

You are not, because it’s not the same thing and I did apologize for my mistake, in general, pride is at the bottom of all great mistakes.

@littleskunk

We can lower the repair costs to some extend but all the way to 0 means storj will lower the expansion factor.

Explain, I don’t see how repair egress at zero or near zero lowers expansion factor?

@littleskunk

The reason storj would want to do that is total disk usage goes down. That will be a lower total payout.

I don’t understand, if you have missing pieces and you repair, my understanding is some other node gets more disk usage because because it gets pieces it did not have, total used space goes up, how does disk usage does down?

@littleskunk

adjust RS numbers to reduce costs and you still end up with the same lower payout you are trying to kill me for

Increase the Storage payout and/or the egress.
And I have been getting a lot of repairs lately, close to egress, still I would rather have more on storage that repair, costs are on the client, they are predictable for storj, and repairs get done with little to no cost to Storj.

@littleskunk

There is no need for any quick repairs. Lets say your node goes offline and lets say a few thousand segments fall below the repair threshold.

OK maybe my English is not that strong or good, maybe I read wrong, maybe I don’t understand how the tech work, but I am pretty sure that bellow the repair threshold means it’s no longer repairable?!

Did you mean average or median threshold?
what is the minimum threshold?

you realize that median means half of, not average, meaning half of that satellite has less or equal 29 pieces for repair, unless I am mistaken storj needs 27?! That is not looking good!

[Vadim]

@Grimm

Explain, I don’t see how repair egress at zero or near zero lowers expansion factor?

Storj boses will make RS expansion smaller, that each file take less space, then for less space they pay less for SNO, because there will be less pices ho hold, then will need make more repair work, that is free and we also will not get any money for that, you node will give more repair egress and will give less paid egress, because load from repair will be bigger and this will slow down your node.

29 pieces needed to rebuild file minimum.

[Grimm]

@Vadim

Storj boses will make RS expansion smaller, that each file take less space

Still don’t get it, if you repair something, it should take more space not less, on another node but more space used, the network it not at full 80 pieces capacity on any satellite.

But that is the thing I said, if you have more used space and they increase the space payout it might be more advantageous, more nodes should get more used space and the network is resilient and performance, you might get less egress competing with another node, but you get more used space overhaul, predictable stable income.

I am not using my node full I/O, I bet that is the case for the vast majority, and if repairs are done fast enough, or consistently enough I doubt that it will hammer any node that hard, it should level out.

[Vadim]

today it is start from 54 not 80. when it repair it do it till 54 not more. if RS numbers will lower initial piece amount will be less. like 45 for example. But minimum always 29. so if RS number will lower, then it will be repaired till 45. and drop till minimum will be much faster. and again every time it repair free egress will be taken from nodes.

[littleskunk]

@Grimm

I am pretty sure that bellow the repair threshold means it’s no longer repairable?!

29/46/54/75 is the current RS setting.
29 pieces are needed to repair the segment. If it ever falls below 29 the file might be lost. I am going to say might because nodes can still come back online. It will be lost if that doesn’t happen.
46 is the repair threshold.
54 the success threshold.
75 long tail. Technically irrelevant for this equasion. More important for performance.

Now we can play around with 2 values.

1. We can lower or increase the repair threshold. That will lower or increase the risk of ever falling below the minimum threshold. Right now the lowest number of pieces I have seen was 36.
2. We can lower or increase the success threshold. That will lower or increase repair costs. With the current payout rate about 10% repair traffic is the perfect balance between used space and repair costs. So each segment gets repaired once every 10 month.

Ok lets pretend we are storj and want to lower costs. Current expansion factor is (54+46)/2 (average number of pieces) / 29 = 1.7. The real expansion factor is a bit higher but lets ignore that for the moment. If the lowest number of pieces seen is 36 that means we can remove 6 pieces and would still be above the minimum threshold. This is just an example that I would not recommend. Just for explaining the math. New RS number would now be 29/40/48/something. Expansion factor gets lowered to 1.5. This would reduce your payout by about 10%.

Again not something I would really want to do. 7 extra pieces sounds great. These 7 extra pieces are the reason we don’t need to repair quickly. We can run it on a steady pace and have weeks to observe if the repair system keeps up or falls behind.

Now to your $0 repair idea. 29/46/49/something was active between the current and the old RS setting. It will cause 30% repair traffic. So each segment gets repaired every 3 month. The reason storj would want to do that is because lower expansion factor means less payout for the total used space. Right now the only reason they don’t do it is because they have to pay storage nodes for that excessive repair traffic.

[Toyoo]

@magnetic_miner

Can you please provide numbers?

Numbers, mostly no, sorry, that’s from my customer. But I can point you to specific challenges, because that’s public knowledge.

One challenge is that a single S3 download speed is capped. Older versions of documentation explicitly stated that you’re supposed to make a separate connection for each 90 MB/s of bandwidth you have.

Make one concurrent request for each 85–90 MB/s of desired network throughput. To saturate a 10 Gb/s network interface card (NIC), you might use about 15 concurrent requests over separate connections. You can scale up the concurrent requests over more connections to saturate faster NICs, such as 25 Gb/s or 100 Gb/s NICs.

They’ve removed it at some point, and I’ve myself observed sustained transfers as low as 20 MB/s. The AWS-provided workaround is to use S3 Transfer Manager, but it has its own limitations effectively giving up on the original S3 protocol. This is no longer a simple HTTP GET.

Though, you’ll probably say that “just” making many concurrent connections would be good enough. And I’d agree. Except that…

There is a cap on requests per second you can do per so-called “prefix”, which is not something you can even control explicitly—so you pretty much have to assume this is a limit per bucket. You can do ~5500 download requests per second. Which might seem like a a large number… but if you want to leverage multiple concurrent connections to saturate the aforementioned c8gn.16xlarge, you need ~300 concurrent connections (best case), or even ~1200 connections (observed worst case) per VM. So five VMs may already saturate this limit.

BTW, this is why AWS introduced a special bucket type, “S3 Tables”, for big data with higher limits… but it’s not perfect either, it is extra cost, and only recently sort-of become stable.

Both limitations strongly suggest that S3 stores each prefix on a small number of physical machines. Torrents and Storj don’t do that, it’s absolutely fine to download from 10k different nodes concurrently—not just because you’re downloading many files concurrently, but also because each file chunk is effectively stored on a different set of nodes.

I wish I could deploy a Storj satellite for that customer…

[Alexey]

@Grimm

it is not recommend because it defeats the purpose of the /24 rule to make data more resilient to loss because of one ISP of several nodes being in the same location.

This remains true. However, our protocol doesn’t prohibit its violation at your expense—bypassing restrictions isn’t free.
We can’t block all VPN nodes, at least not yet, because this method could also be used by nodes behind CGNAT, which would otherwise simply render them offline.
I would prefer to see the node selection adjusted so that piece distribution is more secure: no more than one piece per segment, at least not on the same disk, preferably not on the same hardware, and even better, not in the same location.

For now, the choice-of-n rule slightly improves this situation, and it could be strengthened by increasing it from 2 to 4 or more. Then VPN nodes will inevitably end up at the end of the list due to their inherent latency.
This doesn’t completely solve the problem, but it reduces the likelihood of data loss.

[Grimm]

I am going to try and compile the list of problems or complaints about the nodes on the network and make my case why would lower repair and egress and increase storage would probably be in the best interest of everyone.

I can update the list has we go.

Problems:

• Nodes using VPN
• Several nodes in one drive
• To many nodes
• expensive nodes
• cost to storj (albeit SNO cost alone will not save storj)

What does storj wants:

• one node per drive
• one ip per location
• quality nodes

@littleskunk

Now to your $0 repair idea. 29/46/49/something was active between the current and the old RS setting. It will cause 30% repair traffic. So each segment gets repaired every 3 month. The reason storj would want to do that is because lower expansion factor means less payout for the total used space. Right now the only reason they don’t do it is because they have to pay storage nodes for that excessive repair traffic.

@littleskunk

29/46/54/75 is the current RS setting.

@littleskunk

46 is the repair threshold.
54 the success threshold.
75 long tail.

Lets keep this in mind.

Here is my idea, it might be wrong but you are asking for them.
Let say that storj makes repair almost 0 and/or cheaper egress but increases storage to where the average SNO gets payed the same has now for every TB.

So hopefully this would increase traffic (repair or egress, hopefully both) and would:

• put a strain on VPN connections
• lower the response times on multiple nodes in one drive
• filter out good nodes from bad nodes, not fast response nodes, actual bandwith and stuff

The change on RS setting would make:

• expensive large nodes less profitable, because it would decrease storage for them, smaller quality nodes more profitable or about the same (that is not a bad thing)
• lower the total cost to storj
• increase the ability to make more solutions and plans for the client

Now you cannot have quality nodes without the SNO making some profit, that is not good for business.
Sorry if people disagree with that is not the ethos of the project and that storj should never pay for your drive, but it should because of what the network needs and the business wants, good quality nodes, lots of them, Storj can turn the nob if they want more or less expansion, or even contraction, but it cannot be a negative sum for the average SNO to the point it cannot maintain a good quality node, a dedicated drive with good internet with one IP.

Now here is the kicker, let the client pick and pay how resilient they want the data or fast they want that traffic.

If its possible, you could match those RS setting to low, medium, high in a bucket, with different costs.
Clients that want more resilience and throughput pay more, but the number of pieces and nodes that hold those pieces also increases.

How this would work out and what the sweet spot values would be I think it’s not that difficult.

The only unknown is how US internet works, because in most of Europe internet is cheap, it’s mostly fiber, and you pay 50€ if that much for residential gigabit, no limit or caps in some or most countries.

I can be wrong, I can be right, I might be missing something, it’s an idea and it’s the only one I got.

Feel free to tear it apart, like all ideas should be.

[littleskunk]

@Grimm

Here is my idea, it might be wrong but you are asking for them.

Yes that might work. There are 2 problems I have to point out. I don’t think we have the time to implement that. There is no feedback between failing downloads and getting less uploads. Meaning a node can just accept that it is failing downloads and continue to grow. Meantime customers will run away.
The second problem is that even if we have the implementation for it we still end up having to impact the customer in order to get a measurement. In times with low traffic the bad nodes still get away without getting overloaded. Only when we put enough preasure onto the network it can identify the bad nodes and that will again be noticeable by the customer.

An alternative to that could also be to level the playfield and make it even for all nodes in the other direction. But that also requires more development time than we have currently available. Still the advantage would be that a CDN customer like we have seen yesterday gets fast downloads at all times because we don’t waste it on repair.

Talking about that CDN customer. Its on again. I love to see that traffic on my nodes.

[Grimm]

I understand you probably don’t have time to implement client features.
I assumed that changing the RS, the payout values, and making repairs has fast has possible would be fairly easy.

@littleskunk

Only when we put enough pressure onto the network it can identify the bad nodes and that will again be noticeable by the customer

Honest question, doesn’t that happen now?

@littleskunk

There is no feedback between failing downloads and getting less uploads.

Again that depends, if you have that issue today, probably nothing is going to change in that matter, if a node is far away from that customer and the packet has to many bounces, or a route is completely saturated between BGP routers, it would be false feedback either way, and probably is happening today already.

Question is does making changes on that nob today would it make a better network, better for storj and the average SNO.

@littleskunk

An alternative to that could also be to level the playfield and make it even for all nodes in the other direction.

I dont understand? Make what even?

[littleskunk]

@Grimm

I assumed that changing the RS, the payout values, and making repairs has fast has possible would be fairly easy.

Yes that are all just config changes.

@Grimm

Honest question, doesn’t that happen now?

It was happening in the past. This alternative RS setting that would cause 30% repair traffic was impacting customers for a few weeks. Right now with just 10% repair traffic we basically reserve 3 times more traffic for customer downloads. In my case that CDN customer is consuming that.

@Grimm

Question is does making changes on that nob today would it make a better network, better for storj and the average SNO.

Possibly. An easy way to find out would be to ask that CDN customer about his user experience. Also the more he downloads from my nodes the more I am going to upgrade my internet connection. (After my volunteer $0 payout rate is over)

[Grimm]

@littleskunk

It was happening in the past. This alternative RS setting that would cause 30% repair traffic was impacting customers for a few weeks. Right now with just 10% repair traffic we basically reserve 3 times more traffic for customer downloads. In my case that CDN customer is consuming that.

Dial it slowly until it levels off?

@littleskunk

Possibly. An easy way to find out would be to ask that CDN customer about his user experience. Also the more he downloads from my nodes the more I am going to upgrade my internet connection. (After my volunteer $0 payout rate is over)

That’s not a bad thing, better quality nodes are good for Storj, also the new figures of payout could prevent you from your “aggressive” expansion if that is not desirable.
It would root out nodes that can’t deliver because they can’t handle all that traffic.

I am getting peek upload of 63.5 Mbps, average is about 10 to 20 Mbps, I got 400 and I can upgrage to 10000, but that nob can be turned to compensate better nodes with better delivery.

Use the nob, don’t starve the average SNO.

[alpharabbit]

@Grimm

Use the nob, don’t starve the average SNO.

So your proposal is to help the whale SNO and kill the average instead of starving all? I guess I like that idea.

[Grimm]

@alpharabbit

So your proposal is to help the whale SNO and kill the average instead of starving all? I guess I like that idea.

No, that is not the idea, the idea is the reverse, starve the mega whale that has 20 nodes on one drive, I didn’t say it was a good idea, I did say it can be good, it can be bad and I did say to tear it apart.

If you think it will help the whale SNO and kill the average, can you explain how?
Not saying that it can not happen, if you can please explain, I enjoy honest skepticism.

[alpharabbit]

@Grimm

If you think it will help the whale SNO and kill the average, can you explain how?

I guess you are underestimating the power of hashstore? Multiple nodes per HDD is not the bottleneck. Maybe some whales would have to upgrade their internet lines a bit.

[Grimm]

@alpharabbit

I guess you are underestimating the power of hashstore? Multiple nodes per HDD is not the bottleneck. Maybe some whales would have to upgrade their internet lines a bit.

Lines cost money and IP addresses also, managing all that is also not free.

Maybe the current state of things is actually helping them more, because they get more traffic if they have a collocation on a DC or some special low latency connection.

Shifting the pay a bit more towards storage, reduce the egress and repair, the overall storage Storj needs also reduces, is going to have some impact on expensive whales.

One issue I do see is some of these whales might just limit the upload because it’s no longer paying that much and just focus on storage, that is maybe a problem today also, IDK.

Maybe you are right.

Figuring out what would happen is hard, that’s why ideas need skepticism.

[littleskunk]

@Grimm

Figuring out what would happen is hard

I do have a good picture for my own nodes. This would be my personal proposal:

$1.25 per TB used space
$1.5 per TB egress
$1 for audit and repair traffic

This is somewhere in between. It lowers the payment for egress more than used space. It will impact old nodes with more repair traffic more than new nodes. For my nodes the egress payment would still be high enough to justify upgrading my connection every time I max it out.

Again I need to point out this is the calculation for my personal nodes. I am sure you will tell me now I would suggest it works for everyone. Don’t do that. Repeat after me one more time. My personal nodes!