I know this might be sliding into another thread yet- but what about something like a CDN for Steam/Epic/EA/etc ?
Storj/Tardigrade is really good a geographically dispersed, high throughput, durable content delivery… right? And part of the platform is that it’ll pull only as many pieces as it needs to complete a block (<80 total pieces) from up many SNO’s, simultaneously(checks off agg bandwidth requirements, ie I cap my steam at 400Mbps). I mean games are not getting any smaller, similar to the film industry’s 4k and soon 8k movements, and they all need storage and distribution, world wide, to get their product out to a consumption point. There’s nothing saying that Storj/Tardigrade couldn’t be that S3, or direct uplink.dll/so, backend CDN for all that- right?
yeah generally anything that people don’t download all the time, but do need to download on occasion where they don’t really care much about the small costs, but just that it works and fast.
because the download costs and the storage costs… so to optimize that tardigrades best use is shared storage that many of the users sharing it will use maybe once a week or once a month…
downloading games would fit very neatly into that catagory also because there often is repeated downloads, since people reinstaller or delete games because of local storage capacity issues.
with 10000 people sharing the same online storage it’s basically free and only the person downloading is paying for what they need…
games might be one of the best options for all involved, ofc gamers will tend to loose their nerve quite fast, so maybe it’s best that the universities are the trail blazers… when tardigrade moves to games it has to work work work, fast and flawless… else it may take years to gain the momentum it would have from being new in the space.
I just worry that most SNOs are running on domestic connections and will be trying to serve content from a connection at the time when it’s being strained the most anyway.
I don’t think Tardigrade is there just yet. Yes there is some overprovisioning with pieces, but nowhere near at the level for peak demand from game releases. You would have thousands of people downloading the same pieces at once. This would still easily overload nodes. It would be different if the RS settings could be tuned or even scaled up based on demand. If thousands of nodes could hold pieces for the same segment, then suddenly downloads could be spread out much wider. But functionality like that isn’t yet in place.
And even if it were, we’re talking about millions of pre-orders and gamers who buy Cyberpunk 2077 on day one. And that’s just a single game adding up to 40GB * 1000000 = 40PB. How many nodes now? about 10k maybe? With perfect equal spread that would be 4TB per node and that’s without accounting for download overhead. I have no doubt that in the future there will be enough scale to serve these purposes, but right now it’s simply not an option. This would be a sure fire way to grind everything to a halt.
Auto scaling aka background distribution to more nodes for shards that get downloaded a lot, would be really interesting. Also this is one reason why I keep requesting information on new deals, because as SNO you might be considering more bandwidth from your ISP if there is enough peak demand. But of course this would take much longer than buying a new hard drive.
Well, of course, but I think autoscaling will only really start to be interesting when there are more nodes to scale to to begin with. As my very quick calculation already showed, even with perfect scaling, there just aren’t enough nodes yet to serve that kind of load. So yeah, let the network grow at a natural pace with use cases it can already serve and in the future we may see more CDN like cases and features when that makes sense for the scale of the network.
To be fair: Obviously there is some kind of infrastracture available to these companies that can handle this kind of load, wether your calculation is correct or not.
No sane company would rely only on Tardigrade at this stage. And of course before using Tardigrades there would be massive testing, if it could handle what it is supposed to handle.
But Tardigrade might serve as additional way of delivering, as backup or whatever. So to assume at the current stage of Tardigrade a company with such requirements would completely abandon its working infrastructure and rely on Tardigrade only is not realistic at all.
yeah something like that seems to be the future. Let gamers upload the game for other gamers.
Just like microsoft is distributing updates by using other windows users who will upload the updates without knowing it (or was it only in the local lan? I forgot)…
This saves a lot of bandwidth, similar to torrents.
Blizzard’s actually does use BT on the backend to do this, which is why it commonly got flagged by College campus’ trying to block torrenting when it was just a bunch of WoW nerds players trying to get updates.
the windows update thing is only lan i think… and is in settings.
P2P for games was required because of the vast demand at specific times, P2P scales with activity…
which is something we might count as a significant advantage to P2P data transmission and something which tardigrade doesn’t have the advantage of… which is actually a bit of a shame maybe…
because it would in some ways make P2P superior to Tardigrade.
tho it wouldn’t surprise me if one could include or make Tardigrade into working like P2P eventually…
doubt that’s a big stretch.
The Tardigrade uses p2p technology too. When you download - you download from different nodes, when you upload - you upload to different nodes too. Please, note - directly between your peer and node peer. This is why there is no speed limit, you will likely overload your bandwidth much sooner than overload the Tardigrade.
aside from those uploading and downloading isn’t peers for the network…
the basic torrent P2P concept is kinda based around that isn’t it…
which gives that amazing scaling because the more is downloading then the more will be uploading also…
so no… i wouldn’t call Tardigrade P2P because i think it doesn’t do what the essential concept of P2P is…
i suppose one could argue that it is related yes… but from my view very different also…
however in this world there are rarely any pro’s with out cons… and vice versa…
so one could say tardigrade is more secure compared to the fundamental P2P concept.
also didn’t P2P kinda die to erasure coding
There is no traffic between storage peers for replication, the Tardigrade just do not use a replication to make it resilent.
Thus it’s still uses a p2p technology to provide the peering between the customer and nodes.
The metadata mostly stored on torrent trackers, so it’s the same concept
We have had a kademlia in the protocol, but it was slow for nodes propagation, so we decided to remove it. Instead, each node checking in to satellites themselves.
but i wouldn’t consider anything a p2p network if all the users weren’t peers, i would consider that a fundamental concept of it… and i don’t think tardigrade fills that…
but i support there is a big difference between using p2p technology and having a p2p network.
p2p isn’t all great either… it tends to loss stuff that isn’t popular anymore…
The problem is nodes discovering. If you do not have any widely known server with a metadata (torrent tracker in Bittorrent, satellite in the Tardigrade), the nodes discovering becomes a cumbersome - thus DHT in torrent and kademlia in Storj V2, and alpha Storj v3.
When the node is available only with some probability, the initial start could be very long (while the client search for really available node) and could finish with a error.
This is not Enterprise-ready.
Thus we took the best from p2p - multi threads for traffic on nodes and uplink and servers with metadata (satellites) for fast nodes discovery.
that sort of makes sense… i had been pondering of why during v2 that the network seemed to accelerate depending on uptime… ofc that falls pretty much in line with the whole torrent experience… haven’t noticed it much in v3… even if it does seem to be there a little bit… didn’t realize it was the same actually…
most invention is combining known things in new ways… or just making shit smaller lol
It could be, though… the Gateways are a big step in the right direction for that, as they can be used to offload the compute of encryption (that way I could put one at the edge of my network and all LAN talk to it) and can also work to bake in some of the missing parts like soft fail transfer to node and automatic retry to next available without the “data originator” knowing or needing to care.
