Yes you are 
Our gateway-mt instances are working with nodes over the globe as if you would use an libuplink. The only problem here from what I can see - the amount of Gateway-MT instances is much lower than amount of nodes, but they are powerful!
You can send multiple RANGE requests for a single file to be handled by multiple HTTP/S3 servers.
2 Likes
I would add, this is also true for the Storj native.
The power of Storj in parallelism, independently of the protocol. However, with a native you have an unlimited option.
Why would the S3 backend be slow?
1 Like
Most people use a tool to download mods anyway, Vortex or similar, so it would take considerable load off their other infrastructure.
But Storj is still crazy expensive for egress.
1 Like
Sure, but currently Storj gets $7 per TB for egress and pays only $2 to node operators. And since this would use the native implementation, edge services costs would not apply. So they have a lot of room to provide volume discounts.
I havenāt used mods in a long time, so I wasnāt aware of Vortex mod manager (made by Nexus Mods). Since they already offer that for some of the most popular games, that would be an excellent place to start with integrating Storj.
They added on a demand. You repeatidely false claim that Storj S3 is slow. You are wrong with all respect. This is proven by many customers, that our S3 and especially Storj native is working faster for them.
The power of Storj in parallelism due to a distributed design. You may just increase a parallelism for the every big object to upload/download its segments in parallel and increase the number of threads, you can easily saturate almost any connection and hardware which you currently have (most of the S3 clients will buffer each 64MiB segment into a memory before upload).
If you think that you are correct - please provide a proof from the customers (what is you usually request from us and any evidence is ignored by you). Proofs, please. If your router is not capable, well itās not a Storjās problem, right?
Proofs, please. Otherwise itās just a false claim.
Not necessarily - you may use any download manager, which is able to download ranges of the file, like GetRight or similar.
By the way, itās possible to convert our linkshare link to a torrent:
If someone wants to use BitTorrent with Storj, thatās possible, The āGetRightā method of seeding a Torrent is trivial to implement on top of Link Sharing.
Storj takes a balanced approach to dynamic scaling for simple and rational economic reasons. You canāt be 10x cheaper than Amazon while scaling on Amazonās infrastructure. The infrastructure providers which make economic sense often have issues such as not being everywhere, providing poor quality networks, or scaling in the timeframe of days. Navigating these issues has gotten easier over time though! Weāve scaled out our Gateways dramatically over the past year, and will continue to do so.
Technologies such as BitTorrent scale well when there are plenty of seeders voluntarily donating their egress. It works well, because everything is free! Storj needs reliable and rational / economically motivated Storage Node providers. Unfortunately, centralized bandwidth costs decrease dramatically with scale. It can be challenging for a distributed network to be cost competitive for viral use cases. Thatās why Storj is focused on selling into markets with traffic patterns that make sense for customers and SNOs. This includes large files with high egress, but not high concurrent-use content. In these cases, Storj is striving to be the best possible āoriginā behind those CDN caches.
Circling back to S3, Iāve done so much Storj testing over the years. If youāre seeing S3 as slower than native Storj, youāre likely seeing a well understood aspect of TCP called bandwidth delay product. Itās a concern with any HTTP traffic, and it will get better as we grow the S3 Gateway network. Yes WebTorrent is cool, yes we should do more things like that. But for right now, storage is a nine billion dollar industry; let us win the customs who will pay us $7 egress before we move on to the $2 market.
Finally, if you want to store your files on Storj while leveraging irrational and egalitarian actors for free data distribution; BitTorrent can work for you! The āGetRightā method of seeding a Torrent is easy to implement on top of Linksharing. Standard warnings apply here: donāt share illegal or copyrighted content! Seeding BitTorrent requires your files to be public: people will notice illegal content, we will likely terminate your account, and itās not fun to deal with.
3 Likes
I tested it. It saturated my gigabit connection. The problem must be on your end.
5 Likes
Donāt compare speed of a single file to total bandwidth available.
Same here.
Letās be honest here. You needed multiple threads in order to saturate your gigabit connection.
Thanks @wthorp , I appreciate you taking time to write such a detailed response. What you say makes perfect sense to me and peak demand for a small number of files is something thatās hard to deal with. Itās good to get some context on where things are at.
I have no doubt that its possible to scale up edge services to get performance at scale, but I do worry about the reliance on costly cloud infrastructure. It seems S3 is the go to for most customers currently, which eats away at the significant egress margin you would otherwise have. While some future directions have been discussed to alleviate that, Iāve seen less progress on that so far than I would have hoped.
Having a browser based solution for uplink and differentiating pricing between native and edge services egress would shift a lot of potential use cases to the much more cost efficient and actually distributed and end-to-end encrypted native implementation. It would just be better for everyone. And with those additional costs out of the way, there is a much clearer path towards supporting high demand file use cases by implementing piece availability scaling on demand as outlined as a possible future expansion in the whitepaper. As the reduction in costs would get you closer to perhaps being able to provide CDN like pricing at scale.
So I get where you are at and I understand that this is not yet the time for such customers. But I certainly hope to see some of the steps I outlined being taken as a road towards supporting things like this.
1 Like
Obviously. Nobody expects single thread gigabit throughput.
1 Like
I wonder if having multiple copies and choosing one at random for each download wouldnāt be a good enough workaround for now. It would cost close to nothing compared to egrees, and with high probability each copy would be stored on a different set of nodes.
Sure, but it would mean the customer has to do all the work of detecting demand, dynamically scaling up and load balancing across multiple files. Thatās a lot of effort to implement and Iām guessing most customers wonāt be inclined to do that. Ideally there would just be a switch on upload that enables scaling on demand if Storj implements that at some point.
1 Like
Was thinking of it just as a short-term workaround.
hi,
Yea, and how far it got You?
how much out of that 9 billion We got?
āAre Ya winning son?ā
I guess not much 7$ customers, if they can pay ~1,19 euro at hetznerās
You know?..
Anyway keep up the great work!
You are getting there! 
P.S.
i mean, not really, i donāt care to give all my bandwidth away for even free, just to get my HDD filed to the max for Godās sake, its been 1 year and my 16ās are just half fullā¦
Isnāt that ultimate Storjās advantage?
We, the nodes, can basically offer free egress, we are in fact now, for 2$/TB, just cut the BS, make it ~free and gain customers, to the point i canāt stop plug-ining new (used) HDDs pls.,
thaaanks.
1 Like
I donāt have the means to run a test like that. I guess doing a test on AWS would not be that expensive (there are 200 Gbps machines for <2 USD/hour), but Iām too lazy for that.
I couldnāt even get speedtest.net to admit my connection is 1 Gbps, had to deliberately use a different client to use multiple connections. And I canāt get AWS S3 to do that even from an AWS EC2 machine within the same region. So Iām really not sure if this is such an expected feature.
Besides, the answer is as @Alexey writes. If you want faster S3 with the current state of the network, you use multiple HTTP connections and multiple copies of a file. A customer-side workaround for a missing feature, but doable.
2 Likes
I have sort of done this the past, here
This is the result:
Shape: m4.10xlarge
Connection: 10 Gigabit/s
Zone: us-west-2 (Oregon)
VCPU: 40
Satellite: us1
Download, MBps:
| type\threads | 1 | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|---|
| s3 | 12.2 | 24.1 | 49.0 | 86.6 | 87.5 | 92.5 |
| native | 11.6 | 22.3 | 43.1 | 85.0 | 138.0 | 234.0 |
Caveats:
- This is tested with duplicacy, that was compiled against old version of uplink. There were some perf changes since.
- This was done using explicitly set 64MB chunks side
- Duplicacy itself may be introducing unobvious overhead. Perhaps rclone download test would be better.
- I donāt remember if I checked for CPU saturation. The previous, slightly smaller VM was CPU limited.
Perhaps a new test is in order, e.g using m4.16xlarge. But I have exhausted my budget allocated to this exerciseā¦
3 Likes
c7gn.whateverxlarge is my new best friend. Graviton 3 is a beast, and this instance type has tons of network bandwidth.
Might be.
In my case it [claims to] connects to a competitorās ISP. Whatever 
No, this is exactly what I meant pointing out RANGE requests in post 26 above. And it is possible to do RANGE requests from javascript and concatenate the responses while saving to disk. As such, my understanding is that it is a viable workaround for an impatient customer with spare engineering time while waiting for a proper solution from Storj.