Static website performance

[striker43]

Hi, as I was reading many articles about hosting static websites on STORJ DCS, I am wondering about the performance when doing this. The example pages of some bloggers look really nice and were loading fast for me, but I am wondering what would be the limit for STORJ hosted static websites. I mean how many simultaneous visitors are the maximum?

Let’s assume I am hosting a 200 KB HTML file which references images with a total size of 1MB. If I have 1.000, 10.000 or even more simultaneous users for that site who want to load the content, all the traffic would go to only a very few storagenodes which will most probably not be able to handle the load, I guess.
Does someone have some experience with what maximum number of parallel requests to a file are realistically possible?

[zuik]

I’m not sure but maybe it is possible to upload the static site multiple times to storj to increase the nodes where the site is saved.

[Alexey]

It doesn’t work like this. The previous version would be deleted and there would be only one version and the same number of nodes, but they could be a little bit different (from other locations).

The solution for that is scaling, as described in the Whitepaper: https://www.storj.io/storjv3.pdf

6.1 Hot files and content delivery

Occasionally, users of our system may end up delivering files that are more popular than
anticipated. While storage node operators might welcome the opportunity to be paid for
more bandwidth usage for the data they already have, demand for these popular files
might outstrip available bandwidth capacity, and a form of dynamic scaling is needed.
Fortunately, Satellites already authorize all accesses to pieces, and can therefore meter
and rate limit access to popular files. If a file’s demand starts to grow more than current
resources can serve, the Satellite has an opportunity to temporarily pause accesses if necessary, increase the redundancy of the file over more storage nodes, and then continue
allowing access.
Reed-Solomon erasure coding has a very useful property. Assume a (k, n) encoding,
where any k pieces are needed of n total. For any non-negative integer number x, the first
n pieces of a (k, n + x) encoding are the exact same pieces as a (k, n) encoding. This means
that redundancy can easily be scaled with little overhead.
As a practical example, suppose a file was encoded via a (k = 20, n = 40) scheme, and
a Satellite discovers that it needs to double bandwidth resources to meet demand. The
Satellite can download any 20 pieces of the 40, generate just the last 40 pieces of a new
(k = 20, n = 80) scheme, store the new pieces on 40 new nodes, and—without changing
any data on the original 40 nodes—store the file as a (k = 20, n = 80) scheme, where any
20 out of 80 pieces are needed. This allows all requests to adequately load balance across
the 80 pieces. If demand outstrips supply again, only 20 pieces are needed to generate
even more redundancy. In this manner, a Satellite could temporarily increase redundancy
to (20, 250), where requests are load balanced across 250 nodes, such that every piece of
all 250 are unique, and any 20 of those pieces are all that is required to regenerate the
original file.
On one hand, the Satellite will need to pay storage nodes for the increased redundancy,
so content delivery in this manner has increased at-rest costs during high demand, in
addition to bandwidth costs. On the other hand, content delivery is often desired to be
highly geographically redundant, which this scheme provides naturally.

But it’s not implemented yet.

[BrightSilence]

With the new pricing using Storj as CDN actually becomes really interesting. The pricing basically beats everything on the market. That scaling feature would be really great to serve those use cases.

[KernelPanick]

Until scaling is implemented, the storj static hosted site could be a node behind a load balancer. I’m thinking about doing this myself for failover. Storj will always be up, my servers might not.

[jtolio]

For what it’s worth, you can get a really great experience by using Cloudflare or Fastly here. A CDN frontend such as one as these will not only handle high traffic situations for you, but will also add SSL.

If you’re interested in a really easy webhosting solution, I suggest

1. Upload your site to sj://bucket/prefix/. You can use gateway.us1.storjshare.io with the awscli which supports recursive copy and folder sync. (uplink cp is getting a recursive feature natively very soon.)
2. Registering a domain name and making Cloudflare (for example) your domain’s DNS name servers. Let’s say your domain is mytestwebsite.com
3. Adding the DNS settings from uplink share --dns=mytestwebsite.com sj://bucket/prefix/ to Cloudflare and enabling Cloudflare proxying and SSL.

This works great for both providing SSL and adding a layer of caching for heavily trafficked websites. We have a number of customers doing this type of thing. My personal site is small potatoes and only gets ~56k requests/month, but https://www.jtolio.com/ is set up this way. Check out https://www.jtolio.com/?wrap=1

In the long term we hope to make Fastly/Cloudflare/etc unnecessary, but we’re not quite there yet.

[striker43]

Thank you for the fast replies
Scaling sounds really interesting! Also for other use cases as static website hosting.

I will definitely try the Cloudflare solution for now, thanks for that hint