[s3 gateway] browser limitation of parallel uploads with pre-signed URLs

Developer Discussions
1

Hello folks :wave: ,
I am building an application that requires performant uploading of many smaller files, say 1 user in a session would upload 1000 files of 500kb, 1000 files of 1.5Mb and 1000 files of 5Mb.
I am currently generating pre-signed URLs for each of the files on the server side and provide them through API such that the user can upload them from the client side. What I am noticing now is that the number of files that I can upload in parallel is limited by the browser, which looks in line with this answer on Stackoverflow, stating that browsers limit the number of parallel connections to 6-12 connections per domain.

As all the pre-generated URLs would go through https://gateway.storjshare.io, this would then throttle the upload. Are there any other URLs provided (e.g. eu1.gateway.storjshare.io and eu2.gateway.storjshare.io) that I can use to unlock the parallelisation limit?

Are there any other ideas to overcome this limitation?
Anything else that I should consider? Ideas on the scalability of this design? Alteratives?

Thanks for you help :pray: !

1 Like
2

You can use a regional endpoint, however, it would limit your non-EU users.
The EU1 one is https://gateway.eu1.storjshare.io

By the way, using small files will be less effective. It would be much better if you could pack them to a bigger zip archive.
If you would use a linksharing service for content delivery, it will allow to access files inside the zip archive without downloading the whole archive.

1 Like
3

Are there any other URLs provided … that I can use to unlock the parallelisation limit?

You can generate presigned URLs for virtual-hosted–style buckets which would give you a different subdomain for each bucket: bucket1.gateway.storjshare.io bucket2.gateway.storjshare.io.

I suspect that opening 3000 connections may not be the fastest way to upload 3000 objects and you may end up with better performance in the 6-12 connection range anyways.

2 Likes
4

Thank you for your insights @Alexey and @pwilloughby. I really appreciate it :pray: !

1 Like