Migrating 20TB of files to storj

I’m keen to find out if this is the right solution for me, im currently hosting around 20TB of users media files on a dedicated server for a website i run for music artists, the server is fairly costly and non-redundant in its current form.

I would like to utilise decentalized storage, storj.

First question which i presume is ok is the ability for users to stream mp3 files from storj.

Second…what would be the best way for me to migrate all my existing files, i am presuming this would be a manual process with a script i need to create (as i would need to update my local database which references the location of the mp3 files at the same time).

Third…is there a C# library to handle the core functions of uploading/downloading files etc?

Is there anything else i should be aware of that could hinder me?

Thank you.

2 Likes

Hi Phinett,

Welcome to the Storj forum! I am not sure what you’re paying in terms of hosting for your web server, but keep in mind that if Storj hosts your files and people are streaming from your server, you do pay egress charges on each of those streams. That can add up. Of course if the amount of streaming is low and your hosting costs are high, Storj may be a good option for you.

It might be a good idea for you to try Storj out with one of the tools such as Filezilla and get an understanding of how to setup everything. You can also utilize S3 compatible tools and resources. Once you have more familiarity with how it works, we can point you at resources to better fit in with what you want to do.

Bindings for C# can be found here

2 Likes

Ok egress is not something i have accounted for, i just seen the headline of the pricing for storage and bandwidth.

Is there somewhere i could estimate costs?

I need to check latest stats, but lets assume around 10TB of bandwidth used per month for streaming/downloading files at the moment.

10TB of egress is $70

I’m a solutions architect over at Storj and would be happy to discuss the best migration path for you. Generally, I use rclone hosted on a VPS with a non-metered internet connection to a cloud to cloud migration.

Uplink-c (Just updated) GitHub - storj/uplink-c: Uplink C library

4 Likes

This all sounds good!

The next issue i see ahead without doing any meaningful research for now is what URL is used to access the migrated files, say for example at the moment my file URL is https://files.hosting.com/xyz123.mp3

In my database i store the file URL so its easy to access it on my web app, but would i then need to update every record in my database with a new URL for every file migrated? or could the filename be kept the same to make this a simple?

Appreciate the help.

Storj provides a service called linksharing you can use to expose files in your buckets. You could access these files using a domain/subdomain you control using something like the following steps: gateway-mt/linksharing.md at 58462bf62a91dde527635aaf84dcb24761ecfb3f · storj/gateway-mt · GitHub

5 Likes

I have begun to attempt to setup a few things, im kind of stuck setting up uplink at the moment.

I am doing “uplink setup” command at the moment.

It’s asking for my satellite url, i only have the random string it generated for me when creating my account, i chose the EU region too. But what would be the full URL i need to enter here including port number? I can’t find this info in any documentation.

Also the passphrase it is asking for, which passphrase is this? Where would this have been generated?

I’m super keen to get this up and running, just a little overwhelmed with the initial setup and documentation at the moment.

1 Like

I think i have actually managed to work it out, i have modified my dns records etc and now a test file is serving correctly from storj on my custom domain!

now i just need to start the migration using rclone from my web server.

One last question for now then, i presume it’s not possible to use secure https:// links for my media files at the moment?

There are two ways to generate an access grant to set up uplink CLI with:

  1. Create an Access Grant - Storj Docs
  2. Create Access Grant in CLI - Storj Docs

Unfortunately not, but it’s on our radar. We have an issue that tracks this effort: Allow TLS against custom Linksharing domains · Issue #158 · storj/edge · GitHub.

3 Likes

To add to Artur’s point, you could use a CDN provider like Cloudflare or run your own reverse proxy using something like nginx to terminate TLS and provide https.

4 Likes