How to implement storj as cold storage with NGINX file cache?

Developer Discussions
1

Hello.
I use django + nginx.

I my nginx.conf
location /media { root /var/www/html/media;}

I want to use storj as storage for large media file.

I want to reduce download traffic from storj.

My idea:

User requests http://mysite/media/testfile

Nginx requests testfile in /var/www/html/media (this is hot storage or file cache).

If file doesn’t exist, then my python script is downloading this file from storj to /var/www/html/media.

If file exists, then nginx serves file from /var/www/html/media instead of storj.

Thanks for help.

2

Hi Mike,

I’m not clear on what your question is. Can you clarify what part you need help with?

3

I need to avoid to download media file from storj every time, if this file already on web server.
I use nginx which serves media. I need nginx check if file exists on server or not.
May be storj already has own solutions?

User requests http://mysite/file.txt. We download it from storj to /media folder and show to user.
User requests http://mysite/file.txt second time. We show it from webserver ( from /media folder)

4

Rclone mount + VFS cache?

3 Likes
5

Hello @mikesombr,
Welcome to the forum!

You may also use your bucket as a static web site.

1 Like
6

Thank you, I will try!

7

Using nginx as a caching proxy should do this. I have not done this myself but happened to be researching it this morning.

  1. Setup storj s3 gateway
  2. Setup nginx to proxy requests to the gateway (this may be confusing because you’ll be connecting to localhost, proxying connections to localhost, don’t configure nginx to listen on the same port the storj s3 gateway is listening on and pay attention to port numbers and you’ll be ok.)
  3. Mount your storage using the nginx proxy address (not the storj gateway)

Please read these sources:

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_path