Error while uploading large files via django-storages

I am using Storj for serving and storing my static and media content for a Django project. For interacting with Storj buckets I am using django-storages. I am facing no problem while uploading small file (~10 MB), however for files which more than 25-30 MB size I am getting the following error :

ClientError at /post/new/

An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied.

Request Method:	POST
Request URL:	http://127.0.0.1:8000/post/new/
Django Version:	4.0.3
Exception Type:	ClientError
Exception Value:	
An error occurred (AccessDenied) when calling the UploadPart operation: Access Denied.
Exception Location:	C:\Users\Sam\Envs\proj\lib\site-packages\botocore\client.py, line 745, in _make_api_call
Python Executable:	C:\Users\Sam\Envs\proj\Scripts\python.exe
Python Version:	3.8.8
Python Path:	
['D:\\proj\\django_proj',
 'C:\\Users\\Sam\\Envs\\proj\\Scripts\\python38.zip',
 'c:\\programdata\\anaconda3\\DLLs',
 'c:\\programdata\\anaconda3\\lib',
 'c:\\programdata\\anaconda3',
 'C:\\Users\\Sam\\Envs\\proj',
 'C:\\Users\\Sam\\Envs\\proj\\lib\\site-packages']
Server time:	Thu, 28 Apr 2022 17:57:14 +0000

This makes my app almost useless since I cannot upload large files.

Second question, how convenient is it to use a DCS like Storj for serving and storing media files by a webapp built on framework like Django ? If my app targets public users, would it be a secure and reliable to use Storj ?

Hello @samuel ,
Welcome to the forum!

It’s better to use a binding for python: GitHub - storj-thirdparty/uplink-python: Python bindings for libuplink

With s3 library you need to have a support of multipart uploads, which it seems doesn’t have. If you can specify a multipart_threshold and multipart_chunksize to 60MB, it could help. But I didn’t find such parameters in django-storage documentation. So perhaps you need to implement it in the code.

Please try also use a Storj hosted Gateway instead of Self hosted Gateway to exclude influence of low upstream bandwidth.

Alternatively you can configure AWS CLI and use it to upload big files.

See also: Host a Static Website | Storj Docs

2 Likes

This error is surprising. All of these file sizes should work, is there anything else different about the files that succeed and the ones that fail? Do you have any more details you can share that might help in tracking down the issue? Things that would help are: django-storages version and backend settings (make sure not to include anything sensitive), any additional logs.

Storj should absolutely be a good use case for storing media files. We have designed it to be especially secure and reliable.

2 Likes

I believe @Alexey here is right. While I don’t work in python regularly, it looks like the boto3 library provides you with ways to configure the multipart upload size, but the django-storages driver does not provide you with the ability to configure it. I see references to supporting _multipart, but I don’t see where the transfer config is updated accordingly.

One option that you can try here is skipping the django-storages layer of abstraction and just go to the boto3 client directly (if you prefer the S3 interface), or (as @Alexey calls out) simply using the python client directly.

References:

2 Likes