S3QL with Storj S3 gateway

Hello,
I tried to mount a s3ql (GitHub - s3ql/s3ql: a full featured file system for online data storage) filesystem with Storj S3 Gateway.

For that purpose, I used the s3c backend and tried to build the FS with:

mkfs.s3ql --debug --plain s3c://gateway.eu1.storjshare.io/s3ql/

This ends with an error:

Traceback (most recent call last):
File “/usr/bin/mkfs.s3ql”, line 11, in
load_entry_point(‘s3ql==3.3.2’, ‘console_scripts’, ‘mkfs.s3ql’)()
File “/usr/lib/s3ql/s3ql/mkfs.py”, line 159, in main
dump_and_upload_metadata(backend, db, param)
File “/usr/lib/s3ql/s3ql/metadata.py”, line 313, in dump_and_upload_metadata
upload_metadata(backend, fh, param)
File “/usr/lib/s3ql/s3ql/metadata.py”, line 327, in upload_metadata
cycle_metadata(backend)
File “/usr/lib/s3ql/s3ql/metadata.py”, line 135, in cycle_metadata
cycle_fn(“s3ql_metadata_new”, “s3ql_metadata”)
File “/usr/lib/s3ql/s3ql/backends/comprenc.py”, line 310, in copy
self._copy_or_rename(src, dest, rename=False, metadata=metadata)
File “/usr/lib/s3ql/s3ql/backends/comprenc.py”, line 343, in _copy_or_rename
self.backend.copy(src, dest, metadata=meta_raw)
File “/usr/lib/s3ql/s3ql/backends/common.py”, line 108, in wrapped
return method(*a, **kw)
File “/usr/lib/s3ql/s3ql/backends/s3c.py”, line 413, in copy
resp = self._do_request(‘PUT’, ‘/%s%s’ % (self.prefix, dest), headers=headers)
File “/usr/lib/s3ql/s3ql/backends/s3c.py”, line 528, in _do_request
self._parse_error_response(resp)
File “/usr/lib/s3ql/s3ql/backends/s3c.py”, line 562, in _parse_error_response
raise get_S3Error(tree.findtext(‘Code’), tree.findtext(‘Message’), resp.headers)
s3ql.backends.s3c.S3Error: NotImplemented: A header you provided implies functionality that is not implemented

Nevertheless I tried with a local S3 gateway.

mkfs.s3ql --debug --plain s3c://127.0.0.1:7777/s3ql/

With: backend-options: no-ssl

Returned without error but when I try to mount the FS I get an error:

$ mount.s3ql --debug s3c://127.0.0.1:7777/s3ql/ ./s3ql
Using 4 upload threads.
Autodetected 1048532 file descriptors available for cache entries
ERROR: No S3QL file system found at given storage URL.

Wouldn’t it be nice to be able to mount an unlimited filesystem to store any data from any application?

1 Like

Hello @xtrmrider ,
Welcome to the forum!

Please, use s3fs or rclone mount instead.
The rclone could be configured natively with client-side encryption.

1 Like

Hello @Alexey ,

Thank you for your answer.

I tried s3fs but it gives poor performances and user experience.

rclone mount is working great with native integration (even if it still refer to tardigarde…) but it gets error when copying files to the mounted drive when using S3 gateway.

Anyway, the native rclone mount seems very promising.
Browsing is fluid and user experience is good.

I originaly wanted to use s3ql for the caching capability.

What would you advice to save bandwidth for multiple access to a file with rclone ?

In fact rclone mount with --vfs-cache-mode full gives really nice user experience!

I will be able to continue my setup of a Piwigo server with unlimited storage on a very small VPS ! (Alternative to new policy of google photo…)

3 Likes

Its currently a known issue that s3ql does not work with Storj Gateways, because it requires a features of S3 that we do not currently support (server side copies). We want to make sure we get this feature correct, so this may take some time to resolve. It is definitely on the roadmap though!

2 Likes