Is it possible to use a storj bucket as a remote working directory?

Hello all,

Is it possible to use a storj bucket as a remote working directory?
Or it is not recommanded…

Or (if not) is there a way to configure a bucket to be able to do so?
Or storj was not designed at all for that purpose?

Thanks

PS : I am using storj on a GNU/Linux laptop via “rclone mount”

Hi @climb65!

Storj is considered “object storage”, which is a class of storage popularized by Amazon S3. Object storage has different semantics than file system storage. For example, instead of directories, object storage considers “key prefixes” - there are no directories per se.

This means that there are important and perhaps subtle differences between how operations work with object storage vs local filesystem storage. For the same example, you can’t rename a directory because there is no directory to rename! Some object storage platforms don’t even have support for renaming an object (though ours does). If an object storage platform supports server-side copying (ours does), you can also copy each object to a new path (with a new path prefix) and then delete the old path. To rename a prefix, you’d have to do this individually for every object that has this prefix. Suddenly, something that should be efficient in a local filesystem (renaming a directory) is extremely resource intensive in an object storage platform.

So, there are some techniques to try and expose an object storage platform as a local filesystem, (e.g., rclone mount, which often uses FUSE), but all of these techniques are imperfect, since the mapping of semantics doesn’t line up right. Another difference - in object storage, you can have a file named a and a file named a/b in the same bucket! Hard to represent that in a filesystem.

Applications that expect local filesystem semantics will struggle to run out of a mapping filesystem like rclone mount or s3fs or similar, especially if the application is trying to get behavior like flock or more. This is why most folks use applications against a local filesystem, and then either integrate object storage natively with the application, so the application knows what to expect, or sync local filesystem data with object storage separately.

I know that’s kind of a bummer if you’re trying to get a globally available local filesystem, but that’s kind of the scoop these days. Does that help?

3 Likes

Thanks for your answer,

That helps.

To get something quite close to what I want with storj I have found options to “rclone mount” that greatly improve the usage of remote “folders” (buckets). My entry points are always storj buckets.
Here are below the two options I use :

–cache-dir
–vfs-cache-mode full

The cache-dir parameter is a folder on the local filesystem.

I’m happy with those two options, and with storj.

3 Likes