Is there a caching mechanic on tardigrade (or storj)

Hi, I did not find the information about a caching system that tardigrade or storj currently use or will implement.
Because the cost of storing data in storj network depends on the usage I want to know if there is some optimization about getting files.
For exemple I have a nextcloud server at home and (imagine) the storj package for nextcloud is alive. Is there a possibility to store some 50Gb of cache on my nextcloud server or will it try to contact storj network every time I need a file?

Thanks Storj for this awesome project !

I don’t think so. However, you can implement this feature.
Or you can submit or vote for idea there: https://ideas.storj.io

2 Likes

I don’t know much about this but a way (not necessarily a good way) of doing it is mounting the Storj bucket, then creating a zpool using a file on that bucket. With ZFS ARC and L2ARC caching I think that would have the desired effect?

1 Like

Or you could use s3fs which supports a local cache as well.
https://linux.die.net/man/1/s3fs

1 Like

Thanks, I was not aware about ZFS ARC and L2ARC technologies.
I will take a look but it seems very nice :slight_smile:

This seems to be a question for nextcloud, don’t you think?

Maybe not because if something is implemented with storj it could be use for other solution.
I imagine somthing like ... --cache-folder=/cache ... in a get file command that tell the system to check this folder before checking in storj network.

But yes it could be easier to develop nextcloud compatibility.

I don’t have a perfect solution, but I’ll list a few things you could do.

  1. If you want storj to implement something like this, posting on ideas.storj.io is by far the best way. People could upvote your idea there and this list is monitored closely for the most popular ideas.
  2. Nextcloud could implement this as well, but from what I can tell they don’t (yet) support caching for other cloud storage solutions. So the likelihood that they will do it for storj isn’t that high. However, if you can push them to implement this for existing cloud storage providers including S3, you could use that in conjunction with the Storj S3 gateway.
  3. You can use S3FS to mount S3 compatible storage to your nextcloud server. Keep in mind that the local cache s unbounded, so you either need to set up quotas separately or schedule a purge of old files yourself. From there I’m sure you could make the mounted file system available through nextcloud.
  4. You can use S3FS without nextcloud inbetween on linux systems as well of course
  5. If you want to use this on windows you can use stablebit clouddrive to have a local cache of S3 storage.

3 through 5 rely on the S3 compatible Storj gateway. A lot of existing solutions are already available for S3, which makes this much easier. If you want a solution that doesn’t use the S3 gateway, for now I think you will need to build it yourself.