Allow Downloading partial files, custom ranges, custom starting position

This would allow features like:

  • continue download on large file when it was interrupted
  • restore only some files from a backup (if the backup solution supports this of course)
  • jump to specific time when streaming video

Libuplink can already do all the things you are suggesting at the stripe level. I think that is also the case when using the gateway MT. (probably also self hosted gateway)

I’ve used this myself to even test storing Chia plots on Storj DCS. Those files are 101GB each, but only small parts are ever read. It worked surprisingly well. Latency was just a little too much so that sending partials took too long. I think you could even get that part to work with a little tuning. It’s surprisingly performant even with the ghetto setup I used for that experiment. Either way, clearly it didn’t download the entire 101GB file each time it was checking whether a plot matched a challenge. I believe it just downloads the stripes it needs.

1 Like

Great to hear. Can you point me to some place where I can read about it, preferably using the nodejs bindings? I seem to only find information about downloading complete files.

I’m going to leave that to someone who has actually worked with either those bindings or libuplink in general.

Since my chia test setup involved janky software to mount S3 buckets to a local folder combined with the gateway-MT, I don’t think that approach would be helpful to you.

To download a custom range, at least with the standard Go API, you would use a DownloadOptions struct, where you can find members for specifying the desired Offset and range Length:

https://pkg.go.dev/storj.io/uplink#DownloadOptions

I’m not entirely certain that is exposed through the nodeJS bindings. It doesn’t appear to be mentioned in the documentation. I’ll try and find out. [Edit: Yes, it is exposed. See Types, Errors, and Constants for an example.]

If you have the option of using a Gateway (whether MT or self-hosted), then you can get this functionality using the standard HTTP Range: header. See HTTP range requests - HTTP | MDN.

6 Likes