Hi Team,
We want to download a file which has a size of 1.5TB(Blockchain data) in tar form uploaded on Storj. We saw download failure in StorJ intermittently. Is there a way to download the tar file in incremental manner ?
Any Sort of help is appreciated.
How do you want to download? Via link sharing? Pre signed url? Via s3? Via uplink? Using third party tools?
Both s3 tools and uplink support ranged requests, and third party tools, like CyberDuck support resuming downloads.
Hello @shivraj001 ,
Welcome to the forum!
@arrogantrabbit is correct, you may use third party tools which supports a resume of the download.
You may also use uplink too, in an automatic manner using a parallelism or use ranges.
If you use a linksharing URL, then you can use a downloader extensions/apps, which can support a ranged download and resume of the download.
There is also a way to convert it to bittorent
The “GetRight” method of seeding a Torrent is trivial to implement on top of Link Sharing.
@arrogantrabbit @Alexey We are trying to download blockchain snapshot provided by https://chainsnap.io/ . We got uplink command to download a tar file. It fails in middle of download. Is there a way to convert the uplink command to a http link?
below is the error message which we got while using uplink command ?
uplink cp --parallelism 16 --access ***************** sj://xyz/snapshot.tar.zst ~/datadrive
failed to download part 222: uplink: quiescence
failed to download part 230: uplink: context canceled
failed to download part 225: uplink: context canceled
failed to download part 236: uplink: context canceled
failed to download part 234: uplink: context canceled
failed to download part 233: uplink: context canceled
failed to download part 227: uplink: context canceled
failed to download part 228: uplink: context canceled
failed to download part 229: uplink: context canceled
failed to download part 223: uplink: context canceled
failed to download part 237: uplink: context canceled
failed to download part 235: uplink: context canceled
failed to download part 238: uplink: context canceled
failed to download part 226: uplink: context canceled
failed to download part 231: uplink: context canceled
failed to download part 232: uplink: context canceled
Oh, you have a weak router seems.
then I would suggest to reduce a parallelism to at least 8 (maybe even less because it’s overloading your router). Or you can try to use a wired connection instead of WiFi.
You may convert it to a Linksharing URL:
uplink share --url --access ***** sj://xyz/snapshot.tar.zst
Or to an S3 credentials:
uplink access --access **** register
You may also use the provided access grant to configure rclone and try to use it (it has an integrated retry functionality).
The uplink actually has built in retry logic. If you get that “quiescence” error, I believe that means it has done a couple of retries (6) and they have all failed. I would recommend reducing your parallelism. If you don’t use the advice from Alexey, you can also download parts of the file directly with --range.
Thank you team!
@Alexey’s linking sharing technique worked out. We used aria2c client command along with it and it got downloaded without any issue.