A chain snapshot will be only needed to be downloaded in the server CLI (curl, wget, bittorrent, etc.) - no end user web browser access needed
We are sharing these snapshots publicly to other people who run their blockchain nodes. My question is that is there any difference between the link sharing or uplink based access grant sharing, e.g. in
Download speed
Download resumes
Egress cost
Other factors
Because all the downloaders are technical users, we can write our instructions to use uplink command instead of curl and link sharing if there are some tangible benefits.
Using uplink or rclone with a native integration can be faster, if your users have enough bandwidth.
You may provide them a read-only access grant generated by uplink or even S3 keys, something like
It will generate a read-only access grant (for uplink CLI, rclone or Filezilla with a native integration), S3 credentials (for those who have a low bandwidth, they could configure almost any S3-compatible tool, like AWS CLI, rclone with S3-compatible integration or Cyberduck) and linksharing URL (for all others).
To increase a download speed please take a look on Hotrodding Decentralized Storage
Thank you Alexey and Dominick and everyone for the help.
This answered my question:
Download Tuning (Large Files)
UplinkCLI - Native
Native uplink is the best choice for large files as it bypasses our edge service layer and connects you directly to the nodes. This results in the best possible throughput. If downloading several large files it is commonly faster to enact the parallelism per file yet download the many files in a serial fashion one after the other.
To further understand this, just clarifying if uplink cp can resume interrupted downloads?
Uplink is reasonably resilient but must be restarted if interrupted. We do a lot of blockchain synchronization and uplink is fast and reliable. One idea is that for massive snapshots you could break them up into a few pieces, this is not required or common.
uplink cannot resume a download if it was interrupted, nor does it have a retry. rclone with native integration might help, at least it can retry the interrupted thread or chunk.
There are a few things mentioned in the blog post @Dominick shared that you might want to take note of:
--parallelism flag that on hardware with large specs can increase download speed dramatically. This one takes a little trial and error based on the specs of the machine, the internet connection, etc. There are diminishing returns (i.e. the download might start going slower or doubling the parallelism doesn’t double the speeds) if you increase it too high.
Extracting directly during the download. Note in the blog post that the tar file is extracted directly during the download so it doesn’t have to be saved first then extracted. This can be a cost savings for blockchain nodes as they don’t need double the SSD disk space on the node to save the tar file first then extract it after the download is done. Uplink requires additional RAM based on the parallelism you choose, but as far as I’m aware this is the only tool out there that allows for fast parallel download streaming to stdout.
Don’t hesitate to reach out if you’d like further help tuning your downloads. We are happy to hop on a call as well and help tune/debug anything.
