Using Storj to share blockchain snapshots

Hi all,

We have created a blockchain data snapshot for BNB Chain for Erigon full nodes and detailed this process in a lengthy blog post.

The BNB Chain snapshots are now publicly available to download from the Storj bucket. We estimate using Storj is ~10x - 20x more efficient (and easier) than Amazon S3.

In the post, you can find

  • What is BNB Chain
  • What is Erigon (one of EVM-compatible node applications)
  • What is Storj and its decentralised storage
  • What is a chain snapshot, and why are they important
  • Compressing 4 TB data to 1.2 TB using Zstd, and notes and ideas of the compression ratio of blockchain data, Zstd, ZFS
  • How Storj is optimal (cost, efficiency) for distributing these kinds of files
  • Instructions on how to set up your own Erigon and use the snapshot
  • Instructions and scripts on how to set up your snapshots stored on Storj

If someone at Storj is interested in this, we would be happy to see if there exist any opportunities for case studies or co-promotion.

16 Likes

Amazing. :heart_eyes: :star_struck: :heart_eyes:

That sounds like a great use case! While someone else might reach out about opportunities, from the technical perspective, let me suggest these great posts Hot Rodding Decentralized Storage Part 1 Hotrodding Decentralized Storage.

In your post, you’re reaching ~13MB/s when downloading using uplink binary ran on OVH’s servers. This indeed might be OVH’s capacity limit. Still, you might get even better results (we’ve seen hundreds of megabytes per second on good internet lines) if you optimize some configuration, starting with parallelism :slight_smile:

5 Likes

To go a lot faster replace

uplink cp sj://bnb-chain-snapshot/bnb-chain-snapshot-2023-23.tar.zst .

With

uplink cp --parallelism 16 sj://bnb-chain-snapshot/bnb-chain-snapshot-2023-23.tar.zst .
4 Likes

With --parallelism 16 I am getting a 5x download speedup. I updated the blog post to reflect this change.

I recommend making this option a default :sweat_smile:

7 Likes

Your document states that Storj runs on its own blockchain model, but for the interaction, the user does not need to know about the blockchain at all.

My understanding (although I’m happy to be corrected) is that, whilst Storj used the Ethereum blockchain for hosting its STORJ token used for the transactional and economic model of the company, the storage technology itself is not in any way blockchain-based as you seem to imply :slight_smile:

8 Likes

I confirm Storj does not use blockchain for data storage

2 Likes

Hi @miohtama – great to meet you. It would be awesome to find a way to promote your blog post and findings to our wider community. Can you send an email to kevin@storj.io and we can start a conversation!

2 Likes

Storj does not run its own blockchain, but it is a token on the top of Ethereum network.
STORJ token is one thing, Storj distributed storage is another, different ecosystems. Only similarily is that Storagenodes are paid in STORJ tokens.