Any Update On Further Decentralizing The Network / Satellites?

[Shmigamie]

In February 2020 there was a post titled “Single point of Failure if a Satellite is down?” and what interested me was the reply that jtolio gave. He stated and I quote “In the long term, we plan to architect the Satellite out of the platform. We hope to eliminate Satellite control of the metadata entirely via a viable Byzantine-fault tolerant consensus algorithm, should one arise. The biggest challenge to this is finding the right balance between coordination avoidance and Byzantine fault tolerant consensus, where storage nodes can interact with one another and share encoded pieces of files while still operating within the performance levels users will expect from a platform that is competing with traditional cloud storage providers. Our team will continue to research viable means to achieve this end.”

Its been four years since this comment was made, so I wanted to ask if the Storj team has made any progress in finding a solution that would architect the satellite out of the platform for a more decentralized approach.

Another issue that was mentioned overall in this thread was " if the satellite goes down then the customer cannot access his data, even though the nodes are online. If something bad happens to the satellite, the customers may lose access to their data, even though the data itself is still on the nodes."

I have a question, but keep in mind I don’t come from a dev or computer science type back end so this question may not actually make sense. However, I realized the storj itself is a ERC-20 token. Meaning the digital asset itself just runs on the Eth chain while Storj has its own separate network. I was wondering if it was somehow possible to combine a platform like Arweave and Storj together. Since Arweave claims to be decentralized and a permanent storage network, the forked Arweave chain could act as an ultimate decentralized Satellite that will store the data forever (I think it could atleast) and this would benefit the Storj network part of the whole operation. The token could also live on the Arweave forked chain. As mentioned, I don’t know if technically it works but was just curious on any comments that can explain if it can or cannot.

[Alexey]

Hello @Shmigamie,
Welcome to the forum!

@Shmigamie

We hope to eliminate Satellite control of the metadata entirely via a viable Byzantine-fault tolerant consensus algorithm, should one arise.

Nobody invented it yet so far. So you still need to select any two from these three: fully decentralized, enterprise grade, economically sustainable.

The Electric Car Example Applied to Decentralized Cloud Storage

Pick 2: Applying Lessons from the Electric Car to the Path to 100%...

However we opened a Community Satellites program:

Announcing Community Satellite Pilot Program

Hello Storj Community! The final incomplete task from the v3 whitepaper is getting Satellites run by the community, and we’re ready to start that process! Our end goal is to allow anyone to run their own Satellite if they are so inclined, for maintaining and managing their own data against the public storage node network. We want to make sure that community Satellites are run well and are good partners with our storage node community. We see this progressing in phases. There are large complic…

So now you may join the Community Satellites or run your own, however, each of them will be represented as a separate network.

@Shmigamie

However, I realized the storj itself is a ERC-20 token. Meaning the digital asset itself just runs on the Eth chain while Storj has its own separate network.

The Storj token is an utility token used only to pay for the service and get 10% bonus to your account if you choose this payment method or being paid as a Storage Node Operator for the used shared resources.

@Shmigamie

Since Arweave claims to be decentralized and a permanent storage network,

This is a very expensive and not so fast as needed to be an Enterprise Grade, it’s very slow compared to Storj, because they still need a consensus, which slowdown any interactions with the network.
See

Is it possible to link the uploaded data with the mined block hash value?

Storing data on blockchain is too expensive and inefficient as for upload and for the access. The only small chunks of data can be stored on blockchain and also in most cases they are immutable, i.e. you can’t delete or edit it. To make change you must generate the next block with that change. It’s not fast. And also the size of the blockchain will include all world data, at the moment it’s greater than 460 Exabytes. And you would need to store it on the each node in the network, because blockc…

@Shmigamie

the forked Arweave chain could act as an ultimate decentralized Satellite that will store the data forever

The Satellite doesn’t store data, data is stored on nodes. The satellite is an address book and bookkeeping, also auditors and repairs. It’s a distributed service similar to storage nodes, just have less instances than storage nodes and working in the trusted network to be blazingly fast.
The main problem is to have an encrypted distributed and fast database, which can be run on Byzantine nodes. It’s not invented so far. The second problem is to have a distributed and encrypted compute service, which can be run on Byzantine nodes too. There are attempts to implement it, but all of them are not secured from the access by the operator, thus you can run only miners or process the non-commercial or public data on a such fog compute, as it called.
I hope, that someday we would speedup our nodes even more than after 1.104.5 release to be able to use them as nodes for a such database (but it also requires a compute, and this is a big challenge to keep the processing data in safe, when no one except the data owner can access or dump it from the RAM).

We use a highly distributed and fast Cockroach database, but it requires to be run in the trusted environment.

[Mitsos]

@Alexey doesn’t the satellite store inline data if they are too small to be worth uploading to nodes?

[Alexey]

Yes, it’s. Together with a Metadata, i.e. in the database…

[Shmigamie]

Hi, thanks for the reply and welcoming me to the forums. I understood alot of what you said (some definitely maybe too complicated for me, but that is fine as I get the overall message).

[pangolin]

@Alexey

Yes, it’s. Together with a Metadata, i.e. in the database…

I am confused now… Does it mean small data files from customers are not uploaded to nodes?

[nerdatwork]

@pangolin

Does it mean small data files from customers are not uploaded to nodes?

Files less than 4KB aren’t uploaded to nodes for obvious overhead.

[Alexey]

Understanding Hierarchical Data Structure and Advanced Terminology - Storj Docs

[pangolin]

@nerdatwork

Files less than 4KB aren’t uploaded to nodes for obvious overhead.

Honestly, this feels like a big no go. As a customer I would expect, to have the same level of redundancy for all my data.

[Toyoo]

@pangolin

Honestly, this feels like a big no go. As a customer I would expect, to have the same level of redundancy for all my data.

The redundancy of satellite data is actually better than storing pieces on nodes.

[pangolin]

@Toyoo

The redundancy of satellite data is actually better than storing pieces on nodes.

Why? Please explain.

[Toyoo]

Because if a satellite loses its database, you can’t recover pieces stored on nodes either.

[Alexey]

This is not a problem, the satellite is a distributed service as well, as its database across the world. Less instances doesn’t mean that it less reliable than the storage nodes, because unlike nodes the satellite’s components are running in datacenters across the globe. When there would be a reliable secure solution to run them on Byzantine nodes - we are happy to migrate!

[Alexey]

It’s simple. The satellite components are running on a datacenters hardware with 24/7 monitoring and support with a redundancy and guaranties not less that we provide for the accessibility. They also have backups and all compliance rules of SOC2/ISO 27001 are filled. In the edge cases we can restore data from nodes, if we could manage to break everything.

The distributed database is running by a different way and separated from the compute components of the satellite, and also SOC2/ISO 27001 compliance. So it’s much more robust than any average node.

The only thing which can kill it - if we would introduce a bug. But for that we have tests and QA engineers, so the probability is low. And we provides SLA, as explained in our ToS.