Don’t forget about Filecoin for important backups, guys

21

Point 1: Your personal success, or anyone’s at anything in particular, does not mean that whatever it is that was successful at the time will remain viable in the future.

Point 2: At the moment there are a total of around 750 Filecoin providers. That’s not very many. The Filecoin network is decentralized but siloed into massive servers. The incentives encourage those running massive servers to become more massive. I seem to remember that Filecoin has a bonus for more capacity.

Point 3: If one was deciding on the viability of a data storage network solely based on the raw capacity, AWS wins. So, the metric needs to be something else.

Point 4: The de-duplication of data on IPFS is also its weak point.

Storj and IPFS are not drop-in replacements. IPFS is really neat, but so is a Logical Volume that spans 200 spare drives. If one wishes to store a lot of data inexpensively, one could simply run an IPFS node… there’s no need to mess with a entire blockchain dedicated to a contractual agreement.

If you had used your STORJ to buy 15 ENS domains using 15 different addresses instead of letting the tokens sit, you could have earned a few hundred thousand USD during their airdrop too…

Financial success in the crypto world is still quite a lot of luck. However, my prediction is that Storj is one of those rare projects that will continue to grow and find success for a long time to come…

EDIT:

This post is as true today as it was in 2020:

https://forum.storj.io/t/storj-vs-filecoin/9760/4

Centralized Silos are baked into the reward system:

https://spec.filecoin.io/systems/filecoin_blockchain/storage_power_consensus/

Quality-Adjusted Power is the number of votes a miner has in the Secret Leader Election and has been defined to increase linearly with the useful storage that a miner has committed to the network.

More precisely, we have the following definitions:

  • Raw-byte power: the size of a sector in bytes.
  • Quality-adjusted power: the consensus power of stored data on the network, equal to Raw-byte power multiplied by the Sector Quality Multiplier.

Summary:

Storj has decentralization baked in. Data pieces are distributed throughout the network in such a way as to flatten data storage across the network. This generally means that large storage nodes are somewhat as likely as small nodes in obtaining traffic. Since the Storj network has duplication of data and error correction capabilities, the “filesystem” is more like ZFS than a RAID 0.

Filecoin has centralized baked in. Data are stored in ever increasingly lower numbers of larger nodes. Since there are no error correction capabilities, the “filesystem” functions more like RAID 0 or a LV spanning a odd collection of drives than something robust like ZFS.

STORJ as a token exists on one of the most popular blockchains in existence. And is directly portable to secondary blockchains running Ethereum.

FIL as a token exists on a dedicated blockchain. While wrapped tokens can be used to track the value, FIL only has “utility” on its dedicated blockchain. This means that FIL is susceptible to being trapped by on or off ramp pathways… Centralized crypto exchanges may arbitrarily shutdown FIL blockchain access. This is also true of Polygon. However, Polygon is not a dedicated blockchain with a singular purpose in mind. So, the likelihood of Centralized Exchanges shutting down access to Polygon becomes less likely over time - while access to FIL blockchain becomes more likely to be shut down if the network becomes stagnate.

All things considered, it’s far more likely that Storj will have long term success than Filecoin. Only time will tell.

I wish @stuberman and any other FIL enthusiasts much success in Filecoin. However, I like the fair distribution and ease of participation of Storj. I strongly suspect that the differences I’ve pointed out above between Filecoin and Storj will mean steady growth and success of Storj and stagnation of Filecoin.

4 Likes
22

I have to disagree with most of your points, particularly what network is decentralized in reality (centrally controlled satellites versus full user control)). I am deeply familiar with both Storj and Filecoin as a node operator, so I speak from experience not an emotional one.

23

Which value?The value of storing your data where ever you want, at terms you choose with the ability to negotiate with on or more providers? Or do you mean being free at the moment as a marketing subsidy?

24

OK… address my points then… As you requested, let’s have a discussion. I’m more than happy to have an evidence based discussion of the merits of multiple solutions to decentralized storage of data. But when I put my thoughts in words, you exited the discussion.

EDIT:

Here’s an honest question…

According to the documentation on making a deal:

https://lotus.filecoin.io/docs/storage-providers/manage-storage-deals/

Deal prices are set per GiB. The example in the documentation is 5GiB.

Since I am not familiar with running a Filecoin node, and you do run a node yourself… why are so many of the deals in the blockchain either precisely 2GB or 32GB?

https://filfox.info/en/deal

Screenshot_2022-02-05_16-09-35
Screenshot_2022-02-05_16-09-35898×603 63.8 KB

Pages and pages and pages of 2 GB and 32 GB.

Why are there no 3GB or maybe 7GB or maybe even 120 GB ?

The pages and pages of deals consisting of precisely 2GB and 32GB deals gives every impression of deals being made for the appearance of there being deals being made rather than actual network use. Is that test data? Why 2GB and 32GB precisely?

Next honest question…

What does this graph mean?

https://filscout.com/en/power

Screenshot_2022-02-05_16-13-44
Screenshot_2022-02-05_16-13-441393×510 51.1 KB

It looks like a graph of decreasing network usage as well as decreasing node participation. But I’m unsure exactly what I’m looking at here… The linear growth of Network Power with a step decline in Network power growth in the last 180 days would seem to indicate a stagnation of node providers with Network Power growing as the existing nodes increase storage space… But again, I’m not sure if I’m looking at the graph correctly.

Third honest question…

The network stats listed here:

https://storage.filecoin.io/

have been nearly stationary for a full 24 hours. Is there a place I can find the historical data on providers? 724 at the moment with 740 clients… this is exactly the same number of both providers and clients 18 hours ago.

1 Like
25

I already started with the reality of decentralization. Having centrally controlled satellites is anything but decentralized whereas Filecoin is fully decentralized, anyone can put their data on the network using any client anywhere and send data directly to any provider of their choosing without having to go through a central authority. The mechanisms are transparent and you can choose as much replication as you want, some gateway services, run by independent parties will also encrypt and shard your data if desired.

26

OK…

I posted some questions in my above post.

File replication is not the same as error correction. From what I’ve read, the Filecoin blockchain might be able to detect lost data, but there’s no repair mechanism built-in to the architecture.

Is what I just wrote true?

The sharding and encryption by independent third parties… that sounds a LOT like running Storj. I would imagine that the independent third parties are individual entities and therefore NOT Decentralized. I would also imagine that the independent third parties require some kind of payment above and beyond the payment for a typical Filecoin deal.

So, how does one compare the services offered?

It seems that one could have something similar to Storj but running on Filecoin… of course, minus the ability to correct errors and repair files that may have been damaged through various Filecoin storage providers mismanagement of the files.

And lastly, what’s the cost of simply forgetting about Filecoin and running an IPFS gateway for one’s own files?

I run my own IPFS. That means I have full control over the host filesystem… which means my IPFS doesn’t lose files… and I can trust it as completely as I trust my host OS and FS. The extra cost for me running my own IPFS node is zero.

So… why would I choose to use Filecoin storage if there’s no error correction or file recovery on the network… while I can have all the space I need while also ensuring both error correction and file recovery via my host OS?

27

New post to include some ongoing network stats on Filecoin…

In the last 60 minutes the number of unique CIDS increased by 282 while the number of deals increased by 578.

I’m not sure if CIDS refer to data stored… as in CIDS on IPFS. If it does than, in the last 60 minutes the unique data on the network increased at about HALF the rate of client access to the data on the network. This would seem to indicate either test data or “popular” data. Also 578 deals made in the last 60 minutes sounds extremely slow…

Maybe I’m not reading the blockchain stats properly.

@stuberman I need some help understanding the network stats of Filecoin. Perhaps, as in many areas of life, I’m missing some important point… just ask my spouse about that too. :wink:

EDIT:

So is Estuary one of those Independent third parties?

If so, what about these errors on some of their nodes:

https://estuary.tech/providers/errors/f062353

I recognize the IPFS node ID format in the error list. There’s also an Internet accessible IP address listed in the errors. This is quite interesting…

I remember when Storj cut access to third parties for the SNO network precisely to protect the privacy of SNOs. I suppose Estuary might want to figure out how to remove IPFS node IDs as well as Public IP addresses from their error records.

Estuary’s website indicates that it’s in Alpha stage. So, I’m not sure one can purchase the encryption and sharding services AND !file repair! from them yet… is there another vendor? That way I can use it to figure out the cost of recreating Storj on Filecoin…

EDIT 2:

More notes and questions…

There was a recent thread on this forum regarding an EU court ruling and Google and privacy issues and IP addresses…

With that in mind, what about this Estuary node Identity information listed in a public space:

https://estuary.tech/providers/stats/f02620

If one looks up the IP address listed, it’s in Poland. I wonder how the EU courts might deal with such information being so publicly detailed.

The more I dig into Filecoin, the less it seems like a finished product… unless I’m totally completely misunderstanding what I’m looking at… which is highly possible.

28

Partly. File replication is not the same as error correction is true. Filecoin does daily proving/auditing of all sectors and there is a repair process that is now in place.

29

Not every service offered by third parties is decentralized that is true of Storj as well. I still do not see how a claim can be made about Storj be truly decentralized as a system (the Storj network). Incentives vary from party to party in some cases this allows that third party to add a fee so that the cost is not $0 per TiB stored. At pebibyte scale even $1 per TiB adds up.

30

I do not see your point here. Storj has a very different architecture than Filecoin. As already mentioned you are incorrect about repairs and errors.

31

Where in the Filecoin documentation can I find information on the standardized file repair process that holds true across the network and is tracked by the blockchain?

EDIT:

New Question…

The total size of the Filecoin Blockchain vs. the total size of the Full BTC and/or ETH blockchains…

Ethereum and Bitcoin:

https://blog.bitmex.com/bitcoin-vs-ethereum-blockchain-size/

  • Ethereum full chain
    • this large 9TB dataset as the amount of data required to follow the flow of funds at each point following on from Ethereum’s genesis.
  • Bitcoin full chain
    • Bitcoin’s 400GB blockchain

Filecoin:

https://lotus.filecoin.io/docs/set-up/install/

  • Filecoin full chain
    • The full history was around 10TiB in June of 2021
    • The chain grows at approximately 38 GiB per day

Bitcoin grows about 50 GB per YEAR.
Filecoin grows about 40 GB per DAY.

There’s information about trimming the Filecoin blockchain to the minimum required for syncing. However, eventually the Filecoin blockchain seems quite likely to be unmanageable for the even the fastest hardware if enough deals are made per reasonable time period.

It’s interesting that the organization developing IPFS is the same organization developing Filecoin, but it is claimed that Filecoin is not IPFS centric.

I like IPFS, but the more I read about Filecoin, the more it seems like hype chasing hype rather than a real solution priced at a real market value.

Again, time will tell…

I’m still searching the documentation for the standard method being deployed network wide on Filecoin for file repair and repair tracking using the blockchain…

EDIT 2:

Found one answer…

https://docs.filecoin.io/about-filecoin/ipfs-and-filecoin/

While interacting with IPFS does not require using Filecoin, all Filecoin nodes are IPFS nodes under the hood

OK… some things are falling into place logically speaking.

EDIT 3:

Found answer 2… !repair!

brought to you by Powergate:

https://docs.textile.io/powergate/

By using the Powergate to persist your data on Filecoin, you gain access to rich storage configuration options such as:

  • Replication factor
  • Miner selection
  • Deal renewal
  • Repair

But beware of the disclaimer:

Do not run the Powergate in production systems and please join the powergate-users channel in the Filecoin community Slack for announcements and support. The Powergate will remain in rapid development until a formal release. During this time, you’re likely to encounter bugs and unannounced API changes.

OK Then…

Got it.

Filecoin is currently operating at Storj Alpha level — maybe a little below as far as a robust Filesystem is concerned.

It’s good to see that repair is being considered, even though it’s definitely not a component of the main architectural design. At this point, I think I’ve read enough to state that Filecoin is definitely not a primetime ready Filesystem for robust storage of mission critical data. Perhaps it will get there one day. However, Storj is light years ahead in terms of practical implementation of a robust distributed filesystem that is readily and easily deployable for mission critical data.

2 Likes
32

Apples and oranges. Anyone can run IPFS, it has little to do with Filecoin other than compatibility. Storj had an IPFS gateway a few years ago. It is meaningless to conflate IPFS with Filecoin. Again, I have corrected your error about repair. And more importantly, “why would I choose to use Filecoin storage” is a much better question answered in short by a few aspects: lower cost; much more user control of their data, far more nodes from which to choose; and daily proof of data integrity.

[I guess the forum is set to lock people out after so many responses - it forced me to wait to add this reply]

33

No, Estuary is run by Protocol Labs. There are many independent third parties engaged in the ecosystem, such as Textile.io Some offer sharding and encryption and some don’t it really depends on their customer base. The first phase of Filecoin was providing files to the general public that were not encrypted, this includes things like the Internet Archive, where anyone that had a CID (including those going to a web site with a link and a browser like Brave Browser with native support for IPFS) could access those files. Now the network is also supporting enterprise customers with other requirements.

34

See this: https://slingshot.filecoin.io/repair and https://slingshot.filecoin.io/restore
I expect this to be a standard feature in an upcoming release.

35

Let’s pull this apart a bit.

  1. “Correct my error about repair”

No you didn’t. You stated, without any example or reference, that Filecoin now has error correction. I searched around and found a reference for an experimental Alpha release of a product that indicates error correction is a component of their Filecoin gateway.

I have not found any documentation on Filecoin’s website that indicates that error correction is an officially supported feature in Filecoin’s architecture. Please provide that reference if I missed it.

  1. lower cost…

How so? The actual market costs of using Filecoin are currently unknown because much of the storage space is being offered at a loss in order to grow market share. And the network has too few clients in order for the market to set a somewhat stable price.

  1. much more user control of their data

I disagree with this one. Storj has user data encryption as a standard component. Therefore, no one except the data owner has access to the raw data. Filecoin has no such standard component, so data stored via Filecoin is subject to third party interception and inspection. Furthermore, there is nothing stopping anyone from setting up an independent Storj satellite for private network use operating outside of the official Storj implementation. Filecoin is dependent on the blockchain, there’s no way to deploy Filecoin as a private network.

  1. far more nodes from which to choose;

This is just blatantly false. Filecoin has two orders of magnitude fewer storage nodes than Storj.

  1. daily proof of data integrity.

This is partially true. The Filecoin blockchain has mathematical proof of a file being stored. However, there is no guarantee that the file will be there tomorrow. The error correction is not standard and depends on Alpha level implementation of a specialty gateway. If any random Filecoin node is utilized, no such error correction exists… and therefore, the daily proof is utterly worthless since there’s no way to ensure the continuation of the data integrity.

Again, I’m not saying that Filecoin isn’t profitable at the moment for the very few miners in the network. But… Storj is truly light years ahead in architecture. In fact, Filecoin is starting to add on the missing components in order to become more Storj-like… that’s not by accident.

I’m all for a discussion, even somewhat heated ones… but please post references to some kind of documentation rather than simply stating something.

5 Likes
36

That is your judgement, not customer’s. Today Filecoin stores more than 33 PiB of real data with a current network capacity of more than 15EiB across almost 4,000 nodes. This year are plans to onboard more than 1EiB of real data by real customers. Sure there is a lot of development as it is an open ecosystem driving innovation including development of the VM that will allow compute on data, how long did Google maintain that Gmail was only in beta? [5 years]

37

33 PiB and 740 clients.

Seems like a very broad base of clientele.

It’s been nearly two days since I’ve been looking at this page:

https://storage.filecoin.io/

It’s still 724 nodes and 740 clients. The growth of the number CIDs seems rather linear and rather slow. The growth of the number of clients is currently zero.

Based on nearly 48 hours of actual network stats, I can conclude that the Filecoin network’s massive unused capacity is not flooded with new clients looking for inexpensive storage.

38

Indeed, let’s discuss these since the issue is accuracy about your statements.

  1. I did provide links earlier today, you are just being overly aggressive here. (See my statement about this forum responses.
    Storj lockout
    Additionally you apply your own requirements, data is audited daily, errors cause a fault, therefore storage providers have an incentive to repair or restore bad data or get penalized. The system works, albeit without your insistence on how it should work.

  2. Lower cost. Absolutely true. A client today is able to store data at $0 - $0.03/TiB on average. Pricing is market based, not based on some number set by a central authority. True, the block reward incentive subsidizes the early stage of growth but this is already decreasing as designed.

  3. Tell me if a customer using Storj can decide where their data will be stored (geographically). who will store it, what the terms will be? I can make an agreement with a Filecoin storage provider to exclusively store my data in the type of data center (or home) I require, at the price we negotiate and with access control set according to my requirements. THIS is user control, you don’t get to play with definitions accepted in Web3.0. Frankly someone can set up a private Filecoin chain, it is done quite often if only for testing, having your own for with few people on it is rather pointless but possible.

  4. Storj has 12,700 vetted nodes. I run 3 ‘nodes’ which are in fact three hard drives with a total capacity of 35TiB. Clients cannot choose which Storj nodes to store their data on, it is simply not part of the Storj design and architecture. Filecoin has nearly 4,000 active nodes, however a node in the Filecoin ecosystem has a very different definition. My single Filecoin node has a current capacity of 170TiB and can be expanded greatly. My node size is a bit smaller than the average Filecoin node. So ‘blatantly false’ is silly and avoids the real issue. If a client wants to store 100PiB, Storj is simply not an option since the Storj network capacity is around 16PiB. And we see clients like this working with Filecoin providers.

  5. You make several poor assumptions. In Filecoin the ‘guarantee’ is based on collateral incentives of the system, a provider that loses data loses money (FIL). Important data is often stored in multiple replicas by clients (who get to decide that). Repair and restore will be automated soon. Unlike Storj, Filecoin is an open and transparent ecosystem which allows many service providers to act as brokers and tool developers providing whatever services the customers need in the system. Your reference to a single Estuary service does not factor in these other brokers that provide services. Calling the proof of storage ‘utterly worthless’ is silly since integrity is ensured by the incentive system. It has proven itself as storage providers have shown they have recovered corrupt data (there are many methods from keeping a backup to using other repair and restore techniques.)

I would say that nearly 4,000 miners is not ‘very small’ and the profits are extremely good with a high probability the network will continue to grow very quickly with real customer data. There is a difference between Storj and Filecoin, as I have pointed out for over a year in this forum, I would really say you are comparing apples and oranges, with Filecoin much more capable at an enterprise level and offering more of the real Web3.0 capabilities than the centrally controlled version that Storj offers. Don’t get me wrong, I like Storj for what it does, simple, set and forget and a very small but steady revenue stream. If I were to turn off my Storj machine with three nodes, I would be forfeiting $75 in held payments. If I shut off my Filecoin node, I would be forfeiting hundreds of thousands of dollars in held collateral, which I am not going to allow. True a part time hobbyist with little technical knowledge is not suitable to running a Filecoin node.

You don’t have to acknowledge the references and links I post, but these are not empty claims.

39

Where is this number?

This Filecoin stats page:

https://file.app/

lists 4000+ miners but also provides a “filtered” list of 671 miners that:

“Updated on Sun, 06 Feb 2022 12:55:00 GMT. Requires successful storage deals to be listed.”

So, that 671 miners actually storing something and more than 3300 miners storing nothing.

In other words:

“There is no standard error recovery implemented within Filecoin.”

40

Actually you are not interpreting those number properly. Use this link for raw statistics.
There are 724 unique providers, and 3,836 workers (nodes). A unique provider is a person or organization often running multiple nodes around the world. The number of clients is not fully represented on this page as they list the sources of their client data, other brokers and storage providers cutting deals with clients is not part of that data set. For instance, a single Lotus client will be seen as one client on the chain but they take in deals from many customers who are not identified. This number (724) does not include storage providers with network capacity available to add data. In the next few months the network is being updated to convert ‘committed capacity’ sectors to quickly accept deal data.

So let’s try to compare the growth of Storj to Filecoin if that is what you are looking for.
Storj real data 9.56 PiB in the last 30 months?
Filecoin real data 33 PiB in the last 18 months.

As I mentioned earlier, the network is now going to kick data ingestion into high gear with programs tailored to enterprises and other large organizations.