Badger cache: are we ready?

[Alexey]

We have had this in the past, we stored data in KFS

Design draft: a low I/O piece storage

Perhaps the most efficient storage was based on LevelDB in Storj v2 (Introducing KFS: a local file store inspired by Kademlia), but it has had a hard limit of max allocated size per node, and since it was a custom solution, the repair tools for LevelDB didn’t work well. It also used more memory to operate a node (exactly like you suggests) and we have had many complains about high memory usage.

And we investigating of possibility to store pieces in alternative systems:

Design draft: a low I/O piece storage

Thanks for this design doc @Toyoo, It’s a well written design, I like it. And finally I had some time to collect my thoughts. TLDR; I like the approach, but I am not sure about the implementation. So far I have heard about multiple attempt to improve the piece store. They are usually save the metadata of pieces in a database like system and save the raw data in custom files. (technically it would be possible to store both metadata and data in database, but it’s quite slow. I tried it, and mea…