# Design draft: a low I/O piece storage

**URL:** <https://forum.storj.io/t/design-draft-a-low-i-o-piece-storage/24864>\
**Category:** Product Discussions\
**Created:** [January 7, 2024, 7:19pm UTC](https://forum.storj.io/t/design-draft-a-low-i-o-piece-storage/24864 "2024-01-07T19:19:38Z")\
**Posts on this page:** 1\
**Showing post:** 14

<div class="post-metadata">

**Author:** ![thepaul](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/thepaul/32/65_2.png) [@thepaul](https://forum.storj.io/u/thepaul)\
**Post date:** [January 8, 2024, 7:17pm UTC](https://forum.storj.io/t/design-draft-a-low-i-o-piece-storage/24864/14 "2024-01-08T19:17:58Z")

</div>

This is great. It’s a really well put-together blueprint. We’ve discussed using journaling before but no one has fleshed the idea out with this much detail. I didn’t even know `FALLOC_FL_COLLAPSE_RANGE` existed; that certainly makes this approach more interesting, at least for the Linux side.

I expect I (or someone) will be looking at this more in the weeks to come. It might need to compete with a proof-of-concept experiment using BadgerDB.

> [@Pentium100](#):
>
> > [@Toyoo](#):
> >
> > I believe it would still be way within the acceptable 2% level of losses in case of problems, with the only operation that is more risky being the compaction procedure.
> 
> If I got this right, a small node could lose 2% of data due to a single crash?

No, I believe xe meant that in a reasonable situation data loss would be much _less_ than 2% (2% being the level of data loss where node reputation begins to suffer sharp penalties).

> [@Pentium100](#):
>
> I mean, if there are two files being uploaded at the same time, how does the node decide the order to write them to the pack file? You can write multiple files to a filesystem at the same time, no problem, but trying to append two sets of adata to a file at the same time could result in problems.

This is the reason for keeping each uploaded piece in memory until it’s committed, at which point it gets written to the journal. Only a single thread is needed to write to the journal, since the full contents of each input will be immediately available.

---

_[View the full topic](https://forum.storj.io/t/design-draft-a-low-i-o-piece-storage/24864)._
