On the impact of fsync in the storage node upload code path on ext4

Had similar tests being performed for btrfs, waiting for analysis… and finally found some time. So, yes, the effect for btrfs is even more dramatic.

For reference, I’ve created the file system with -d single -m single, mounted it with -o nodatacow,max_inline=0,noatime. I found these settings to be the fastest in prior experiments. While they disable a lot of btrfs features, they make the comparison fair to ext4 feature-wise. Given ext4 is believed to be good enough for a storage node, btrfs with these settings should be as well.

Showing just the totals, as all batches exhibit the same properties here:

  • Without the suggested change: 188209 seconds. No sd here, I’ve run this case only once, as the performance was so pathetic I didn’t bother doing more tests. This is more than twice slower than ext4 without the change and so far the most basic reason for not recommending btrfs for storage nodes.
  • With the suggested change: 34480 seconds, sd=195 seconds. Pretty much the same as ext4 with the change! Curiously, variance is also lower than ext4.

I’ll repeat that I don’t know what kind of failure modes would this change introduce on btrfs (nor any other file system except for ext4, which I studied somewhat carefully at some point). btrfs also has the commit mount option, so maybe it would be safe enough? Anyway, this change would make nodes on btrfs quite workable (du is still much slower though).

2 Likes

so far as i know BTRFS is still a bit of a mess, so not really recommended for production.
aside from that running it without Copy on Write kind of defeats one of the major advantages of the filesystem.

sure it won’t be as fast, but it also is unlikely to mess up your entire data store due to an unexpected power outage, which then needs to be mitigated in hardware instead.

if one doesn’t want to roll the dice.