Ext4 + LVMcache, some numbers

Node Operators
, ,
1

While restoring my nodes after a recent PSU failure, taking opportunity of having downtime anyway, I took some time to do measurements on effectiveness of LVMcache for ext4. Note here: I’m using my ext4 settings, which are already more optimized than defaults! I’ve effectively recreated my LVMcache from scratch, then ran fsck and du -hs to fill the cache. Then, to do measurements, I dropped all caches with sysctl vm.drop_caches=3 and ran du -hs again without anything else running on the whole system. I have two cache drives, one NVMe, one SATA, so tested both. All units SI. I’ll drop raw numbers for now, as I don’t have much time to look at them in more depth, but in general—I’m absolutely fine with these numbers.

Raw numbers:

  1. A 460GB storage directory, 1752456 pieces, took 45 seconds to do du -hs with NVMe-based cache.
  2. A 410GB storage directory, 1802916 pieces, took 56 seconds to do du -hs with SATA-based cache.
  3. A 19871 GB file system with 43174332 used inodes with a cache of 327 GB is using 7.86% of cache storage for data, 18.63% for metadata (not sure what metadata entails—my quick guess is some map of which part of cache stores which part of the file system).
  4. A 19871 GB file system with 42711506 used inodes with a cache of 305 GB is using 12.02% of cache for data, 18.84% for metadata.
  5. A 17864 GB file system with 42655376 used inodes with a cache of 274 GB is using 14.54% of cache storage for data, 18.82% for metadata.
  6. A 7781 GB file system with 22042692 used inodes with a cache of 116 GB is using 16.94% of cache storage for data, 18.15% for metadata.
  7. A 5878 GB file system with 14264066 used inodes with a cache of 90 GB is using 10.72% of cache storage for data, 17.58% for metadata.

fsck took between 1.5 and 3.5h, with cache already attached, but obviously being filled up by fsck. I didn’t note the exact numbers, sorry!

As such, I could probably reduce the size of cache to around ⅓ of what I have allocated. Coincidentally, my caches before the downtime were around half of what they are now (I’ve added the NVMe drive as part of bringing the system back), and while I haven’t made any precise measurements back then, the cache was working fine too.

Have fun!

5 Likes