# Ext4 Optimization recap

**URL:** <https://forum.storj.io/t/ext4-optimization-recap/22883>\
**Category:** Node Operators\
**Tags:** linux, hardware\
**Created:** [June 7, 2023, 11:08am UTC](https://forum.storj.io/t/ext4-optimization-recap/22883 "2023-06-07T11:08:26Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![agente](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/agente/32/1137_2.png) [@agente](https://forum.storj.io/u/agente)\
**Post date:** [June 7, 2023, 11:08am UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/1 "2023-06-07T11:08:26Z")

</div>

Small operators growing and one day they will realize that need to optimize entire system to survive.  
Can we have a recap on suggested optimization on a EXT4 (more common filesystem in storj ecosystem) based server?

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [June 7, 2023, 7:06pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/2 "2023-06-07T19:06:24Z")

</div>

Why would you need to optimize the file system? As far as I can see, the file system is almost never the most constraining factor. In my case the download speed on average doesn’t exceed 1MiB/s, which also never saturated the disk cache.

---

<div class="post-metadata">

**Author:** ![arrogantrabbit](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/arrogantrabbit/32/11200_2.png) [@arrogantrabbit](https://forum.storj.io/u/arrogantrabbit)\
**Post date:** [June 7, 2023, 7:16pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/3 "2023-06-07T19:16:51Z")

</div>

Download speed is irrelevant. IOPS is what matters. 1Mbps of 1k files is _at least_ 125 IOPS, even if you don’t count atime updates, metadata writes (mostly batched) but it adds up — hard drives cannot do more than 200, so even today it’s pushing it.

Therefore you definitely want to tune for the usecase — at least disable atime updates, disable sync (especially for databases, or move databases to SSD if you don’t have reliable power and have to rely on sync), maybe tune sector size, etc. but I’m not expert on ext4, let’s someone more knowledgeable chime in.

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [June 7, 2023, 7:46pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/4 "2023-06-07T19:46:52Z")

</div>

I see you are volunteering. Great, I’d love to see someone collect all these nuggets of information! You can probably start from just typing `ext4` into the forum’s search tool, I see at least two relevant threads in just the search preview.

---

<div class="post-metadata">

**Author:** ![agente](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/agente/32/1137_2.png) [@agente](https://forum.storj.io/u/agente)\
**Post date:** [June 7, 2023, 10:27pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/5 "2023-06-07T22:27:23Z")

</div>

My intention was to try to provide a general summary for everyone. As easy as possible. A defrag of forum info about fs managment 🙂

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [June 8, 2023, 8:37am UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/6 "2023-06-08T08:37:14Z")

</div>

Again, I doubt whether the filesystem itself poses a real problem. I have six nodes running, of which 2 rely on just one HDD. One pure SSD. And 3 with a mix of micro-SDs, HDDs and SSDs. I actually don’t see any substantial difference between all of them, all having a ingress of 30-50GB/day, with a grow of 15-20GB/day.

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [June 8, 2023, 9:33am UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/7 "2023-06-08T09:33:03Z")

</div>

Yeah, please do!

Though, I’d hope this kind of information could be included in the official documentation as well, I feel that’s where it belongs to.

---

<div class="post-metadata">

**Author:** ![agente](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/agente/32/1137_2.png) [@agente](https://forum.storj.io/u/agente)\
**Post date:** [June 8, 2023, 12:28pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/8 "2023-06-08T12:28:23Z")

</div>

I’m thinking about my 24hrs for filewalker on a normal sata server hdd with 10tb of data reading at 10/15 MB/s… (ext4 filesystem. l2arc on zfs I understand that helps a lot)

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 9, 2023, 7:08am UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/9 "2023-06-09T07:08:47Z")

</div>

Just made a dumb experiment.  
Created 2 virtual disks, connected them to the Linux VM (Ubuntu 22.04)  
Created lvm volume on the first virtual disk, formatted to ext4 and mounted to `/mnt/lvm-test`

```auto
sudo pvcreate /dev/sdc
sudo vgcreate vg /dev/sdc
sudo lvcreate -L 20G vg
sudo mkfs.ext4 /dev/mapper/vg-lvol0
sudo mkdir /mnt/lvm-test
sudo mount /dev/mapper/vg-lvol0 /mnt/lvm-test
sudo chown $(id -u) /mnt/lvm-test

```

Created zfs pool on the second virtual disk, created zvol, formatted to ext4 and mounted to `/mnt/zfs-test`

```auto
sudo zpool create test /dev/sdb
sudo zfs create -V 20G test/vg-ext4
sudo mkfs.ext4 /dev/zd0
sudo mkdir /mnt/zfs-test
sudo mount /dev/zd0 /mnt/zfs-test
sudo chown $(id -u) /mnt/zfs-test

```

Small dumb test

```auto
$ head /dev/urandom -c 1G > 1G.raw
$ rsync 1G.raw -P /mnt/zfs-test/
1G.raw
  1,073,741,824 100% 138.39MB/s 0:00:07 (xfr#1, to-chk=0/1)
$ rsync 1G.raw -P /mnt/lvm-test/
1G.raw
  1,073,741,824 100% 201.41MB/s 0:00:05 (xfr#1, to-chk=0/1)

```

So, if you want to have a faster disk - it should be without underlaying zfs. What’s the point to create zvol there? why not a native zfs dataset?  
here is my `test/plain` zfs dataset:

```auto
$ rsync 1G.raw -P /test/plain/
1G.raw
  1,073,741,824 100% 203.82MB/s 0:00:05 (xfr#1, to-chk=0/1)

```

See

```auto
$ rm /mnt/zfs-test/1G.raw
$ rsync 1G.raw -P /mnt/zfs-test/
1G.raw
  1,073,741,824 100% 141.39MB/s 0:00:07 (xfr#1, to-chk=0/1)

```

---

<div class="post-metadata">

**Author:** ![Pentium100](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/pentium100/32/6082_2.png) [@Pentium100](https://forum.storj.io/u/Pentium100)\
**Post date:** [June 9, 2023, 12:37pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/10 "2023-06-09T12:37:27Z")

</div>

I use a zvol because I run the node inside a VM and zfs is the underlying storage of the host. All VMs have their virtual disks as zvols.

For a raidz pool, you should create a zvol with 64K block size (the exact minimum differs, but to small block size on a raidz pool with 4K sector drives can make the zvol take up double the amount of space).

---

<div class="post-metadata">

**Author:** ![SGC](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/sgc/32/1132_2.png) [@SGC](https://forum.storj.io/u/SGC)\
**Post date:** [June 9, 2023, 3:01pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/11 "2023-06-09T15:01:59Z")

</div>

add another node to split the iops… problem solved…

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [June 9, 2023, 7:24pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/12 "2023-06-09T19:24:22Z")

</div>

Jup, but means also an additional drive and halfs the ingress per node. So probably doesn’t solve that much in the end, unless you already planned on starting more than one node.

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [June 9, 2023, 7:31pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/13 "2023-06-09T19:31:01Z")

</div>

Actually, I don’t think this experiment does say something about the use case of STORJ. Files are on average below 1MB, so not eligible for sequential writes. Also a lot more meta data writes.

---

<div class="post-metadata">

**Author:** ![Alexey](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/alexey/32/41_2.png) [@Alexey](https://forum.storj.io/u/Alexey)\
**Post date:** [June 10, 2023, 1:43am UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/14 "2023-06-10T01:43:44Z")

</div>

I’m agree with you. But you can do such test yourself with multiple files, the result will not change: zvol with ext4 will be slower than lvm/mbr ext4 and slower than zfs dataset.  
And also zfs dataset (on the simple single-disk pool) will be slower than lvm (single disk)/mbr ext4 for multiple files.

---

<div class="post-metadata">

**Author:** ![snorkel](https://storj.bcdn.literatehosting.com/letter_avatar/snorkel/32/6_449c533f586939b6e5f246632a46421f.png) [@snorkel](https://forum.storj.io/u/snorkel)\
**Post date:** [June 12, 2023, 9:23pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/15 "2023-06-12T21:23:46Z")

</div>

Are these limitations for ext4 on Synology NAS a problem for storagenodes? Is there a maximum alocated space recomanded for a node in these conditions?

 ![IMG_2715](https://storj-s3.bcdn.literatehosting.net/original/2X/2/261dbed055211ec2fa9642311023f64996331afb.jpeg)

---

<div class="post-metadata">

**Author:** ![JWvdV](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/jwvdv/32/7814_2.png) [@JWvdV](https://forum.storj.io/u/JWvdV)\
**Post date:** [June 12, 2023, 9:55pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/16 "2023-06-12T21:55:11Z")

</div>

I think these are huge problems, that can’t be overcome.  
Especially, if you’re searching in previous posts you see many problems arose with people using ext4. Especially those, who used 200TiB volumes, symbolic links and those who tried to download 16TiB segments from storj. You won’t find any of those people and nodes still alive… 😉

But really, why’re you asking if ext4 is one of the best recommended file systems. Opposed to for example fat, exfat or btrfs? See the many examples, if you were able to find the search function. Such like [1](https://forum.storj.io/t/btrfs-vs-ext4-vs-zfs-filesystem-for-storj/18069/15) and [2](https://forum.storj.io/t/what-is-the-best-file-system-for-storj-node/2388).

---

<div class="post-metadata">

**Author:** ![Toyoo](https://storj.bcdn.literatehosting.com/user_avatar/forum.storj.io/toyoo/32/1126_2.png) [@Toyoo](https://forum.storj.io/u/Toyoo)\
**Post date:** [June 12, 2023, 10:57pm UTC](https://forum.storj.io/t/ext4-optimization-recap/22883/17 "2023-06-12T22:57:32Z")

</div>

😁 ᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟᅟ
