I am building a new server right now and wanted to test Truenas Scale with it. It’s my first time working with ZFS and I was researching and reading a lot about it (also in this forum), but don’t have any experience so far.
Anyways, I wanted to share some tests that I did that might help others in future.
When I installed Truenas Scale I found the easiest method to run multiple storagenodes would be to create a debian VM and pass a zvol of my storage pool into it because in Truenas Scale directly I am not able to run docker commands and I didn’t want to create all my containers via the Kubernetes-GUI that Truenas Scale provides.
I found many discussions in this forum about the best record size of ZFS datasets for a storagenode, but not so much about the best block size when using zvol’s. So I just created one zvol for each block size possible, passed them to my Debian VM and formatted the virtual disks as EXT4 using this config (mke2fs -t ext4 -m 0 -i 65536 -I 128 -J size=128 -O sparse_super2) and copied a small node from my Unraid-Server over to each of the virtual disks:
root@debian:/mnt# find . -maxdepth 1 -type d -exec sh -c 'echo -n "{}: "; find "{}" -type f | wc -l' \;
.: 1237938
./storj_8: 206323
./storj_16: 206323
./storj_32: 206323
./storj_4: 206323
./storj_64: 206323
./storj_128: 206323
root@debian:/mnt# du -sh *
389G storj_128
389G storj_16
389G storj_32
389G storj_4
389G storj_64
389G storj_8
root@debian:/mnt# df -h
Filesystem Size Used Avail Use% Mounted on
...
/dev/vdb 25T 389G 25T 2% /mnt/storj_16
/dev/vdc 4.0T 389G 3.7T 10% /mnt/storj_4
/dev/vdd 8.0T 389G 7.7T 5% /mnt/storj_8
/dev/vde 20T 389G 20T 2% /mnt/storj_32
/dev/vdf 6.4T 389G 6.1T 6% /mnt/storj_64
/dev/vdg 13T 389G 13T 3% /mnt/storj_128
So the node has 206323 files and is 389GB big, so the average node file has 1,886mb. Debian shows that all of the volumes are equal. I then checked the Truenas Dashboard to see how efficiently each block size can store my node’s data (the storj zvol has block size 16KiB):
So as expected, there is a huge difference in terms of storage efficiency with the different block sizes.
Questions:
- I think I will go with a sector size of 64K as a trade-off between storage efficiency and performance/overhead. Are you aware of any disadvantages?
- If you are also using Truenas Scale and don’t want to access your pool via a network protocol (SMB, NFS, iSCSI,…), how are you running your node then? Also via passing zvol’s into a VM? Or directly on the host somehow?
- I am currently testing on a 4x18TB disk RAIDZ2 pool. Once all the remaining drives will arrive, I will create a bigger pool. Will that have any impact on the efficiency with using different block sizes?