What is the best file system for storj node?

[Ruskiem]

Coudnt find, so what is the best file system for storj node? especialy large discs 6-10TB?
NTFS? Ext2-4? exFAT? FAT32 ;D ?
And what Cluster Size? Under Windows 10

best Regards

[Alexey]

The best filesystem for the OS is native.
Since you are on Windows the answer is obvious - NTFS with default parameters.

[SlavikCA]

I’m running StorJ in Docker on my Synology DS412+, which has BTRFS.
Running for last 3 months.
No issues.

[pwnorbpwnd]

I am also running BTRFS, no problems or errors related to my array

[Alexey]

Please, don’t use the BTRFS on regular systems except Synology. They fixed most of bugs and it’s working fine on Synology.
Unfortunately they didn’t share the code with a community. The mainstream of BTRFS code is still not ready for production.

How many failed audits are acceptable?

What is your device? If that is not Synology, then please, change the filesystem ASAP There is currently (2019-07-07, linux ≤ 5.1.16) a bug that causes a two-disk raid1 profile to forever become read-only the second time it is mounted in a degraded state—for example due to a missing/broken/SATA link reset disk (unix.stackexchange.com, How to replace a disk drive that is physically no more there?). It is probable that this issue is mitigated by using three-disk raid1 profile filesystem rather …

[donald.m.motsinger]

I have to disagree on that blanket statement with you. You base it on the fact that I reported failed audits and after I mentioned that I use BTRFS you quickly blamed BTRFS for that. But the truth is, that there was a bug in Storj where already deleted pieces got audited and therefore failed. Failed Audits (v0.21.1)

The bug seems to be fixed now and I don’t have failed audits anymore since then.

BTRFS on a single disk is perfectly fine. The BTRFS status page only considers RAID5 unstable and the issue you quoted affects only a 2 drive RAID1 when 1 drive fails and the user mounted it a second time in degraded state. I have a 3 drive RAID1 as my media storage and after 1 drive failed and multiple degraded mounts I was able to replace the disk without problems.

[Alexey]

This is just a link with explanation why I do not recommend to use BTRFS on regular systems

[oty]

What about ZFS? I would like find the best file system for handle IOPS. I am hosting 5 nodes on proxmox server. IO wait to hight with ext4.

[Alexey]

Hello @oty,
Welcome to the forum!

See: Topics tagged zfs

@SGC

running a checksum filesystem isn’t for the speed its for the data integrity, zfs speeds up some tasks but raw writes and suchs will always be slower due to more iops being written to the drives… also there is a 3 or 4x minimum io amplification if xattr and atime is enabled, which they are by default.

and

Best filesystem for storj

Hello guys! I decided to conduct some research to choose the optimal file system in terms of the performance/cost ratio per 1 TB. The series of articles is a summary of my personal experience operating storj nodes and does not claim to be the ultimate truth, but rather represents my personal opinion. The storj node itself is a lightweight application and under normal/typical circumstances, it does not consume a large amount of resources in terms of CPU/RAM. However, even in normal situations, …

@oty

IO wait to hight with ext4.

you should run one node per disk. If you use RAID with parity, the whole pool will work as a slowest disk in the pool. This is true for any implementation. The speed will be LVM ext4, btrfs, zfs. For the single disk configuration it will be the same. However, in case of zfs you may improve with throwing resources to it (SSD special device, more RAM, more tuning).
See why (it’s about BTRFS, but it’s a FS with a greater Metadata too, which is applied to zfs as well):

Need some tricks survive with BTRFS

Some short notes from the top of my head, as I don’t have time to elaborate. The -t 300 thread was about a default non-Synology setup on a single HDD. As you cheerfully agreed, it’s not optimized for speed, having duplicated metadata and such. To support all these fancy new features, btrfs metadata is, by necessity, larger. This results in bigger RAM usage for caching metadata, and as such, when you reach the threshold where you can no longer fit metadata in RAM cache, you fall off a performan…

See also

Notes on storage node performance optimization on ZFS

My notes from performance optimization efforts on ZFS. Hardware setup for reference: Node holds around 9TB of data today, running on an old 2012-ish era freebsd server; there is a single zfs pool consisting of two 4-disk RAIDZ1 VDEV’s. Out of 32GB of ram, 9 are used by services, and the rest available for ARC. There is 2TB SSD used as L2ARC, but it is not really being utilized, I just had it, it is not necessary in any way. To speed up synchronous writes I have a cheap 16GB Optane device mounte…

[oty]

Hmm. Interesting. Will try move one node to ZFS single disk + special device. I have planty of free RAM and good enough CPU.

[oty]

And yes. Every node located on separated HDD. Maybe I should buy SCSI HBA controller.

[Alexey]

You may compare only on a full node usually. The small node will work faster at the beginning on BTRFS, but after some time it will become very slow. zfs doesn’t have this “feature” though - it works steady independently on size like ext4, but usually in 2-3 times slower without a tuning than ext4.

[oty]

Hmm. Very interesting.

[s-t-o-r-j-user]

@Alexey

You may compare only on a full node usually. The small node will work faster at the beginning on BTRFS, but after some time it will become very slow. zfs doesn’t have this “feature” though - it works steady independently on size like ext4, but usually in 2-3 times slower without a tuning than ext4.

With full respect, isn’t this part of you advice somehow slowly becoming slightly misleading? It looks like despite providing links to various posts on this forum, now you are providing info only about vanilla configs for btrfs and zfs. How is the tuning changing the perspective … and … how about XFS with metadata separated to another location?

[Alexey]

Please post your link to this comparison - it will give more opinions to the author.
BTRFS and zfs here were mentioned because the question was about zfs and high iowait for ext4

@oty

What about ZFS? I would like find the best file system for handle IOPS. I am hosting 5 nodes on proxmox server. IO wait to hight with ext4.

so, I do not know how to offer a fair comparison to answer this question, you may try better
my opinion is based on mine tests:

Doh.. Looks like hardware failure coming up!

I know all of that. As an argument I must say the btrfs has the same, but it’s at least in 2-3 times faster than zfs in the same level of RAID. I also tested the RAID10 in all three systems, as it’s the only viable RAID for production databases. The results have not changed: LVM raid, btrfs, zfs BUT. Since LVM raid have no checksum checks, and btrfs is unstable, there is no other choice. Maybe only Ceph. the same for LVM snapshots (wi…

[s-t-o-r-j-user]

I hear you @Alexey! However, this is the second thread on this topic recently and it looks like you are again asking me to do some testing on your behalf. Please kindly be informed that I am not planning to do any comparisons in the foreseeable future on this topic. Such tests would require significant commitment of time and in general I think that such a task lays in the area of Storj Inc. competencies. So what is the best filesystem to run a node, if you are a beginner, an intermediate or an advanced user, is it ext4 in all of those cases?

[Alexey]

I do not want to test anything, because my nodes are far away from the proposed configurations, they are strictly as described: no RAID, no weird FS, 1 Windows, 2 docker nodes. Just to make sure that our default recommendations are still works.

You always suggest to do tests on some weird FS or configurations, so I always ask you to do it yourself, please forgive me for that.

[s-t-o-r-j-user]

I hear you again, so if you do not want to do any testing please do not ask anybody to do any testing on behalf of Storj Inc., particularly, I hope that you are not going to ask me again to do any testing on behalf of your employer.

The case is about your advice on questions about performance of different filesystems. As I wrote two posts above, it seems that your advice on this topic sometimes and somehow slowly becoming slightly misleading, or to use other words, maybe your advice is not quite precise enough, not sure if you noticed.

So if you do not have any precise test results that stand behind your advice on the topic of filesystem performance and you are using some other people words, please do stay calm and let the topic flow.

BTW, those are not “some weird FS or configurations” just the situation is starting to look like you guys are beating the drives again and again without any reasons.

[JWvdV]

@Alexey

I do not want to test anything, because my nodes are far away from the proposed configurations, they are strictly as described: no RAID, no weird FS, 1 Windows, 2 docker nodes. Just to make sure that our default recommendations are still works.

I can only emphasize the convenience of these kind of setups. I still regret my first node I set up, using mergerfs and some further nodes choking on BTRFS.

Moreover, additional layers like md-/zfs-/btrfs-raid and such, make it more complex and are actually doing additional the same thing the network as a whole is doing already: adding redundancy in order to be resilient to some data losses.

[s-t-o-r-j-user]

Yeah, mergerfs might be too heavy and too fragile at the same time for a task that it being discussed here, however, to challenge @arrogantrabbit ZFS referential config, I was wondering about XFS in this thread and about Lustre in the other. What do you think, @arrogantrabbit?