Benchmark different ZFS pools

Node Operators
,
41

The way I understand it, @s-t-o-r-j-user stated a hypothesis, not a fact. Note how they say:

cant say much about its influence on storagenode operations

I assume at this point you believe everyone who states a hypothesis also has to test it?

42

That is where we differ. I think it is bad form to just bombard others with papers or a 1h Youtube link. Don’t get me wrong, I very much like sources, but I my opinion that does not prevent you from writing your own thoughts in one or two sentences.

Not gonna do that. You don’t have a saying in that.

Yeah, at first. Then I said “My guess none (influence)” to which he replayed with “you are probably very wrong”

43

I made a statement regarding my hypothesis after conducting preliminary tests with RDP protocols, which are a mix of TCP, UDP, and QUIC respectively, along with several SSH streams. A quick look at my dashboard potentially implied that everything was becoming much cozier again. :- ) EDIT: Of course also storj load was present on the machines along a few other things.

44

I consider you a friend of mine so I had to think for a moment, and after this moment, I hope you do not mind me asking, what’s the problem @Toyoo?

45

Just that I’m rather allergic to confident technological statements that are not supported by measurements. It’s really easy enough to add a “My guess is…” or “I believe…” to a statement to make it clear it’s just an opinion, and not absolute truth.

1 Like
46

That’s one way of doing it, out of the many correct ones. And if I am recalling this whole conversation correctly, one should not have any reservations to me in this regard. I am not very enthusiastic to provide completely unnecessary explanations. Anyway, all this conversation unfortunately really got off-topic.

47

If not @Toyoo then maybe @BrightSilence could save the situation. All in all we are all neighbors.

48

Technically we’re competitors trying to get as much Storj data as possible :stuck_out_tongue:

49

TBH, I am mostly for some IT fun so kindly please do speak for yourself. :- )

50

Back to topic, here is how I plan on moving forward.

To speed up the transfer, I will stop the node. That should not get me disqualified.

I have a group of 4 and two drives that are identical. I will use single drives for pools.

pool1 just a plain disc, ARC only. This is where I will copy the data to originally with rsync.
pool2 has special vdev metadata.
pool3 has reboot persistent L2ARC.
pool4 is ext4. Not really a pool :slight_smile:

All ZFS are with record Size 1MiB, Sync disabled, Atime off, lz4.
ext4 defaults and atime off? Any suggestions?
Database is always on the node SSD.

Steps:

  • copy data to pool1 with rsync
  • reboot
  • run test twice on pool1.
  • copy data to pool2
  • reboot
  • run test on pool2
  • destroy pool2 to get the SSD back an create pool3 with said SSD.
  • make L2ARC reboot persistant
  • copy data from pool2 to pool3
  • copy data from pool3 to pool4 to warm up L2ARC
  • reboot
  • run test on pool3 twice.
  • run test on pool4
51

if this is a VDEV, it’s a pool, but stacked: ext4 above ZFS :man_facepalming:

52

None of this is a vdev!

These will all be datasets. I don’t beliefe vdev to be useful for any kind of data expect VMs.
The fixed volblocksize has to many downsides. No, these are all datasets. Except “pool4” that is just a ext4 drive.

1 Like
53
  • copy data to pool2 with replication
  • reboot
  • run test twice on pool2.
  • copy data to pool1
  • reboot
  • run test on pool1
  • destroy pool2 to get the SSD back an create pool3 with said SSD.
  • make L2ARC reboot persistant
  • copy data from pool1 to pool3
  • copy data from pool3 to pool4 to warm up L2ARC
  • reboot
  • run test on pool3 twice.
  • run test on pool4
54

Current status:

  • copy data to pool2 with replication

Done! Also was a nice refresher on how TrueNAS can create keypairs in the webGUI but because you have to store the keys in the home directory of the user, this is pretty pointless and you are better of by doing it in the shell to begin with, which seems like a strange way to me but ok fine :laughing:

  • reboot
  • run test twice on pool2.

Skipped these two steps to do this first

  • copy data to pool1

That way I hope to have fewer differences between the two datasets.

Ingress is stopped by limiting the 5TB node to 1TB, but maybe there are some deletions that are will screw the results for pool3 and 4 a little bit. This ain’t a scientific paper :wink:

55

Vanilla. :- )
[20 …]

56

?

Anyway, first run went through.

So for pool2 with special vdev, first run with cold ARC took from 07:28:49 to 08:56:00.
Or in other words 1h and 27min for 5,5TB.

Second run is currently running. Look like after 1min, read speed falls from 50MB/s back to 20MB/s. 20MB/s is exactly the speed of the first run. So I expect that to be ARC.

Why is there no disk busy anymore in TrueNAS Scale? Really liked that in TrueNAS Core.

57

I dont know, I joined this topic mostly because of interest in Solaris ZFS; besides tbh, its hard to track the methodology and your developments; whats the summary if I may ask?

58

I’d like to add that I just simply cant wait for the results. :- )

59

Still don’t understand what Vanilla means in this context.
But to answer your question, it is still a work in progress.

Second run just finished in 1h 26min. So warming up ARC did not really help any further.

Next up is pool1 without special vdev.

60

Still don’t understand what Vanilla means in this context.

In the context of networking or in the context of storage or both?

Or maybe there is any other context you have on your mind?