ZFS performance, ARC vs L2ARC vs special vdev

[IsThisOn]

@arrogantrabbit

That… probably invalidates the whole test, as NFS introduces fixed latency and SCALE had (still has?) some known performance regressions from Core.

First point, these performance regressions are mostly gone according to LawrenceSystems.
Second point: This is exactly why I think this test gives some insight.

I am a big believer in “only use unused resources”.

I personally would not use TrueNAS Scale with STORJ on top of it for multiple reasons.
For one, I want STORJ to run on a different VLAN, ratelimiter, with the DB on a SSDs which I don’t have on bigtank machines, the list goes on.

So for me personally, running some STORJ Docker that access a dataset via NFS on a datatank is more realistic than TrueNAS SCALE with STORJ.

And even if NFS did add some latency, the difference between the results is still massive.

@arrogantrabbit

L2ARC fill rate is limited to a few megabytes per second. In 85 minutes it would at most cache under 100Gb of data.

Yes.
It did even less, 15GB “only”.
Dataset was set to metadata only.

@arrogantrabbit

This is highly suspect.

Why?

@arrogantrabbit

How did you test this? Did you add special device, configure small file threshold, then zfs send | zfs receive, and then run the test?

That is exactly how I did it

@arrogantrabbit

It can, and it is, but not due to data compression in the conventional sense, but due to partial sector space saving.

But why? I would get it if there were some RAIDZ stuff involved, but this is a single drive.
There is no pool geometry or padding or anything like that.
I highly doubt that without LZO, the dataset would be 25TB. Because that should be the case right, if the 5TB has a compression ratio of 5x?

@arrogantrabbit

See, 15GB is nothing. It’s either due to all your workload bypass cache, and/or limit on cache fill rate.

I am not worried about the 15GB on the SSD
I am worried about how much ARC it uses.
Some people used to recommend setting L2ARC not higher than 8x the amount of RAM because of this.

@arrogantrabbit

The big difference is that former accelerates second and subsequent accesses, and only if first one was not sequential (i.e. pre-warming the cache with file walker usecase is destroyed) while special device accelerates all accesses, including first, and sequential.

Not sure if I would agree 100% here.
I would say it this way, both methods can save all metadata.
And that is a benefit for a HDD.

@arrogantrabbit

Both of these are accomplished with special device.

True, but also for L2ARC.
Again, I would not bet on L2ARC being reasonably scalable for STORJ, but it did also perfectly fine.

There seems to be a small misunderstanding on your part:
L2ARC does not sped up the process, because it cached STORJ blobs.
It cached all metadata.

My theory is, that because L2ARC is basically just evicted ARC, the L2ARC drive was filled up during the replication task. Metadata has either evicted ARC from pool1 or evicted ARC from the writes to pool2. That is why the first run did not offer worse results than the second run. L2ARC was already hot.