Is it possible to make read only cash for node, to serv hot files?

The idea is make possible to nodes to cash some frequently reeded files of node on fast drives possible.
This will make hot files possible to read very fast from nodes.
Also node will make less seek operations and make faster response on writes.
Like every HDD have cash for better performance.
It shold be possible to give directory and size on it. For example My servers have NVME that are not used and only 200-500GB it not logical to use as node, but as buffer will be perfect.

may be it is possible somehow to make it by Hardware ?

This is supported with LVM in linux, ie if you have an LVM layer with your FS on it.

See lvmcache(7) - Linux manual page and similar pages.

I haven’t tested this at all myself so I don’t know if this yields any actual performance gains for this use case.

My HDDs read at 200MB/s and my gigabit connection is 119MB/s. You have some fast internet if thats not the bottleneck.

This is only meaningful for a sequential read of a very large files located on the outer tracks of your drive. When random access is involved (e.g. small files, especially scattered across the disk) it is not unusual to see as low as 0.01MBps top throughput.

The reason is — seek time is not zero.

Hence, caching is employed to offload random IO from disks, to minimize time disks move heads and wait for the sector to fly by and maximize time disks read and write data.

Since you uses Windows, you can try to use Windows Storage Spaces and their cache capability:

See also

Its called ZFS. Basically every storagenode can run it. No special hardware needed.

I found software that makes this cash in RAM or SSD. I tried for now make it on NVME.

OK now after some day I have some statistiks. About use of cache.
I made on 8x 4TB nodes cach NVME 1 TB shared for read/write.
this is statistics about in weekend started on friday, first 24h filewalker made hdds working 100%

Volume Node4 (J:)
2022-09-05 12:55:03

Total Read 228.38GB
Cached Read 54.77GB (24.0%)
L2Storage Read 54.77GB (24.0%)
L2Storage Write 34.68GB
Total Write (Req) 215.97GB
Total Write (L1/L2) 0 / 215.97GB
Total Write (Disk) 212.03GB (98.2%)
Urgent/Normal 0 / 212.03GB
Deferred Blocks 367 (0.0%)
Trimmed Blocks 689
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 23.98%

Volume Node5 (K:)
2022-09-05 12:55:25

Total Read 74.88GB
Cached Read 48.67GB (65.0%)
L2Storage Read 48.67GB (65.0%)
L2Storage Write 34.85GB
Total Write (Req) 3.21GB
Total Write (L1/L2) 0 / 3.21GB
Total Write (Disk) 2.85GB (88.8%)
Urgent/Normal 0 / 2.85GB
Deferred Blocks 85 (0.0%)
Trimmed Blocks 1
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 65.00%

Volume Node2 (F:)
2022-09-05 12:55:38

Total Read 69.52GB
Cached Read 35.95GB (51.7%)
L2Storage Read 35.95GB (51.7%)
L2Storage Write 34.05GB
Total Write (Req) 3.53GB
Total Write (L1/L2) 0 / 3.53GB
Total Write (Disk) 3.16GB (89.5%)
Urgent/Normal 0 / 3.16GB
Deferred Blocks 89 (0.0%)
Trimmed Blocks 3
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 51.71%

Volume Node7 (I:)
2022-09-05 12:55:50

Total Read 92.75GB
Cached Read 45.28GB (48.8%)
L2Storage Read 45.28GB (48.8%)
L2Storage Write 44.89GB
Total Write (Req) 9.51GB
Total Write (L1/L2) 0 / 9.51GB
Total Write (Disk) 8.93GB (93.8%)
Urgent/Normal 0 / 8.93GB
Deferred Blocks 196 (0.0%)
Trimmed Blocks 4
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 48.82%

Volume Node6 (G:)
2022-09-05 12:56:06

Total Read 456.91GB
Cached Read 124.45GB (27.2%)
L2Storage Read 124.45GB (27.2%)
L2Storage Write 55.92GB
Total Write (Req) 461.17GB
Total Write (L1/L2) 0 / 461.17GB
Total Write (Disk) 452.26GB (98.1%)
Urgent/Normal 0 / 452.26GB
Deferred Blocks 0 (0.0%)
Trimmed Blocks 670
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 27.24%

Volume Node3 (L:)
2022-09-05 12:56:20

Total Read 103.52GB
Cached Read 34.46GB (33.3%)
L2Storage Read 34.46GB (33.3%)
L2Storage Write 42.31GB
Total Write (Req) 34.41GB
Total Write (L1/L2) 0 / 34.41GB
Total Write (Disk) 33.88GB (98.5%)
Urgent/Normal 0 / 33.88GB
Deferred Blocks 5 (0.0%)
Trimmed Blocks 1
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 33.29%

Volume Node (D:)
2022-09-05 12:56:35

Total Read 188.04GB
Cached Read 40.78GB (21.7%)
L2Storage Read 40.78GB (21.7%)
L2Storage Write 36.48GB
Total Write (Req) 127.91GB
Total Write (L1/L2) 0 / 127.91GB
Total Write (Disk) 126.94GB (99.2%)
Urgent/Normal 0 / 126.94GB
Deferred Blocks 22 (0.0%)
Trimmed Blocks 5
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 21.69%

Volume Node1 (E:)
2022-09-05 12:56:48

Total Read 99.08GB
Cached Read 15.79GB (15.9%)
L2Storage Read 15.79GB (15.9%)
L2Storage Write 34.77GB
Total Write (Req) 51.33GB
Total Write (L1/L2) 0 / 51.33GB
Total Write (Disk) 50.67GB (98.7%)
Urgent/Normal 0 / 50.67GB
Deferred Blocks 12 (0.0%)
Trimmed Blocks 4
Prefetch Inactive
Unused Cache (L1) 0
Unused Cache (L2) 32.00MB
Cache Hit Rate 15.94%

As we see it is very useful, on different nodes 15-65% reeded from cache.
so i see that cache is very useful thing.

But does it give you a higher success rate compared to an uncached node?

1 Like

Primocache with huge load corrupt my files and i lost some nodes. Be careful with this soft

did you used RAM cache or NVME?

I use it differently. Ram->ssd->hdd write. ssd->hdd write. Read speed is not problem becouse it not high in storj. Next step, you want speed up write with this soft, and corrupt or lost some files and disqual.
P.S. nvme for cache is very expensive. This is read data, ssd from aliexpress is best :slight_smile:

I dont use RAM, so i disabled L1 cache at all. Because possible power cut will kill All data in L1.(mai be this hapened with your node?)
My main purpose is speed up reeds, from cache, by this way, I also load down HDD for writes.
As you see on my post, 15%-65% reeds made from cache. this makes more time to write normally.

After your post, i made cache read only, this will protects from writing problem, speed up reading.
and make more time for faster writing, because it not read same files several time from HDD.

You can use ram for read caching too as it will always have the original on HDD.

RAM is too small to make it effective, i used 1 TB NVME.
Interesting, to change config yesterday i made restart, file runner on all HDD was completed in 6h wile i was sleeping, last restart took 24h to complete. After restart cache ssd was reading 120 MByte/S and other HDD was on 4-8% of load. So looks like cache also holding all file structure and it helps make all faster.

If you need to do that a “real” fast way you need to get out metadata of files on nvme or ssd array. It can be done with zfs special device.
Or you can simple add

storage2.piece-scan-on-startup: false

into node’s config file

8 posts were split to a new topic: Storage2.piece-scan-on-startup