You should be able to test this empirically.
Create some sparse files, then set up zfs on top of them, mount it, and create as many empty files as you desire. Then look at statistics, or just allocated size of the sparse files.
Can’t speak about details, I don’t know zfs—but this is what I’d do with any file system.
Yes.
But no.
Your thinking would be correct, if you only wanted to know how much space meta data records take. In this case we want to test difference in total meta data size depending on record size, given a certain mixture of file sizes.
ZFS allocates one record for every file up to recordsize, but multiple records for any file sized above. So, with a recordsize of 1MiB all files up to 1MiB get 1 metadata record (and real size on disk size of ~2^int(log(size)/log(2)+1)). Meaning a file op 512k gets one meta record, but a file of 4.5MiB gets 5 records. Don’t know whether ZFS implements measures to reduce meta data size in case of adjacent record (feeling like extents many other filesystems).
@arrogantrabbit
Already any results back?
I rewrote the awk script once again:
for i in /storj/n*; do echo $'\n\n'$i; find $i -type f -print0 | xargs -0 -P 40 -n 100 stat -c "%s" | awk '{ n=log($1)/log(2); if (n<12) { n=12; } else if (n > int(n)) { n = int(n) + 1; } size[n]++ } END { t=0; t128=0; t512=0; for (i in size){ r128=size[i]*(i<18? 1 : 2^(i-17)); r512=size[i]*(i<20? 1 : 2^(i-19)); printf("%d: %d files; 128k-records: %d, 512k-records: %d\n", 2^i, size[i], r128, r512); t += size[i]; t128 += r128; t512 += r512; } printf("Total amount of files: %d, 128k-records: %d, 512k-records: %d\n", t, t128, t512); }' | sort -n; done
If I execute this, I’m starting to understand why meta data almost tripled with 128k records (unless ZFS implements extent-like-measures for adjacent records):
/storj/nd10
Total amount of files: 14224060, 128k-records: 34344923, 512k-records: 17615884
4096: 2772866 files; 128k-records: 2772866, 512k-records: 2772866
8192: 967477 files; 128k-records: 967477, 512k-records: 967477
16384: 2110422 files; 128k-records: 2110422, 512k-records: 2110422
32768: 1707719 files; 128k-records: 1707719, 512k-records: 1707719
65536: 1923367 files; 128k-records: 1923367, 512k-records: 1923367
131072: 559020 files; 128k-records: 559020, 512k-records: 559020
262144: 2998000 files; 128k-records: 5996000, 512k-records: 2998000
524288: 555531 files; 128k-records: 2222124, 512k-records: 555531
1048576: 104659 files; 128k-records: 837272, 512k-records: 209318
2097152: 97519 files; 128k-records: 1560304, 512k-records: 390076
4194304: 427435 files; 128k-records: 13677920, 512k-records: 3419480
8388608: 17 files; 128k-records: 1088, 512k-records: 272
16777216: 15 files; 128k-records: 1920, 512k-records: 480
33554432: 11 files; 128k-records: 2816, 512k-records: 704
67108864: 1 files; 128k-records: 512, 512k-records: 128
536870912: 1 files; 128k-records: 4096, 512k-records: 1024
/storj/nd11
Total amount of files: 2306458, 128k-records: 5844101, 512k-records: 2789888
4096: 526155 files; 128k-records: 526155, 512k-records: 526155
8192: 121403 files; 128k-records: 121403, 512k-records: 121403
16384: 169032 files; 128k-records: 169032, 512k-records: 169032
32768: 127691 files; 128k-records: 127691, 512k-records: 127691
65536: 136790 files; 128k-records: 136790, 512k-records: 136790
131072: 115350 files; 128k-records: 115350, 512k-records: 115350
262144: 863094 files; 128k-records: 1726188, 512k-records: 863094
524288: 149313 files; 128k-records: 597252, 512k-records: 149313
1048576: 19736 files; 128k-records: 157888, 512k-records: 39472
2097152: 20583 files; 128k-records: 329328, 512k-records: 82332
4194304: 57273 files; 128k-records: 1832736, 512k-records: 458184
8388608: 23 files; 128k-records: 1472, 512k-records: 368
16777216: 14 files; 128k-records: 1792, 512k-records: 448
134217728: 1 files; 128k-records: 1024, 512k-records: 256
/storj/nd16
Total amount of files: 2682706, 128k-records: 6484996, 512k-records: 3231820
4096: 457079 files; 128k-records: 457079, 512k-records: 457079
8192: 159825 files; 128k-records: 159825, 512k-records: 159825
16384: 369413 files; 128k-records: 369413, 512k-records: 369413
32768: 285822 files; 128k-records: 285822, 512k-records: 285822
65536: 316844 files; 128k-records: 316844, 512k-records: 316844
131072: 148113 files; 128k-records: 148113, 512k-records: 148113
262144: 615498 files; 128k-records: 1230996, 512k-records: 615498
524288: 223580 files; 128k-records: 894320, 512k-records: 223580
1048576: 19543 files; 128k-records: 156344, 512k-records: 39086
2097152: 19906 files; 128k-records: 318496, 512k-records: 79624
4194304: 67073 files; 128k-records: 2146336, 512k-records: 536584
8388608: 4 files; 128k-records: 256, 512k-records: 64
16777216: 3 files; 128k-records: 384, 512k-records: 96
33554432: 3 files; 128k-records: 768, 512k-records: 192
/storj/nd18
Total amount of files: 13618008, 128k-records: 32090169, 512k-records: 15690440
4096: 2994556 files; 128k-records: 2994556, 512k-records: 2994556
8192: 474146 files; 128k-records: 474146, 512k-records: 474146
16384: 586171 files; 128k-records: 586171, 512k-records: 586171
32768: 468725 files; 128k-records: 468725, 512k-records: 468725
65536: 457034 files; 128k-records: 457034, 512k-records: 457034
131072: 342837 files; 128k-records: 342837, 512k-records: 342837
262144: 7350592 files; 128k-records: 14701184, 512k-records: 7350592
524288: 553077 files; 128k-records: 2212308, 512k-records: 553077
1048576: 64917 files; 128k-records: 519336, 512k-records: 129834
2097152: 71583 files; 128k-records: 1145328, 512k-records: 286332
4194304: 254166 files; 128k-records: 8133312, 512k-records: 2033328
8388608: 49 files; 128k-records: 3136, 512k-records: 784
16777216: 75 files; 128k-records: 9600, 512k-records: 2400
33554432: 56 files; 128k-records: 14336, 512k-records: 3584
67108864: 23 files; 128k-records: 11776, 512k-records: 2944
2147483648: 1 files; 128k-records: 16384, 512k-records: 4096
/storj/nd22
Total amount of files: 4455721, 128k-records: 10018700, 512k-records: 4952723
4096: 744847 files; 128k-records: 744847, 512k-records: 744847
8192: 110688 files; 128k-records: 110688, 512k-records: 110688
16384: 157000 files; 128k-records: 157000, 512k-records: 157000
32768: 128383 files; 128k-records: 128383, 512k-records: 128383
65536: 139130 files; 128k-records: 139130, 512k-records: 139130
131072: 95318 files; 128k-records: 95318, 512k-records: 95318
262144: 2833047 files; 128k-records: 5666094, 512k-records: 2833047
524288: 147716 files; 128k-records: 590864, 512k-records: 147716
1048576: 18099 files; 128k-records: 144792, 512k-records: 36198
2097152: 24121 files; 128k-records: 385936, 512k-records: 96484
4194304: 57317 files; 128k-records: 1834144, 512k-records: 458536
8388608: 22 files; 128k-records: 1408, 512k-records: 352
16777216: 15 files; 128k-records: 1920, 512k-records: 480
33554432: 7 files; 128k-records: 1792, 512k-records: 448
67108864: 4 files; 128k-records: 2048, 512k-records: 512
134217728: 6 files; 128k-records: 6144, 512k-records: 1536
1073741824: 1 files; 128k-records: 8192, 512k-records: 2048
/storj/nd4
Total amount of files: 13424917, 128k-records: 34646551, 512k-records: 16610670
4096: 2346371 files; 128k-records: 2346371, 512k-records: 2346371
8192: 556238 files; 128k-records: 556238, 512k-records: 556238
16384: 1157066 files; 128k-records: 1157066, 512k-records: 1157066
32768: 1042392 files; 128k-records: 1042392, 512k-records: 1042392
65536: 1419211 files; 128k-records: 1419211, 512k-records: 1419211
131072: 302835 files; 128k-records: 302835, 512k-records: 302835
262144: 5661895 files; 128k-records: 11323790, 512k-records: 5661895
524288: 377122 files; 128k-records: 1508488, 512k-records: 377122
1048576: 72722 files; 128k-records: 581776, 512k-records: 145444
2097152: 79326 files; 128k-records: 1269216, 512k-records: 317304
4194304: 409553 files; 128k-records: 13105696, 512k-records: 3276424
8388608: 105 files; 128k-records: 6720, 512k-records: 1680
16777216: 39 files; 128k-records: 4992, 512k-records: 1248
33554432: 15 files; 128k-records: 3840, 512k-records: 960
67108864: 25 files; 128k-records: 12800, 512k-records: 3200
134217728: 1 files; 128k-records: 1024, 512k-records: 256
536870912: 1 files; 128k-records: 4096, 512k-records: 1024
/storj/nd6
Total amount of files: 6403557, 128k-records: 13085941, 512k-records: 6991228
4096: 1936071 files; 128k-records: 1936071, 512k-records: 1936071
8192: 130343 files; 128k-records: 130343, 512k-records: 130343
16384: 175922 files; 128k-records: 175922, 512k-records: 175922
32768: 141963 files; 128k-records: 141963, 512k-records: 141963
65536: 159290 files; 128k-records: 159290, 512k-records: 159290
131072: 122704 files; 128k-records: 122704, 512k-records: 122704
262144: 3440046 files; 128k-records: 6880092, 512k-records: 3440046
524288: 181489 files; 128k-records: 725956, 512k-records: 181489
1048576: 20094 files; 128k-records: 160752, 512k-records: 40188
2097152: 26515 files; 128k-records: 424240, 512k-records: 106060
4194304: 69040 files; 128k-records: 2209280, 512k-records: 552320
8388608: 40 files; 128k-records: 2560, 512k-records: 640
16777216: 17 files; 128k-records: 2176, 512k-records: 544
33554432: 19 files; 128k-records: 4864, 512k-records: 1216
67108864: 3 files; 128k-records: 1536, 512k-records: 384
1073741824: 1 files; 128k-records: 8192, 512k-records: 2048
/storj/nd9
Total amount of files: 6635767, 128k-records: 16957408, 512k-records: 8170514
4096: 2205165 files; 128k-records: 2205165, 512k-records: 2205165
8192: 195258 files; 128k-records: 195258, 512k-records: 195258
16384: 342006 files; 128k-records: 342006, 512k-records: 342006
32768: 311580 files; 128k-records: 311580, 512k-records: 311580
65536: 324749 files; 128k-records: 324749, 512k-records: 324749
131072: 201164 files; 128k-records: 201164, 512k-records: 201164
262144: 2492441 files; 128k-records: 4984882, 512k-records: 2492441
524288: 292879 files; 128k-records: 1171516, 512k-records: 292879
1048576: 37390 files; 128k-records: 299120, 512k-records: 74780
2097152: 34199 files; 128k-records: 547184, 512k-records: 136796
4194304: 198742 files; 128k-records: 6359744, 512k-records: 1589936
8388608: 185 files; 128k-records: 11840, 512k-records: 2960
16777216: 5 files; 128k-records: 640, 512k-records: 160
67108864: 3 files; 128k-records: 1536, 512k-records: 384
134217728: 1 files; 128k-records: 1024, 512k-records: 256
So, a recordsize of 512k quite consistently halves the amount of metadata records (it’s probably not that bad, because I didn’t calculate on file level, because that would take ages).
Not so fast. My smallest node is 15TB and the initial send/receive is still in progress. It will take a while.
AWK is quicker than I thought, so:
for i in /storj/n*; do
echo $'\n\n'$i;
find $i -type f -print0 | xargs -0 -P 40 -n 100 stat -c "%s" | awk '{
n=log($1)/log(2);
if (n<12) { n=12; } else if (n > int(n)) { n = int(n) + 1; }
size[n]++;
r128 = int($1 / 131072) + (($1 % 131072) > 0);
r512 = int($1 / 524288) + (($1 % 524288) > 0);
records128[n] += r128;
records512[n] += r512;
loss128[n] += (n < 18? 2^n : r128 * 131072) - $1;
loss512[n] += (n < 20? 2^n : r512 * 524288) - $1;
totalsize += $1;
}
END {
t=0; t128=0; t512=0; l128=0; l512=0;
for (i in size){
printf("%d: %d files; 128k-records: %d (loss: %d), 512k-records: %d (loss: %d)\n", 2^i, size[i], records128[i], loss128[i], records512[i], loss512[i]);
t += size[i];
t128 += records128[i];
t512 += records512[i];
l128 += loss128[i];
l512 += loss512[i];
}
printf("Total amount of files: %d (totalsize %d), 128k-records: %d (loss: %d), 512k-records: %d (loss %d)\n", t, totalsize, t128, l128, t512, l512);
}' | sort -n;
done
Result:
/storj/nd10 Total amount of files: 14425945 (totalsize 2252536281582), 128k-records: 27616495 (loss: 357451568658), 512k-records: 16486900 (loss 592687535634)
4096: 2783759 files; 128k-records: 2783750 (loss: 3929095698), 512k-records: 2783750 (loss: 3929095698)
8192: 972035 files; 128k-records: 972035 (loss: 2072509015), 512k-records: 972035 (loss: 2072509015)
16384: 2114147 files; 128k-records: 2114147 (loss: 7625174970), 512k-records: 2114147 (loss: 7625174970)
32768: 1710650 files; 128k-records: 1710650 (loss: 20412948858), 512k-records: 1710650 (loss: 20412948858)
65536: 1925923 files; 128k-records: 1925923 (loss: 47420580630), 512k-records: 1925923 (loss: 47420580630)
131072: 560868 files; 128k-records: 560868 (loss: 23127283566), 512k-records: 560868 (loss: 23127283566)
262144: 3167139 files; 128k-records: 6334278 (loss: 179982391276), 512k-records: 3167139 (loss: 179982391276)
524288: 559257 files; 128k-records: 1752166 (loss: 44015135380), 512k-records: 559257 (loss: 107566967444)
1048576: 105338 files; 128k-records: 614007 (loss: 7292039212), 512k-records: 210676 (loss: 37267812396)
2097152: 97720 files; 128k-records: 1156117 (loss: 3826079040), 512k-records: 335095 (loss: 27977798976)
4194304: 429055 files; 128k-records: 7684684 (loss: 17744105993), 512k-records: 2145368 (loss: 135287902729)
8388608: 19 files; 128k-records: 719 (loss: 1750351), 512k-records: 192 (loss: 8172879)
16777216: 22 files; 128k-records: 1914 (loss: 1426085), 512k-records: 486 (loss: 5358245)
33554432: 11 files; 128k-records: 1847 (loss: 889608), 512k-records: 466 (loss: 3117832)
67108864: 1 files; 128k-records: 284 (loss: 118016), 512k-records: 71 (loss: 118016)
536870912: 1 files; 128k-records: 3106 (loss: 40960), 512k-records: 777 (loss: 303104)
/storj/nd11
Total amount of files: 2301357 (totalsize 423198041183), 128k-records: 4719088 (loss: 75008641953), 512k-records: 2581592 (loss 120390486945)
4096: 522815 files; 128k-records: 508937 (loss: 681816165), 512k-records: 508937 (loss: 681816165)
8192: 121198 files; 128k-records: 121198 (loss: 266938873), 512k-records: 121198 (loss: 266938873)
16384: 168942 files; 128k-records: 168942 (loss: 708726815), 512k-records: 168942 (loss: 708726815)
32768: 127609 files; 128k-records: 127609 (loss: 1323333488), 512k-records: 127609 (loss: 1323333488)
65536: 136639 files; 128k-records: 136639 (loss: 2631076813), 512k-records: 136639 (loss: 2631076813)
131072: 115151 files; 128k-records: 115151 (loss: 4945044794), 512k-records: 115151 (loss: 4945044794)
262144: 862808 files; 128k-records: 1725616 (loss: 47876053501), 512k-records: 862808 (loss: 47876053501)
524288: 148634 files; 128k-records: 461556 (loss: 12318094691), 512k-records: 148634 (loss: 29748049251)
1048576: 19696 files; 128k-records: 110366 (loss: 1391607981), 512k-records: 39392 (loss: 7578468525)
2097152: 20563 files; 128k-records: 213761 (loss: 485049953), 512k-records: 65168 (loss: 6633768545)
4194304: 57270 files; 128k-records: 1026507 (loss: 2378753074), 512k-records: 286398 (loss: 17987462194)
8388608: 17 files; 128k-records: 656 (loss: 1390647), 512k-records: 173 (loss: 6109239)
16777216: 14 files; 128k-records: 1170 (loss: 685526), 512k-records: 298 (loss: 3569110)
134217728: 1 files; 128k-records: 980 (loss: 69632), 512k-records: 245 (loss: 69632)
/storj/nd16
Total amount of files: 2682684 (totalsize 426638958102), 128k-records: 5173367 (loss: 82538615274), 512k-records: 3013476 (loss 139986817514)
4096: 457071 files; 128k-records: 457071 (loss: 701857559), 512k-records: 457071 (loss: 701857559)
8192: 159826 files; 128k-records: 159826 (loss: 344430429), 512k-records: 159826 (loss: 344430429)
16384: 369414 files; 128k-records: 369414 (loss: 1398463082), 512k-records: 369414 (loss: 1398463082)
32768: 285823 files; 128k-records: 285823 (loss: 2978327709), 512k-records: 285823 (loss: 2978327709)
65536: 316843 files; 128k-records: 316843 (loss: 6064901378), 512k-records: 316843 (loss: 6064901378)
131072: 148117 files; 128k-records: 148117 (loss: 6463225475), 512k-records: 148117 (loss: 6463225475)
262144: 615480 files; 128k-records: 1230960 (loss: 40556268085), 512k-records: 615480 (loss: 40556268085)
524288: 223578 files; 128k-records: 695439 (loss: 19384045069), 512k-records: 223578 (loss: 45450726925)
1048576: 19543 files; 128k-records: 107640 (loss: 1408859954), 512k-records: 39086 (loss: 7792590642)
2097152: 19906 files; 128k-records: 200461 (loss: 503447296), 512k-records: 62626 (loss: 7062683392)
4194304: 67073 files; 128k-records: 1200843 (loss: 2733856759), 512k-records: 335375 (loss: 21170051063)
8388608: 4 files; 128k-records: 176 (loss: 361343), 512k-records: 46 (loss: 1409919)
16777216: 3 files; 128k-records: 255 (loss: 337509), 512k-records: 65 (loss: 992869)
33554432: 3 files; 128k-records: 499 (loss: 233627), 512k-records: 126 (loss: 888987)
/storj/nd18
Total amount of files: 13603903 (totalsize 2516255704307), 128k-records: 27463891 (loss: 503069473549), 512k-records: 14852171 (loss 674692397837) 4096: 2985524 files; 128k-records: 2985083 (loss: 3169546327), 512k-records: 2985083 (loss: 3169546327)
8192: 473560 files; 128k-records: 473560 (loss: 1068433977), 512k-records: 473560 (loss: 1068433977)
16384: 586011 files; 128k-records: 586011 (loss: 2399812019), 512k-records: 586011 (loss: 2399812019)
32768: 468576 files; 128k-records: 468576 (loss: 4942098709), 512k-records: 468576 (loss: 4942098709)
65536: 456657 files; 128k-records: 456657 (loss: 8757065444), 512k-records: 456657 (loss: 8757065444)
131072: 342213 files; 128k-records: 342213 (loss: 13681375877), 512k-records: 342213 (loss: 13681375877)
262144: 7349557 files; 128k-records: 14699114 (loss: 407009279401), 512k-records: 7349557 (loss: 407009279401)
524288: 551086 files; 128k-records: 1747855 (loss: 44738768385), 512k-records: 551086 (loss: 104571694593)
1048576: 64797 files; 128k-records: 373154 (loss: 4568420348), 512k-records: 129594 (loss: 23602958332)
2097152: 71562 files; 128k-records: 763662 (loss: 1913893573), 512k-records: 229834 (loss: 22318396101)
4194304: 254164 files; 128k-records: 4532115 (loss: 10805865433), 512k-records: 1270941 (loss: 83111603161)
8388608: 47 files; 128k-records: 1880 (loss: 4556033), 512k-records: 496 (loss: 18187521)
16777216: 74 files; 128k-records: 6723 (loss: 5278351), 512k-records: 1710 (loss: 20613775)
33554432: 51 files; 128k-records: 9417 (loss: 3504737), 512k-records: 2374 (loss: 13859425)
67108864: 23 files; 128k-records: 7391 (loss: 1562647), 512k-records: 1859 (loss: 7460887)
2147483648: 1 files; 128k-records: 10480 (loss: 12288), 512k-records: 2620 (loss: 12288)
/storj/nd22
Total amount of files: 4449535 (totalsize 828766393703), 128k-records: 8884182 (loss: 188207860377), 512k-records: 4753191 (loss 234325674649)
4096: 741592 files; 128k-records: 741592 (loss: 705430855), 512k-records: 741592 (loss: 705430855)
8192: 110623 files; 128k-records: 110623 (loss: 248625448), 512k-records: 110623 (loss: 248625448)
16384: 156961 files; 128k-records: 156961 (loss: 638276221), 512k-records: 156961 (loss: 638276221)
32768: 128368 files; 128k-records: 128368 (loss: 1324308015), 512k-records: 128368 (loss: 1324308015)
65536: 139082 files; 128k-records: 139082 (loss: 2750804530), 512k-records: 139082 (loss: 2750804530)
131072: 95255 files; 128k-records: 95255 (loss: 3896618682), 512k-records: 95255 (loss: 3896618682)
262144: 2830544 files; 128k-records: 5661088 (loss: 162113744475), 512k-records: 2830544 (loss: 162113744475)
524288: 147524 files; 128k-records: 456895 (loss: 12283338960), 512k-records: 147524 (loss: 29742260432)
1048576: 18095 files; 128k-records: 102982 (loss: 1269801433), 512k-records: 36190 (loss: 6745727449)
2097152: 24119 files; 128k-records: 250547 (loss: 603198980), 512k-records: 76882 (loss: 8071812612)
4194304: 57317 files; 128k-records: 1026645 (loss: 2369783096), 512k-records: 286612 (loss: 18072601912)
8388608: 22 files; 128k-records: 875 (loss: 2089044), 512k-records: 229 (loss: 7462996)
16777216: 15 files; 128k-records: 1303 (loss: 789277), 512k-records: 331 (loss: 3541789)
33554432: 7 files; 128k-records: 1193 (loss: 402761), 512k-records: 301 (loss: 1844553)
67108864: 4 files; 128k-records: 1502 (loss: 202099), 512k-records: 377 (loss: 988531)
134217728: 6 files; 128k-records: 4296 (loss: 360485), 512k-records: 1076 (loss: 1409061)
1073741824: 1 files; 128k-records: 4975 (loss: 86016), 512k-records: 1244 (loss: 217088)
/storj/nd4
Total amount of files: 13423808 (totalsize 2551182964472), 128k-records: 28148855 (loss: 443987749128), 512k-records: 15355462 (loss 638052034824)
4096: 2345822 files; 128k-records: 2345529 (loss: 2667624342), 512k-records: 2345529 (loss: 2667624342)
8192: 556240 files; 128k-records: 556240 (loss: 1188658165), 512k-records: 556240 (loss: 1188658165)
16384: 1157067 files; 128k-records: 1157067 (loss: 4179085464), 512k-records: 1157067 (loss: 4179085464)
32768: 1042393 files; 128k-records: 1042393 (loss: 12278901008), 512k-records: 1042393 (loss: 12278901008)
65536: 1419181 files; 128k-records: 1419181 (loss: 36284051969), 512k-records: 1419181 (loss: 36284051969)
131072: 302774 files; 128k-records: 302774 (loss: 12155388074), 512k-records: 302774 (loss: 12155388074)
262144: 5661424 files; 128k-records: 11322848 (loss: 320374264550), 512k-records: 5661424 (loss: 320374264550)
524288: 377121 files; 128k-records: 1187233 (loss: 29481569148), 512k-records: 377121 (loss: 71588580220)
1048576: 72721 files; 128k-records: 420937 (loss: 5026784467), 512k-records: 145442 (loss: 26107225299)
2097152: 79326 files; 128k-records: 978349 (loss: 3340031454), 512k-records: 280462 (loss: 22148732382)
4194304: 409553 files; 128k-records: 7393390 (loss: 16994474754), 512k-records: 2062005 (loss: 129012538114)
8388608: 105 files; 128k-records: 3707 (loss: 11446139), 512k-records: 996 (loss: 47753083)
16777216: 39 files; 128k-records: 3529 (loss: 2656531), 512k-records: 895 (loss: 9341203)
33554432: 15 files; 128k-records: 2805 (loss: 651221), 512k-records: 708 (loss: 4190165)
67108864: 25 files; 128k-records: 9049 (loss: 1948426), 512k-records: 2268 (loss: 4963082)
134217728: 1 files; 128k-records: 531 (loss: 106920), 512k-records: 133 (loss: 237992)
536870912: 1 files; 128k-records: 3293 (loss: 106496), 512k-records: 824 (loss: 499712)
/storj/nd6
Total amount of files: 6397986 (totalsize 1055503909533), 128k-records: 11734840 (loss: 176588651875), 512k-records: 6755223 (loss 231772192099)
4096: 1933026 files; 128k-records: 1933023 (loss: 1177728601), 512k-records: 1933023 (loss: 1177728601)
8192: 130287 files; 128k-records: 130287 (loss: 290348591), 512k-records: 130287 (loss: 290348591)
16384: 175876 files; 128k-records: 175876 (loss: 716402612), 512k-records: 175876 (loss: 716402612)
32768: 141933 files; 128k-records: 141933 (loss: 1449199986), 512k-records: 141933 (loss: 1449199986)
65536: 159237 files; 128k-records: 159237 (loss: 3137508594), 512k-records: 159237 (loss: 3137508594)
131072: 122641 files; 128k-records: 122641 (loss: 5128370044), 512k-records: 122641 (loss: 5128370044)
262144: 3438022 files; 128k-records: 6876044 (loss: 144980958473), 512k-records: 3438022 (loss: 144980958473)
524288: 181269 files; 128k-records: 565256 (loss: 14671501556), 512k-records: 181269 (loss: 35619428596)
1048576: 20074 files; 128k-records: 114593 (loss: 1411245696), 512k-records: 40148 (loss: 7440426624)
2097152: 26510 files; 128k-records: 274461 (loss: 627981262), 512k-records: 83938 (loss: 8661515214)
4194304: 69037 files; 128k-records: 1227654 (loss: 2992004776), 512k-records: 345355 (loss: 23146421928)
8388608: 34 files; 128k-records: 1328 (loss: 2668571), 512k-records: 347 (loss: 10532891)
16777216: 17 files; 128k-records: 1412 (loss: 1256035), 512k-records: 362 (loss: 5974627)
33554432: 19 files; 128k-records: 3780 (loss: 1300824), 512k-records: 955 (loss: 6543704)
67108864: 3 files; 128k-records: 882 (loss: 159870), 512k-records: 221 (loss: 422014)
1073741824: 1 files; 128k-records: 6433 (loss: 16384), 512k-records: 1609 (loss: 409600)
/storj/nd9
Total amount of files: 6634934 (totalsize 1247864631628), 128k-records: 13639972 (loss: 144863522484), 512k-records: 7547816 (loss 253475548852)
4096: 2204846 files; 128k-records: 2204832 (loss: 1427453605), 512k-records: 2204832 (loss: 1427453605)
8192: 195260 files; 128k-records: 195260 (loss: 415096776), 512k-records: 195260 (loss: 415096776)
16384: 342005 files; 128k-records: 342005 (loss: 1288130734), 512k-records: 342005 (loss: 1288130734)
32768: 311579 files; 128k-records: 311579 (loss: 3315884317), 512k-records: 311579 (loss: 3315884317)
65536: 324748 files; 128k-records: 324748 (loss: 6434006313), 512k-records: 324748 (loss: 6434006313)
131072: 201166 files; 128k-records: 201166 (loss: 8274273639), 512k-records: 201166 (loss: 8274273639)
262144: 2491939 files; 128k-records: 4983878 (loss: 86781895178), 512k-records: 2491939 (loss: 86781895178)
524288: 292868 files; 128k-records: 918597 (loss: 24819556388), 512k-records: 292868 (loss: 57964388388)
1048576: 37389 files; 128k-records: 215771 (loss: 2596740300), 512k-records: 74778 (loss: 13520411852)
2097152: 34199 files; 128k-records: 378003 (loss: 1243630053), 512k-records: 112479 (loss: 10669410789)
4194304: 198741 files; 128k-records: 3555616 (loss: 8243901974), 512k-records: 993890 (loss: 63286801942)
8388608: 185 files; 128k-records: 6110 (loss: 22384289), 512k-records: 1666 (loss: 94998177)
16777216: 5 files; 128k-records: 498 (loss: 273280), 512k-records: 127 (loss: 1584000)
67108864: 3 files; 128k-records: 1142 (loss: 246486), 512k-records: 287 (loss: 1032918)
134217728: 1 files; 128k-records: 767 (loss: 49152), 512k-records: 192 (loss: 180224)
So, just a little bit below two times as much space used for meta data with 128k recordsize. But about 25-50% more efficiency loss. However, losses seem to be incredible high… I don’t see why.
Well, that’s what I assumed is the question. Though, I might not be aware of what these metadata records store. So…
Why does metadata size depend on file size? This is not the case in any of the file systems I know of, except maybe FAT. What is stored there, if not just the standard POSIX metadata like file name, permissions and timestamps?
With zfs metadata is understood to be literally everything that is not user data. We are talking file and directory information, allocation tables/btrees, information about pool configuration, any snapshots, etc.
If the file requires multiple records to store — there will be more metadata in the form of those records
Ah, thank you! This makes sense then.
If so, it still should be possible to simulate a small setup, and then assume metadata would scale linearly.
What are you counting as “loss”? Remaining unused record space? I.e when you write 1 kbyte files in 512k record? Are you counting 511k loss?
If so, don’t; zfs truncates the record, nothing is lost. That record becomes 1k record effectively.
No, when you have a file of 513k in 512k-recordsize system then 511k, since you need 2 records (if the file is not compressible).
If you write 1kb, I would expect a 4k record in a ashift=12-system.
So, to revisit:
Yeah, but what in case of 4.1MiB? In 512-recordsize, my understanding is it means 4.5MiB used disk space (if uncompressed).
I found a lot of information but it mostly feels like rule of thumb based on typical use cases. But storj is very special so I decided to run a real world test: 18TB HDD, 512GB ssd for metadata only, databases on another ssd. I will come back with results later.
What recordsize will you be using, and compression will be applied?
As far as I know compression must be enabled in order to make ZFS not using the blocksize for smaller files, so compression yes.
With 4MB recordsize storj pieces would always fit into one record but we might run into trouble when disk is almost full because of fragmentation? However, to avoid this we probably need to go much smaller which leads to much more metadata. So I will try 4MB first if OpenZFS allowing that size.
Actually, if you turn on compression even if it’s zle, it apparently prevents slack for every file. I don’t know why, I mean: why don’t make zle standard in the filesystem? But it’s how ZFS apparently works.
You can see this behaviour quite easy: on my 512k datasets, the compression factor is 1.1. on the 1M datasets, the compression factor is 1.2. So essentially compression only compresses slack space.
Storj data should be uncompressable like any other encrypted stuff and ‘slack’ is all zeros, so this is the expected result.
I turned on compression on my zfs drives just to avoid slack wastage. I think lz4 won’t waste time compressing uncompressible data.
incidentally, I’m using l2arc for metadata and compression seems very effective there. 1000GB of metada compressing down to around 66GB of actual SSD space.
What.
How much data, and how many files are there?
You can’t just look at L2ARC usage and conclude how much metadata is there. You need to look at output from zdb.
ooh, I don’t know how to use zdb, what tells me metadata size?
run zdb -PLbbbs <pool> and then sum up the ASIZE for types that are not plain file data or plain zvol data and not aggregates of other entries.
oohh, thanks, but that would take, like 13 hours on my 12TB drive. more time than I wish to spend.