Correct. That is why I have executed the speed test on europe as well for better comparison. You can see the difference in speed. My target is to optimize for customer data and not for test data even if that would be beneficial short term.
europe is still europe west for me. I know that we have a europe north as well.
in my command tar simply pack many files in one stream without compression, next send it to pipe viewer where we can see current and average speed, next send data to null ie trash it. This emulate reading many files with actually random file order.
i really hate the du commands but doing those now to compareā¦ tho most of my storagenode was written with 256k records so should be interesting.
great i did a du for the entire poolā¦ lol
How do I know which folder is for which satellite? The folder name is not satellite ID, but something else.
# du -hd0 -B1 /zPool
11743747000832 /zPool
# du -bd0 -B1 /zPool
11797343601454 /zPool
# du -bd0 -B1 /zPool/storagenodes/storj
7552277561142 /zPool/storagenodes/storj
# du -hd0 -B1 /zPool/storagenodes/storj
7697093245952 /zPool/storagenodes/storj
i get this on mineā¦
the biggest one of the storagenode dataset turns out to be 7TB exactly which is what zfs reportsā¦ so basically no point in doing du when i instantly get the same result from zfs
did seem like that arc finally figure out what i was doing nowā¦ took forever on the first scan but then again it is 10tb and iām drive short so running on degraded which makes my performance a bit low
this is satelliteID in Base32
like a pro lol
do i need to install something to do those tar, pv apt-gtar commands?
thats a really interesting and good pointā¦ not sure i can apply that sadlyā¦ as my SLOG lives on an old SATA controller on my mobo, so its limited to like 300MB/s maybe 600MB/s at best and thus 5sec is a pretty good current spot for my settingā¦ else i would most likely end up saturating my SATA disk bandwidth depending on how zfs is at load balancing and prioritizing in this case.
(did test that the old SATA isnāt introducing latency, so for now until i got more bays, this will have to do
and its really just the low latency i need, the throughput in this case is almost irrelevant)
but i will most def write the command down into my little zfs list of commands and what they doā¦
not always easy to find whatever is relevantā¦
any of you understand how to change the L2ARC feedrateā¦ been wanting to look at thatā¦ or the metadata ratioā¦ havenāt been able to find a straight answer on how this is done.
but if its to convoluted ill just wait awhile until i got better time for itā¦ feedrate isnāt to unreasonableā¦ but still takes 72hours for my server to warm up lol which can be a bit annoying
ofc persistent l2arc is around the corner soon hopefullyā¦ so maybe not worth the hassle of doing anything really aside from upgrading when ZoL becomes OpenZFS v2.0
Okay rookie questionā¦
how do i stop whatever i got half started withā¦
# tar -c /zPool/storagenodes/storj/storj/storage/blobs/v4weeab67sbgvnbwd5z7tweqsqqun7qox2agpbxy44mqqaaaaaaa/a2/* | pv -brap > /dev/null
-bash: pv: command not found
tar: Removing leading `/' from member names
Like you do most Linux commands - Ctrl+C
but it wasnāt active in the console, it vanished into the background like with && or is it just &
ended up rebooting, had to try and fix an issue anywaysā¦ so good enough excuse lol
think i got my drive back online thoā¦ so thatās a winā¦ fāing corroded backplane is killing meā¦ lol
but pretty sure i got the bays with issues isolated nowā¦ but will be interesting to see how the resilvering goes because i zeroed the drive, and then the first scrub should also be kinda interestingā¦ its been almost a week and iāve been doing crazy stuff lol zfs gets so mad when i pull more than two drives in a running zpoolā¦ amazing how it just recovers without loosing a byte thoā¦
Running that inside storage/blobs/6r2ā¦ gives me:
17 4608
28 1049088
62 2105856
170 2335232
418 2630144
452 1896960
10316 1057280
76033 2331136
Not sure I interprete this correctly but if I do, it means that 76k files have a size of ~2.33MB.
Would still mean that most data is 2.33MB big, not only the testdata.
Get some contact cleaner and clean it.
i will when i got some downtime to spare or get another server setupā¦ donāt have any more drives to put in it anywaysā¦ but yeah i intent toā¦ just difficult when its the only thing that can server my storagenode presentlyā¦ and i got way to much downtime already from all kinds of issuesā¦ so iām just trying not to get suspendedā¦ if thats active yetā¦ 
if nothing else its good practice for when it isā¦
thanks for the tip tho, had been pondering how best to do thatā¦
when the fan wires corrode over and vibrate till they breakā¦ take a hint and control the humidity and temperature of the room xD
Which satellite is it? If you go over your entire disk you will hit a lot of test data. You would need to select a specific satellite like us-central, asia-east or europe-west. They should have less or maybe even no testdata.
we where looking at file sizes and ended up making an assumption, that the maximum piece size was 2.something MB, because most files seemed to cap out at that sizeā¦ just seemed like that made sense at the timeā¦ ofc you got some insider knowledgeā¦ or more educated knowledge than our hipshotā¦
is there a fixed max size of pieces or can they basically grow unlimitedā¦ i sort of assumed so since this is basically live data, that like a filesystem and harddrive sectors they would have certain max sizes so that customers could stream parts of their files depending on what they demanded, instead of being forced to download an entire fileā¦ like say if they where to (i duno whats possible and not) to edit video straight of the data on the tardigrade platform.
you get the ideaā¦ so you can seek to the middle of a big file by getting a piece that represents the middle of the files ā¦ object or whatever its called.
BTW have you set autotrim on? its off per default in ZoL, gives you like 30% better performance on ssdās if you donāt like the trimming of deleted data ofc iām sure it doesnāt get easier to recover after trimmingā¦ xD
zpool set autotrim=on or off poolname
if you are interested in the details about it.
and in case anyone want to go full retard on zfs configuration, iāve used this a lot by nowā¦
seems this is how to do that ARC stuff iāve been looking forā¦ but havenāt dared to do that deep dive yet
https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/zfs-advanced.html
6r2fgwqz3manwt4aogq343bfkh2n5vvg4ohqqgggrrunaaaaaaaa
not sure which one that is. Got about 170GB in that directory so it isnāt stefan-benten or saltlake.
That should be europe north with a lot of testdata
this is kinda interestingā¦
had disk trouble, which might have been backplane troubleā¦ no surprise there, pulled the drive did some tests on another comp deleted the partition and re partitioned it, just because i wanted to see how my resilvering time was while i was at it.
all tests seemed kinda fine, so i put the drive back and zfs resilvered it inā¦ maybe under an hourā¦
maybe lessā¦ which seemed kinda oddā¦ but okay kinda knew that would be BS
tho nice to know what one cannot always count on a resilvered drive to have verifiable good data.
ran a scrubā¦ and now itās thrown me 3.14k cksum errors thus far and itās only 28% doneā¦
xD seem more rightā¦
but watch out for resilvering, always scrub afterwards. else your data might be wrong or gone
10322 16896
16309 12800
30719 8704
58167 2331136
125921 4608
on qstuylguhrn2ozjv4h2c6xpxykd622gtgurhql2k7k75wqaaaaaa
and on ukfu6bhbboxilvt7jrwlqk7y2tapb5d2r2tsmj2sjxvw5qaaaaaa:
8110 258560
8962 16896
13696 12800
20929 221696
26330 8704
49530 2331136
109895 4608
So this does clearly show that the majority of files are about 4k-16Kā¦ But also a lot of files with 2.33MB.
The question is: Does it make more sense to optimize the recordsize for big files or for small files? Also would the difference be visible in successrates for uploads and especially for downloads?
What other consequences does optimizing have?
Right now on most nodes (oddly enough) the download rates are >99% so there wonāt be any visible change. In theory however reading and sending a file should be faster if less IOPS are needed. For 4K files that makes no difference as the recordsize is already bigger but with 2.33MB files there is a difference of 3 IOPS with 1MB recordsize and 37 IOPS with 64K recordsize. But I have no idea how much difference this would actually make in reading latency and will also depend on the general load on the drive (which is probably mostly defined by the db load?).
Similarily if the file is 4K but the recordsize 1MB then reading takes a lot longer than needed, maybe 4ms with 4k recordsize (assuming 1MB/s reading rate) and 16ms with 1MB recordsize (assuming 60MB/s reading rate)?
I am however not sure how zfs compression will play into this if you have it activated (which you probably should, at least with higher recordsizes). Would a 4k file be compressed to a 4K record and when reading would then only be 4K read and not a whole record of 1MB?
Optimizing for uploads is hardly possible because afaik the writes are async and therefore independent of the recordsize, unless the drive canāt handle the load at some point and needs to reject pieces/becomes too slow. Only then recordsize might help to reduce the IO load on the drive. To reduce the IOPS youāll need a bigger recordsize but if the majority of files is 4K then you will have a bigger recordsize anyway and it will probably make no difference for the IOPS if you have 64K or 1MB as recordsize.
So my conclusion is that the recordsize might reduce the IOPS needed, which might help some SMR drives (idk?) but (without having proof) I guess it doesnāt make much of a difference and the DB load might be a bigger problem. I will definitely test that with the new release when I can put all the DBs onto my SSD.