Why? Decompressing pages is still faster than paging them back in from disk. And honestly you should be never be in situation when you have to use paging. It’s too late, performance is gone aalong with disk cache.
Remember, only pages that would have been otherwise paged out are compressed. There is no reason to disable that.
It was sugested by someone who has a website/blog that deals with Synology ecosystem. I realy don’t understand what it is and what it does, but I imagine compressing and decompressing something takes CPU cycles and drops performance. And since I have 18 GB of RAM, I thought dissabling it is the best way. And on my 216+ with 1GB RAM and 2 nodes, the DSM is pretty responsive and nodes are doing OK, without memory compression.
https://mariushosting.com/synology-should-i-use-memory-compression/
I see the description Syno has put out there, but I imagine a scenario were data is compressed and forgot, and it is building and building occuping a lot of memory; and CPU trying to keep track of what’s used and not, and trying to compress and decompress things by some algo, and maybe the implementation isn’t perfect and becomes overhelmed. I don’t know if Marius is right, if the system is good or bad, but since I don’t see usage in disk cache, I will limit the things that can use computation, including mem compresion, off.
Yeah, I’ve seen that exact article,
“Even though they claim that this feature improves system responsiveness, in reality, Memory Compression slows everything down.”
Yeah, totally, a bunch of engineers created a feature to slow down a system. And this, in his mind, makes total sense.
Actually, after reading a few of his musings years ago, I’ve blocked that resource from appearing in my search results with uBlacklist 
. (highly recommend this plugin – works with most search engines, and over time will make your search experience much better by eliminating spammy content farms from appearing in your search results)
Anyway, very briefly:
Memory is managed in blocks, called “pages”. If the applications or OS services need more ram than available, the page manager saves the least recently used pages to disk, to free up space in ram for new allocations. Then, if the application whose data has been paged out to disk tries to read that data – an exception, called Page Fault, is generated – because that page is no longer in ram. That is intercepted, additional memory is freed (likely by moving pages belonging to other apps to disk), requested pages from disk are read back into ram, and the read request is retried, this time succeeding.
Because this involves reads from disk – it’s very slow.
When memory compression is enabled, instead of paging data to disk to free space, data is compressed in-place, with a very fast compression algorithm like LZO, LZ4, or ZSTD. This has virtually no impact on CPU utilization, any modern CPU won’t even notice extra workload.
Compressing memory, however, is orders of magnitudes faster than writing and reading stuff to disk. Hence, if your system is under such stress that it has to keep paging in and out all the time, enabling memory compression will help improve performance.
So worrying about extra CPU cycles is moot: the alternative is to have the CPU be stalled on an IO request waiting for the read from disk, for orders of magnitude longer.
My second point was, however, that if you find your system struggling to find free ram – it does not matter what you do, it’s too late, you need to add more physical ram, because you want to have free unused ram available for file system cache.
Wouldn’t this make the statement in the article advising to disable memory compression and add ram instead true? No. Because there is no reason to disable memory compression. Depending on the implementation, compressing old very infrequently used pages in ram can help free up even more ram, for the benefit of the disk cache, that ultimately significantly improves performance.
@arrogantrabbit
My memory isn’t the best (I’ll wait for un upgrade from Elon 
), but I remember doing some tests runing file walker with compression on and off, and disabling it showed a clear advantage. I think they are somewere in the File Walker thread? Anyway, for machines built only for storagenodes, I believe mem compression off is the best option. Theory can only take you so far… the practice reveals the best answear. 
I did’t dug up to much about how mem works, but if the pages aren’t used, wht they are still kept there and not discarded? Why keep them in ram or on disk? Or is an expiration period set in place for that?
Found it. It seems not a great impruvement, but backthan I just had 2GB RAM.
https://forum.storj.io/t/tuning-the-filewalker/19203/100?u=snorkel
I have a ton of questions re: your measurement methodology (e.g. how did you ensure same system state before runs with and without memory compression, to ensure validity of your correlation), but with just 2GB this would be pointless. There is no reason to hunt for last few percent making the systems agony marginally less painful, when adding ram will give you 10x improvement.
Because an application allocated that memory. You can’t just discard it.
Strongly disagree. If practice disagrees with the theory – you did not take all factors into account in the theory or the experiment is faulty. With memory compression the “theory” is flawless. It’s always beneficial. It’s for that reason it’s by default enabled on every Windows10+ and macOS machine.
And finally, why are you benchmarking file walker to begin with? It does not matter how fast it goes. Running file walker faster does not increase your payout. You shall optimize average response time, not speed of enumerating files… And for that – you want to be able to fit all metadata into ram. And it’s an entirely different ball game.
Yes, I was thinking now about those tests, being not representative for a normal daily workload. Maybe in the long run you are right. Time for some more tests .
Au contraire, mon ami.
The scientific method is underpinned by experimentation, and if the experimental results don’t support the theory it’s the theory that needs rethinking.
We’re at the stage of validating unsubstantiated claims and/or doubting quality of measurements performed that resulted in that dubious paragraph in the article, blindly recited by many.
Memory compression is not a groundbreaking new bleeding edge development, it’s a pretty mundane thing that has been widely adopted and validated by millions across OSes and hardware types. There is absolutely no room for doubt, and if observations differ – there is something else not taken into account in the testing setup. This is one of those things when there is no downside. None.
An analogy would be if you try to measure acceleration of a rock at the surface of the earth in free fall and get 15 m/s2, you don’t rethink theory of general relativity, or doubt the value of gravitational constants; instead you go and look for a definitive fault in the measurement methodology, and don’t stop until you find one. I could bet anything if you measure g=15, it’s you who is wrong, not the theory of gravity.
In the same vein, if you turn on memory compression and see worse performance – there is a fault somewhere in how you measure performance, or environment, etc.
Or a bad implementation…
Which won’'t be out of character for Synology indeed! Even though they should have no business messing with linux internals, but I know painfully well that they do. Oh gawd.
So, on the 18GB machines, the swap file is not accesed, even with mem compression off. There are 2GB reserved for it, used like 250MB, but never accesed. So, there is no point in testing, because I will not find anything, maybe just worse performance if the mem compression is realy bad implemented and takes up resources.
With the 1GB machine, maybe there we can have something, but, not realy sure if I want to spend my time with this, because I don’t realy know what to look for and if it’s relevant.
If you have an ideea of how to conduct the tests, the exact methodology, please let me know, and will see.
The machine specs: Syno 216+, 1GB RAM, running 2 nodes on 2 IronWolfs, 6TB and 4TB out of 8TB. Machine not used for anything else. Mem compression off, antivirus off, firewall off.
No Graphana, Prometheus, logs on fatal, Filewalker off.
I can enable the usage history in DSM and I can access the debug metrics from storagenodes.
I think such a memory constrained system should show discernible benefit from memory compression: Compressing infrequently accessed pages leaves more room for filesystem cache (mainly, metadata cache), and larger cache should improve response time of you node by increasing chances of cache hits for metadata lookups, thereby reducing average time-to-first byte. The smaller the amount of ram – the more dramatic will be the effect, to a degree. If there is way too little ram – the cache will either be nonexistent or thrashed all the time, and memory compression will make no difference – if there are virtually no cache hits ever – it does not matter if now you get twice more, it’s still virtually nothing.
I would enable compression, reboot, run the node for a week, and draw a histogram of times to first byte on download requests (see if this can be extracted from the storagenode logs), or perhaps rate of race wins… Then turn it off, reboot, and run for another week, and measure the same thing.
There would be some variability in access patterns, so it won’'t be a clean experiment, but it shall provide some insight.
it will be very surprising. Storage node uses almost no CPU time. Entire CPU is idle.
You know what, estimate amount of metadata on your node (depends on filesystem and number of files). If you get a number comparable to amount of ram – it makes sense to experiment. If not – don’t bother. The system is too undersized and will struggle no matter what.
Here is your problem: you’re comparing wrong alternatives.
The alternatives are:
Memory compression OFF: performance may be lost when memory swapped to disc. If swap file is not used - no performance loss.
Memory compression ON: performance may be lost with:
So, asserting that “if memory will not be compressed - the system will slow down to swap pages” may be incorrect. The scenario may likely be: “if memory will not be compressed - nothing else will happen (no swapping)”. And that’s why disabling memory compression theoretically may result in significant system performance increase.
Hehe, yes, and no.
You are correct if you only consider generic use cases under heavy memory use.
This is where you are wrong. For storragenode software disk cache provides dramatic difference in node reponsiveness. You want as much free ram as possible. Compressing old pages helps free up more ram, than otherwise would be feasible, to cache more [meta] data.
In fact, this is not specific to storage node in any way. Even desktop OSes do that. Here is a screenshot from the machine I"m typing this on (not as dramatic, because there is shit ton of ram, that allows for a massive disk cache, but nevertheless some shit has been compressed:
Think about it. OS went as far as compression other memory, to make that cached section larger. For all the reasons I’ve described.
This makes zero sense. The overhead is less than the savings from compression, otherwise compression is bypassed.
No swapping happens for program code because it makes no sense to do so – the program code is already on disk. You just throw away pages when you need extra ram, and load it back from the disk. Compression won’t work on a memory mapped file.
And by the way, it makes a lot of sense to compress data before swapping. Because it’s less data to read on page-in.
I never said that.
This conclusion does not follow from the previous statements. Memory compression is so light on resources that it’s enabled by default on all modern OSes, including in low power mode.
TLDR: Compressing stale data allows to free space for disk cache, that you can’t have enough of if you want to run storage node.
