Memory management and compression in vSphere 4.1

With vSphere 4.1, VMware released a great new feature called Memory compression. At first, after reading the release notes I thought memory compression was just one step before swapping to disk would occur. However, after reading the whitepaper “Understanding Memory Resource Management in VMware ESX 4” I learned some more details I want to share with you. For the full understanding do read the whitepaper as this post is a quick summary of how memory compression works, while the whitepaper gives you an in-depth view on memory management and also gives a lot of performance info on these techniques.

You probably already knew about Transparent Page Sharing, Ballooning and swapping but let’s go through them again rather fast. For guest OS-es that don’t use large pages ESX will store the virtual machine memory in 4K pages in hardware memory and will use Transparent Page Sharing to see if there are duplicate 4K pages (at host level) and will only store them once. If there is memory contention at host level, ESX will start the ballooning process. Ballooning will try to reclaim unused pages from the VMs and return them to the host. If after ballooning there is still memory contention, ESX will start swapping VM memory to harddisk.

When host and guest OS are capable of using large pages, the VM memory will be stored in 2MB pages without searching for duplicates with TPS. This is because the chances of finding duplicate 2MB pages are far more smaller than with 4K pages and the time to do this comparison would take too much time, however a hash of the 4K pages within the 2MB pages is still created. With large pages, ballooning is the first memory saving technique, but if this doesn’t solve the memory contention the ESX host will start the process of swapping memory to disk. In this process the 2MB pages will be broken into small 4K pages and the pre-generated hashes are used to share the small pages before they are swapped to disk.

Memory compression

With memory compression an extra step is added in both scenario’s. Just before memory is swapped out to disk, ESX will compress the memory. That is after ballooning and TPS have not resolved the memory contention. With memory compression ESX will try and compress the 4K swap candidate pages. When a compression of more than 50% is obtained the page (now 2K or less) will then be stored in the compression cache. If a compression of more than 50% is not obtained, the 4K page will be swapped out (to disk). If in a later stadium this compressed page is access by a VM, it will first be decompressed and removed from the compression cache.

Compression cache

This compression cache is memory inside of the virtual machines memory with a maximum of 10% (can be changed through the advanced settings Mem.MemZipMaxPct) which means ESX will not allocate additional host physical memory for the compression cache. When the VM memory is undercommitted no space is allocated to the compression cache, but as memory pressure grows, the space allocated for compression cache grows to a maximum of 10%.

Should the memory pressure keep growing and more memory is needed, pages from the compressed memory cache will be decompressed and swapped out.

Compression performance

As with other memory management techniques VMware introduced, people often are a bit skeptic when first reading about it and performance is always the number on concern. To try and ease your mind a bit, some facts:

– ESX will not pro-actively compress pages

– Compression will cost about 2-3% host cpu time

– Compression takes about 20 micro seconds

– The penalty of the extra CPU time and compression time is small compared to having to swap out to disk

– The compression used is based on GZip but adopted to specific ESX needs