You probably already knew about Transparent Page Sharing, Ballooning and swapping but let’s go through them again rather fast. For guest OS-es that don’t use large pages ESX will store the virtual machine memory in 4K pages in hardware memory and will use Transparent Page Sharing to see if there are duplicate 4K pages (at host level) and will only store them once. If there is memory contention at host level, ESX will start the ballooning process. Ballooning will try to reclaim unused pages from the VMs and return them to the host. If after ballooning there is still memory contention, ESX will start swapping VM memory to harddisk.

When host and guest OS are capable of using large pages, the VM memory will be stored in 2MB pages without searching for duplicates with TPS. This is because the chances of finding duplicate 2MB pages are far more smaller than with 4K pages and the time to do this comparison would take too much time, however a hash of the 4K pages within the 2MB pages is still created. With large pages, ballooning is the first memory saving technique, but if this doesn’t solve the memory contention the ESX host will start the process of swapping memory to disk. In this process the 2MB pages will be broken into small 4K pages and the pre-generated hashes are used to share the small pages before they are swapped to disk.

Memory compression

With memory compression an extra step is added in both scenario’s. Just before memory is swapped out to disk, ESX will compress the memory. That is after ballooning and TPS have not resolved the memory contention. With memory compression ESX will try and compress the 4K swap candidate pages. When a compression of more than 50% is obtained the page (now 2K or less) will then be stored in the compression cache. If a compression of more than 50% is not obtained, the 4K page will be swapped out (to disk). If in a later stadium this compressed page is access by a VM, it will first be decompressed and removed from the compression cache.

Compression cache

This compression cache is memory inside of the virtual machines memory with a maximum of 10% (can be changed through the advanced settings Mem.MemZipMaxPct) which means ESX will not allocate additional host physical memory for the compression cache. When the VM memory is undercommitted no space is allocated to the compression cache, but as memory pressure grows, the space allocated for compression cache grows to a maximum of 10%.

Should the memory pressure keep growing and more memory is needed, pages from the compressed memory cache will be decompressed and swapped out.

Compression performance

As with other memory management techniques VMware introduced, people often are a bit skeptic when first reading about it and performance is always the number on concern. To try and ease your mind a bit, some facts:

- ESX will not pro-actively compress pages

- Compression will cost about 2-3% host cpu time

- Compression takes about 20 micro seconds

- The penalty of the extra CPU time and compression time is small compared to having to swap out to disk

- The compression used is based on GZip but adopted to specific ESX needs

Memory overcommit, the basics.

In short: When you assign more RAM to your VMs than available in your host.

Good memory overcommit: When you assign more RAM to your VMs than available in your host BUT never cross the line where the amount of RAM that is USED by your VMs is more than available in your host.

Bad memory overcommit: When you assign more RAM to your VMs than available in your host AND cross the line where the amount of RAM that is USED by your VMs is more than available in your host.

A simple example:
Host has 48GB of RAM and just for the sake of argument we’ll pretend the hypervisor doesn’t use any RAM and we don’t have memory overhead per VM. I now start loading it with VMs that have 4GB RAM assigned. Without any memory overcommit I can load this host with 12 VMs of each 4GB.

Now let’s say, these VMs normally use only 2.5GB of RAM but sometimes they peak to 4GB. With memory overcommit I could now load the host with 19 VMs of 4GB RAM assigning a total of 76GB RAM and demanding 19 x 2.5GB = 47.5GB of physical memory. Even to me this is a bit on the edge, so I’d reserve some RAM for spikes and would go back to 17 VMs, which would leave me with 17 x 2.5GB = 42.5GB of actively used physical RAM, 17 x 4GB = 68GB of RAM assigned and therefore 68GB-48GB = 20GB of overcommitted RAM. So, 20GB of RAM I didn’t have to pay for. This is a good use of memory overcommit.

Bad use of memory overcommit is when in the previous example I would place more VMs on this host, to the point where the use of physical RAM is higher than the amount of physical RAM present in the host. ESX will start some memory optimization and reclaim techniques, but in the end it will swap host memory to disk, which is bad. It is essential to carefully monitor your hosts to see if you’re moving from good memory overcommit to bad memory overcommit.

Misconceptions on memory overcommit

-          Transparent Page Sharing (TPS) is a performance hit

-          Overcommit is always a performance hit

-          Real world workloads don’t benefit

-          The gain by overcommitment is neglectable

-          Overcommitment is dangerous

Transparent Page Sharing (TPS) is a performance hit
The mechanism below memory overcommit is called Transparent Page Sharing (TPS). Using this technique ESX scans if pages in memory for one VM are identical to pages of another VM by using hash values and if they are ESX doesn’t store that second page in physical memory but just places a link to the first page. When the VM wants to write to that page, then the link is removed and a real page in physical memory is created. This technique called Copy-on-Write (CoW) will incur an overhead compared to writing to non-shared pages.

What is transparent page sharing

The scanning of the guests performed by the hypervisor is done at base scan rate specified by the “Mem.ShareScanTime” which specifies the desired time to scan the VMs entire guest memory. By default ESX will scan every 60 minutes, but depending on the amount of current shared pages, ESX can intelligently adjust this scan frequency.

When performance is measured between a system with no page sharing, default page sharing or excessive page sharing (forced by the test), it shows that default page sharing performs 0,2% slower and sometimes 0,2% faster than without page sharing. Page sharing sometimes improves performance because the VM’s host memory footprint is reduced so that it fits the processor cache better.

Transparent Page Sharing influence on performance

Overcommit is always a performance hit

To be short, no it is not. But you probably expect more from me than just this. Let’s be clear about doing bad overcommit which is always a performance hit. As soon as your ESX hosts is out of physical RAM and starts swapping to disk, there is a big performance hit which should be avoided at all costs. Scott Drummonds (VMware) recently wrote about using Solid State Disks in your SAN to be used as swap space for your ESX host (http://vpivot.com/2009/12/24/solid-state-disks-and-host-swapping/), which would make less of an issue but still there is a performance hit.

But with good overcommit the performance hit is very low as you read in the previous section. There is also a fase in between good and bad overcommit and that is when your good overcommitted host is getting low on free physical memory. When this situation occurs, ESX will start looking for unused but claimed memory by the VMs and will try to reclaim memory by using a technique called ballooning.

The balloon driver inside the Guest OS is triggered by ESX to try and claim free pages by demanding memory (inflating) from the Guest OS. The Guest OS will then use page reclaiming algorithms to determine which pages can be assigned to the balloon driver. First the OS will assign pages marked as free and if this is not enough the OS will move less important pages to the OS swap file to free up pages and assign these to the balloon driver. Next the balloon driver tells ESX which pages it has received and since the balloon driver will not actually use them, ESX can safely re-use these pages for other VMs.

As with TPS, there is a performance penalty for this ballooning, but again the penalty is very low. In the graph below you can see that even with reclaiming 3/4 of the VMs memory, the performance penalty is only 3%. Looking at the performance impact of swapping host memory, we see that the throughput loss is about 34% which is a huge difference compared to the 3% for ballooning.

Impact of ballooning on performance

FIRST CONCLUSION: No performance hits !!!

The conclusion to draw from this, is that the performance impact of Transparent Page Sharing and ballooning neglectable and memory overcommit is NOT a performance hit !!!

Let’s move on to the remaining misconceptions on memory overcommit.

Real world workloads don’t benefit.
A common misconception is that TPS and ballooning will not work for the majority of applications. I have done a lot of searching for problems with applications in a virtual environment that could lead back to TPS or ballooning but could only find two situations in which there is a “problem” with TPS or ballooning. One situation is when running a Java application.

Copied from Scott Drummonds website:
“Java provides a special challenge in virtual environments due to the JVM’s introduction of a third level of memory management. The balloon driver draws memory from the virtual machine without impacting throughput because the guest OS efficiently claims pages that its processes are not using. But in the case of Java, the guest OS is unaware of how the JVM is using memory and is forced to select memory pages as arbitrarily and inefficiently as ESX’s swap routine. Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed inside the JVM and efforts by the host or guest to remove pages will both degrade Java applications’ performance. In these environments it is wise to manually set the JVM’s heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.”

The second exception I found is not really a problem but only something to keep in mind when working with the newer type of CPU’s that can do Virtualized MMU. Read this post from Duncan at Yellow-Bricks: “Virtualized MMU and Transparent page sharing” http://www.yellow-bricks.com/2009/03/06/virtualized-mmu-and-tp/. In this post Duncan explains that when using large pages (2MB) in combination with a CPU that uses virtualized MMU “the ESX page sharing technique might shares less memory when large pages are used instead of small pages” and “When free machine memory is low and before swapping happens, the ESX Server kernel attempts to share identical small pages even if they are parts of large pages. As a result, the candidate large pages on the host machine are broken into small pages. This can degrade performance at a point in time when you don’t want performance degradation. Keep this in mind when deploying VMs with virtualized MMU enabled, make your decision based on these facts! Do performance testing on what effect overcommiting will have on your environment when virtualized MMU is enabled.”

Second Conclusion: Workloads being not compatible with TPS or ballooning:

-          Java is the only known application that can have a performance hit when ballooning kicks in.

-          When using V-MMU compatible CPU’s in combination with large pages a performance degradation can be expected when ballooning kicks in.

The memory gain by TPS is neglectable

How much memory would you expect to gain from using TPS? Keep in mind that TPS works at page level, not only between VMs with the same Guest OS, but across all Guest OSes. Not just over VMs but also within the same VM. You could theoretically already save memory through TPS with just one VM running on your host.

Looking at real world examples provided me by a number of people running ESX in production, it is clear that TPS saves a lot of memory. See the table below:

Third conclusion: Memory savings by TPS:

-          As the table shows, memory savings thanks to TPS can be really huge and the price of memory saved by TPS is easily worth the extra buck for ESX licenses.

Overcommitment is dangerous

In the paragraphs above I think I made clear that there is no reason to NOT use TPS unless for specific applications. The only reason left for some people to not use TPS is that if you just keep on filling your host with VMs, it will soon start swapping (which is the bad form of overcommit) and they’re absolutely right. But may I ask this question: “Don’t you monitor what you are doing?”.

When I have a one liter bottle and I want to fill it with water, while filling it I usually keep an eye on (monitor) how much water I already pored into it and stop just before it’s full. I think this same practice can be used in many places, even in IT infrastructures.

Number of users on your Citrix hosts, I bet you monitor performance and start adding more hosts when certain limits will be reached. Or adding data to your disks, I bet you daily check to see if your disks aren’t near their limit. Creating snapshots of VMs in your virtual infrastructure, check your datastore if these snapshots aren’t filling up your space and please, do tell me you keep a sharp eye on those thin provisioned disk. Now, if you do all that and consider those techniques as save, I can’t see why overcommitment would be dangerous. Just monitor !!!

Fourth conclusion: Danger of overcommitting

-          There is no danger, no hidden risks, just as long as you monitor what you are doing.

I would to thank my twitter friends how promptly responded with esxtop screenshots when I called out for help on twitter. Thanks guys.

And a big thank you to Duncan and Scott for providing great posts which helped me put together this post. Thanks.

Sources:

After you installed your hypervisor you want to virtualize your physical servers on it. When doing the research for my presentation I was very surprised to find some strange limitations in which guest are supported with Hyper-V. Can you believe that not all Windows Server versions are supported? Well neither could I.

The guest OS support of Hyper-V:

• Windows 2000 Server is only supported with SP4 and 1 CPU.
• Windows 2003 Server is only supported with SP2 and 1 or 2 CPUs.
• Windows 2008 Server is supported with 1, 2 and 4 CPUs.
• and there is Linux support for SUSE Linux Server 10 sp1 / sp2 with only 1 CPU.

This does look quite reasonable at first glance, because these are current versions. True, but when looking at the ESX environment I work in now, we would not be able to virtualize as many servers as we have running on ESX right now. Let’s see:

• Customer has 721 VMs,
• 4 of them are running Linux (RedHat EL),
• then we have 2 running on Windows NT4 (YES you read it correctly, NT4),
• 8 on Windows 2000 mostly 2 CPU,
• 15 Windows 2003 4 CPU guest,
• about a 100 VMs are running Windows 2003 Sp1
• and the remaining is on Sp2 with 2 or less CPUs.

Now, I admit, we are not the quickest with upgrading our systems and sp2 is around for almost one year now, but we have many financial applications whose suppliers are also quite slow certifying their application for new service packs. This customer deals with millions of euro’s per day. They cannot afford downtime for this set of financial apps and will only upgrade to OS versions / service packs if the supplier supports it.

Looking at this environment, there is quite a number of systems that would not be supported on Hyper-V. Big deal you say? Actually, yes it is a big deal. Wasn’t cost saving by getting rid of a lot of physical hardware one of the main drivers to go virtual? Leaving a bunch of servers (around 130 in our case) run physical is quite a lot, especially because these will probably be somewhat older servers that are getting more expensive in their support contract and more vulnerable to failure.

VMware does support quite a broad range of OS’es and all with full capabilities, no strange cpu limitations. There are about 12 flavors of Linux, there is SUN, FreeBSD and of course Windows. Looking at the list, it seems ALL Windows Server versions are supported, from Windows NT4 to Windows 2008.

I can’t fully understand Microsoft on this. Of course I’m not a software engineer who understands anything about software limitations and what effect 1,2 or 4 cpu’s have on an OS and its service pack level, but still… why these support limits? I remember how hard it often is to get companies to run Exchange or SQL on VMware when they don’t get an official support statement from Microsoft. Now why is Microsoft doing this to themselfs? Is it really such an effort to get Windows 2003 supported on all service packs and on 1,2 and 4 cpu’s?

Memory over-commitment

One major point that Microsoft always mentions in his campaign, is that VMware’s transparent page sharing and memory over-commit techniques are not suited for the enterprise environment. Because, who would make VMs share memory? Well, let me tell you, at first I could agree with Microsoft on this. Surely I would never let my VMs starve for memory and never over-commit memory. I will constantly monitor my ESX environment and check how much memory my guest really need and make sure that the right amount of memory is present. Never let your ESX host (or other hypervisor) swap for memory to disk, never!!!

But wait. After reading some MS articles and VMware articles on this, I think the definition of memory over-commit is not as clear to everybody. What is memory over-commitment???
Memory over-commitment is when you assign more memory to your VMs, than the amount of memory present in your host.

Microsoft says this is not a smart thing to do and they tell their customers that you therefore don’t need this feature. Well, to be honest, VMware to thinks this not smart IF your VMs really need this amount of memory. And there is the big difference: IF

Now we all know those physical servers, the ones running with 2, 3, 4Gb of RAM because the application manual says that application X really needs 4Gb? We also all know, whilst monitoring these applications, the physical box never reached more than ½ the total memory on any given busy day? This is the point where VMware steps in.
If you have 20 VMs, with each 2Gb assigned but they all use no more than 1Gb, then you would be throwing away 10Gb of RAM in the host that never gets used. Now, this example is hypothetical, but you get the point. And in a real world scenario, I wouldn’t draw the hard line at 10Gb, but the point is that some VMs will stay way below their assigned memory and some will use all of it, however after running a couple of weeks, you will find a stable average of memory and peak memory usage, thereby enabling you to scale your hosts memory profile.

I’ve taken a quick snap of the environment of one of my customers. They now have 16 ESX hosts in one cluster. In the first column you see the amount of memory that is present in the host. Second column shows the amount of memory that is assigned to the VMs. Third column is the amount of memory over-commitment.

What does this table tell us? The total of physical memory on all their hosts is 688 GB of RAM, but on some host they have assigned more RAM to VMs then physically present. A total of 102 GB more!!! That is no typo, that really is one-hundred and two Gigabytes of RAM. VMware ESX saved them buying 102 GB by allowing memory over-commit. To me, that is quite a saving, a saving you won’t get using Hyper-V. Actually, there is more.

Transparent Pagesharing

Say, we have 10 VMs running Windows 2003 Server and in each VM there is about 500Mb of identical data, because it is just the basics of Windows 2003. So you have 4,5GB (5GB – 500Mb) of redundant data in your expensive RAM. Not, with VMware transparent page-sharing you haven’t. So there is another great memory saver. Now look at the bigger picture again and see how much memory would be saved if you have 400 Windows VMs and add those savings of memory over-commitment…… I think you get the point and why you can really benefit from these techniques.

Thank you Tom Howarth for checking this post before publishing.

Series:
Hyper-V, not in my datacenter (part 1: Hardware)
Hyper-V, not in my datacenter (part 2: Guest OS and Memory overcommit)
Hyper-V, not in my datacenter (part 3: Motions and storage)

http://virtualgeek.typepad.com/virtual_geek/2008/06/so-how-exactly.html