At the Dutch VMUG event 2011 I gave a presentation on how to check your VMware environment to make sure it is healthy. When creating the presentation I had a lot of doubts because I was afraid everyone would think these points were very obvious. But on the other hand, when visiting customers and doing these health checks for them, I found a lot of those “obvious” issues in their environment. I decided to stick to my plan and test the audience and it turned out they were very happy with my presentation and I saw a lot of people in the audience taking notes. The replies afterwards also showed that for many people there were a lot of eye-openers in this presentation. I therefore decided to convert the power point presentation into this blogpost and hope my readers find it a valuable health check.

In the top menu bar of this blog you’ll find the “Health Check” section. There you can find the various pages that together give you a complete health check.

See full post at: VMware vSphere Health Check

Today I visited the VKernel stand where I met Bryan Semple. He had some great news. Next week VKernel will release a new free tool called the VKernel vScope Explorer which will be great for every vSphere admin. It will give you the ability to show any problems in your environment for up to 8000 VMs. Although this free version is ment to be an overview and will not show you everything in deep technical details, it does give you very good info on the problems a VM is having. VKernel vScopre Explorer is built on the same engine of the full product and will give you similar results, it is almost a light version of the new VMware vCOMS only from VKernel instead of VMware.

Check their website for the official release.

See full post at: Free vScope Explorer from VKernel

End of 2009, beginning of 2010 I put an Iomega StorCenter IX4 to the test and published the results in these two blogposts:
Putting your storage to the test – Part 1 iSCSI on Iomega StorCenter IX4-200D and part 2. Recently Iomega sent me their new StorCenter PX4-300R for testing purpose and I was happy to run some Iometer stress tests on it. I was very anxious to learn if the PX4 would be able to outperform the “good old” IX4.

What’s in the box?

The PX4-300R is a rack mountable model of 1U height and comes with rails and the necessary power and UTP cable. When I read the several websites where the unit is sold, it normally comes without disks. The unit I received had four Hitachi 2.0TB HDS723020BLA642 disks in it configured in RAID5, giving me 5.4TB of disk space. Setup is easy, you plug the UTP cable and power cable, boot the unit and check your dhcp server to see what address it picked up. Then there is a web interface that gives you all the management options you need.

Performance

Despite many features the PX4 holds, I was most interested in the performance of this thing. In the 5.4TB volume I created a big NFS volume and a 500GB iSCSI volume. Connected my vSphere5 ESXi hosts to it and moved my performance test VM on to the volume.  This Windows 2003 VM is running on HW level 7 with 1 OS disk and a 20GB test disk (all VMDKs of course). The first set of test where run with a non-aligned disk, it was formatted in Windows 2003. The aligned tests where run with a disk formatted by Windows 2008 and then reattached to the Windows 2003 server.  For testing I used Iometer with the same config file I ran my IX4 tests, it creates a 4GB dummy file on the disk and performs the reads and writes on that dummy file.

The results

When looking at the results, it clearly shows the PX4-300R is running circles around the IX4. It also shows that for the PX4-300R iSCSI performs a little better than NFS, which was completely different for the IX4 that showed a clear advantage for NFS over iSCSI.

Max Throughput-100%Read

Config IOPS MB/sec PX4 iSCSI Aligned 3.500 114 PX4 iSCSI Non-Aligned 3.486 114 IX4 iSCSI Non-Aligned 1.761 55 PX4 NFS Aligned 3.372 110 PX4 NFS Non-Aligned 3.363 110 IX4 NFS Non-Aligned 3.475 108

Max Throughput-50%Read

Config IOPS MB/sec PX4 iSCSI Aligned 5.267 172 PX4 iSCSI Non-Aligned 5.142 168 IX4 iSCSI Non-Aligned 705 22 PX4 NFS Aligned 4.678 153 PX4 NFS Non-Aligned 4.655 152 IX4 NFS Non-Aligned 684 21

Random-8K-70% Read

Config IOPS MB/sec PX4 iSCSI Aligned 520 4 PX4 iSCSI Non-Aligned 485 3,9 IX4 iSCSI Non-Aligned 64 0,5 PX4 NFS Aligned 145 1,1 PX4 NFS Non-Aligned 129 1 IX4 NFS Non-Aligned 90 0,7

RealLife-60%Rand-65%Read

Config IOPS MB/sec PX4 iSCSI Aligned 645 5,2 PX4 iSCSI Non-Aligned 604 4,9 IX4 iSCSI Non-Aligned 89 0,7 PX4 NFS Aligned 189 1,5 PX4 NFS Non-Aligned 179 1,4 IX4 NFS Non-Aligned 112 0,8

Performance recap

When looking at the charts it is clear that the PX4-300R performs much better than the IX4, sometimes up to 7 times faster. It is good to see that iSCSI and NFS perform almost equal on read performance but surprised to see that NFS can’t keep up with iSCSI in ‘real-world’ tests.

Partition alignment

Running the tests doesn’t take that much time so I decided to also check if partition alignment would really make a difference by formatting one disk with Windows 2003 and one disk using Windows 2008. In Windows 2008 partitions are automatically aligned when formatted. As you can see from the results there is a clear difference between aligned and non-aligned partitions. For heavy read the difference isn’t that big in percentage, but still a 100 IOPS, for the other tests you can see a difference of between 5-7%. That’s free IOPS you get back when making sure you align your partitions.

Conclusion

For the workload I would like to use it for, which is running as my homelab shared storage and store some video’s and foto’s, this would make a great NAS. It holds a number of other features as well, like automatic RSYNC of files to other rsync enabled devices, syncing with cloud services and creating a personal cloud. But I don’t care so much for them personally. What I find important is that it feels fast, almost like I’m working on enterprise storage. Of course it can’t compete with enterprise storage but with the few VMs running in my lab, this is great stuff. The PX4 is more expensive than the IX4, but then again it performs up to 7 times faster in some tests. For a complete overview of the different products, see the Iomega Network Storage overview page.

Test tools

If you want to perform these tests your self, you can download this Gabes-PerfTest.zip which includes iometer and the test configuration file. Would be great if you post your results in the comments.

See full post at: Performance test of Iomega StorCenter PX4-300R

In June 2011 I was part of the TechField Day #6 delegation and one of the companies we visited was VKernel. During the VKernel presentation we, the delegates, had a number of comments on the layout / GUI part of vOPS. Only a few months later, VKernel shows they have been using our feedback and of their customers and made some pretty good adjustments to the GUI with the release of VKernel vOPS 4.

Of course that is not the only change in vOPS 4. One of the biggest new features is that VKernel vOPS is now “Hypervisor Agnostic”, which specifically means that besides VMware vSphere, they now also support Microsoft Hyper-V and will be supporting RHEV in Q4 2011.

VKernel vOPS 4 is not only Hypervisor agnostic in a way that it retrieves performance data from both VMware vSphere and HyperV, but the VKernel appliance can of course also be deploy as VMDK and VHD file.

A considerable change in the GUI is the instant drill-down to VM level from a high level view. This high level view shows you the health of all your VMs in one glimps. Each VM is represented by a single square, the more VMs the smaller the squares get. Even with 2000 VMs you can still get a fairly good indication of your complete environment within seconds. VMs in good health are green, VMs with a slight problem are yellow and if things are really bad…. You guessed it, the square that represents the VM turns red.

Except for performance, it is also important to know if you are using your resources in a cost effective way. VKernel came up with a new feature called Cost Optimization. By using a cost index per VM, vOPS 4 can tell you if your VM is being used cost effectively. The index is calculated based on hardware costs, resource usage and VM density.

When talking about this with Bryan Semple (CMO VKernel) I had my doubts about the calculations made. Using the VKernel cost index, a VM using a lot of resources on very expensive hardware, would get ‘punished’ for using these resources on an expensive host. Is that fair? An expensive host often has the most power, fastest RAM, etc and maybe you should run the more resource intensive VMs on the more expensive hosts. But who is to blame if the VM is not running where it should be running? It isn’t the VMs fault. So maybe the index should be more host oriented instead of VM oriented. Bryan thinks I have a point there and he will discuss this with the development team.

More details can be found at the VKernel website. http://www.vkernel.com/products/voperations-suite/what-is-new?src=box1

See full post at: VKernel presents vOperations Suite 4

With vSphere 4.1, VMware released a great new feature called Memory compression. At first, after reading the release notes I thought memory compression was just one step before swapping to disk would occur. However, after reading the whitepaper “Understanding Memory Resource Management in VMware ESX 4” I learned some more details I want to share with you. For the full understanding do read the whitepaper as this post is a quick summary of how memory compression works, while the whitepaper gives you an in-depth view on memory management and also gives a lot of performance info on these techniques.

You probably already knew about Transparent Page Sharing, Ballooning and swapping but let’s go through them again rather fast. For guest OS-es that don’t use large pages ESX will store the virtual machine memory in 4K pages in hardware memory and will use Transparent Page Sharing to see if there are duplicate 4K pages (at host level) and will only store them once. If there is memory contention at host level, ESX will start the ballooning process. Ballooning will try to reclaim unused pages from the VMs and return them to the host. If after ballooning there is still memory contention, ESX will start swapping VM memory to harddisk.

When host and guest OS are capable of using large pages, the VM memory will be stored in 2MB pages without searching for duplicates with TPS. This is because the chances of finding duplicate 2MB pages are far more smaller than with 4K pages and the time to do this comparison would take too much time, however a hash of the 4K pages within the 2MB pages is still created. With large pages, ballooning is the first memory saving technique, but if this doesn’t solve the memory contention the ESX host will start the process of swapping memory to disk. In this process the 2MB pages will be broken into small 4K pages and the pre-generated hashes are used to share the small pages before they are swapped to disk.

Memory compression

With memory compression an extra step is added in both scenario’s. Just before memory is swapped out to disk, ESX will compress the memory. That is after ballooning and TPS have not resolved the memory contention. With memory compression ESX will try and compress the 4K swap candidate pages. When a compression of more than 50% is obtained the page (now 2K or less) will then be stored in the compression cache. If a compression of more than 50% is not obtained, the 4K page will be swapped out (to disk). If in a later stadium this compressed page is access by a VM, it will first be decompressed and removed from the compression cache.

Compression cache

This compression cache is memory inside of the virtual machines memory with a maximum of 10% (can be changed through the advanced settings Mem.MemZipMaxPct) which means ESX will not allocate additional host physical memory for the compression cache. When the VM memory is undercommitted no space is allocated to the compression cache, but as memory pressure grows, the space allocated for compression cache grows to a maximum of 10%.

Should the memory pressure keep growing and more memory is needed, pages from the compressed memory cache will be decompressed and swapped out.

Compression performance

As with other memory management techniques VMware introduced, people often are a bit skeptic when first reading about it and performance is always the number on concern. To try and ease your mind a bit, some facts:

- ESX will not pro-actively compress pages

- Compression will cost about 2-3% host cpu time

- Compression takes about 20 micro seconds

- The penalty of the extra CPU time and compression time is small compared to having to swap out to disk

- The compression used is based on GZip but adopted to specific ESX needs

See full post at: Memory management and compression in vSphere 4.1

Some time ago I wrote a posting on how to use vscsiStats to gather even more data from your VMs and their SCSI performance ( See: Using vscsiStats – the full how-to). Last week I received an e-mail from Paul Dunn who had written an Excel macro that can read the output from the vscsiStats exported csv file and convert it into Excel histograms.

Using the macro is very straight forward. First you let vscsiStats run for a while and have it export the data to csv file. For example with the following command (Do pay attention to just one capital S in vscsiStats):

This will start vscsiStats and with the following command you stop the collection of data. Keep in mind that this does have a little performance impact, so don’t let it run forever.

After this you download the /root/vscsiStats-export.csv file to your desktop and start Excel. Load the vscsiStat-export.csv file and open the macro window by pressing ALT+F11. Now insert the VBA code you’ll find below into the new macro sheet. After pasting it you can run the code by pressing F5. After the macro has finished you will see that it created a number of charts that are much easier to read than the raw data coming from vscsiStats.

Warning: If you just include all data from all VMs it can easily generate over 100 or 200 charts !!!

Download the macro vscsiStats-excel-macro.

EDIT: The excel macro has been tweaked/re-written. Paul Dunn used some suggestions from Matt Kelliher and implemented them. Check it out: DunnSept’s Blog

Paul Dunn has been in IT in Higher Education for 18 years. Paul has a Master’s degree in Information Resource Management and along with his job duties as sys admin, teaches database design and modeling, networking and telecommunications.

eesh.. sounds horrible. I’m really just a lacky at a medium-sized university. Jack of all trades, master of none I do everything here, from dba to programmer, sysadmin, web monkey, to photographer & videographer.

Other important resources on vscsiStats are these two blogposts from Duncan:

• vscsiStats output in esxtop format?
• vscsiStats

See full post at: Converting vscsiStats data into Excel charts

There was a good discussion on twitter on memory overcommit and the value of memory overcommit and whether you should or should not use it in production. What struck me in this was that on a subject like this, there is so much misunderstanding although there is a lot of documentation available that can explain the subtle difference between good and bad overcommit of memory.

Memory overcommit, the basics.

In short: When you assign more RAM to your VMs than available in your host.

Good memory overcommit: When you assign more RAM to your VMs than available in your host BUT never cross the line where the amount of RAM that is USED by your VMs is more than available in your host.

Bad memory overcommit: When you assign more RAM to your VMs than available in your host AND cross the line where the amount of RAM that is USED by your VMs is more than available in your host.

A simple example:
Host has 48GB of RAM and just for the sake of argument we’ll pretend the hypervisor doesn’t use any RAM and we don’t have memory overhead per VM. I now start loading it with VMs that have 4GB RAM assigned. Without any memory overcommit I can load this host with 12 VMs of each 4GB.

Now let’s say, these VMs normally use only 2.5GB of RAM but sometimes they peak to 4GB. With memory overcommit I could now load the host with 19 VMs of 4GB RAM assigning a total of 76GB RAM and demanding 19 x 2.5GB = 47.5GB of physical memory. Even to me this is a bit on the edge, so I’d reserve some RAM for spikes and would go back to 17 VMs, which would leave me with 17 x 2.5GB = 42.5GB of actively used physical RAM, 17 x 4GB = 68GB of RAM assigned and therefore 68GB-48GB = 20GB of overcommitted RAM. So, 20GB of RAM I didn’t have to pay for. This is a good use of memory overcommit.

Bad use of memory overcommit is when in the previous example I would place more VMs on this host, to the point where the use of physical RAM is higher than the amount of physical RAM present in the host. ESX will start some memory optimization and reclaim techniques, but in the end it will swap host memory to disk, which is bad. It is essential to carefully monitor your hosts to see if you’re moving from good memory overcommit to bad memory overcommit.

Misconceptions on memory overcommit

- Transparent Page Sharing (TPS) is a performance hit

- Overcommit is always a performance hit

- Real world workloads don’t benefit

- The gain by overcommitment is neglectable

- Overcommitment is dangerous

Transparent Page Sharing (TPS) is a performance hit
The mechanism below memory overcommit is called Transparent Page Sharing (TPS). Using this technique ESX scans if pages in memory for one VM are identical to pages of another VM by using hash values and if they are ESX doesn’t store that second page in physical memory but just places a link to the first page. When the VM wants to write to that page, then the link is removed and a real page in physical memory is created. This technique called Copy-on-Write (CoW) will incur an overhead compared to writing to non-shared pages.

What is transparent page sharing

The scanning of the guests performed by the hypervisor is done at base scan rate specified by the “Mem.ShareScanTime” which specifies the desired time to scan the VMs entire guest memory. By default ESX will scan every 60 minutes, but depending on the amount of current shared pages, ESX can intelligently adjust this scan frequency.

When performance is measured between a system with no page sharing, default page sharing or excessive page sharing (forced by the test), it shows that default page sharing performs 0,2% slower and sometimes 0,2% faster than without page sharing. Page sharing sometimes improves performance because the VM’s host memory footprint is reduced so that it fits the processor cache better.

Transparent Page Sharing influence on performance

Overcommit is always a performance hit

To be short, no it is not. But you probably expect more from me than just this. Let’s be clear about doing bad overcommit which is always a performance hit. As soon as your ESX hosts is out of physical RAM and starts swapping to disk, there is a big performance hit which should be avoided at all costs. Scott Drummonds (VMware) recently wrote about using Solid State Disks in your SAN to be used as swap space for your ESX host (http://vpivot.com/2009/12/24/solid-state-disks-and-host-swapping/), which would make less of an issue but still there is a performance hit.

But with good overcommit the performance hit is very low as you read in the previous section. There is also a fase in between good and bad overcommit and that is when your good overcommitted host is getting low on free physical memory. When this situation occurs, ESX will start looking for unused but claimed memory by the VMs and will try to reclaim memory by using a technique called ballooning.

The balloon driver inside the Guest OS is triggered by ESX to try and claim free pages by demanding memory (inflating) from the Guest OS. The Guest OS will then use page reclaiming algorithms to determine which pages can be assigned to the balloon driver. First the OS will assign pages marked as free and if this is not enough the OS will move less important pages to the OS swap file to free up pages and assign these to the balloon driver. Next the balloon driver tells ESX which pages it has received and since the balloon driver will not actually use them, ESX can safely re-use these pages for other VMs.

As with TPS, there is a performance penalty for this ballooning, but again the penalty is very low. In the graph below you can see that even with reclaiming 3/4 of the VMs memory, the performance penalty is only 3%. Looking at the performance impact of swapping host memory, we see that the throughput loss is about 34% which is a huge difference compared to the 3% for ballooning.

Impact of ballooning on performance

FIRST CONCLUSION: No performance hits !!!

The conclusion to draw from this, is that the performance impact of Transparent Page Sharing and ballooning neglectable and memory overcommit is NOT a performance hit !!!

Let’s move on to the remaining misconceptions on memory overcommit.

Real world workloads don’t benefit.
A common misconception is that TPS and ballooning will not work for the majority of applications. I have done a lot of searching for problems with applications in a virtual environment that could lead back to TPS or ballooning but could only find two situations in which there is a “problem” with TPS or ballooning. One situation is when running a Java application.

Copied from Scott Drummonds website:
“Java provides a special challenge in virtual environments due to the JVM’s introduction of a third level of memory management. The balloon driver draws memory from the virtual machine without impacting throughput because the guest OS efficiently claims pages that its processes are not using. But in the case of Java, the guest OS is unaware of how the JVM is using memory and is forced to select memory pages as arbitrarily and inefficiently as ESX’s swap routine. Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed inside the JVM and efforts by the host or guest to remove pages will both degrade Java applications’ performance. In these environments it is wise to manually set the JVM’s heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.“

The second exception I found is not really a problem but only something to keep in mind when working with the newer type of CPU’s that can do Virtualized MMU. Read this post from Duncan at Yellow-Bricks: “Virtualized MMU and Transparent page sharing” http://www.yellow-bricks.com/2009/03/06/virtualized-mmu-and-tp/. In this post Duncan explains that when using large pages (2MB) in combination with a CPU that uses virtualized MMU “the ESX page sharing technique might shares less memory when large pages are used instead of small pages“ and “When free machine memory is low and before swapping happens, the ESX Server kernel attempts to share identical small pages even if they are parts of large pages. As a result, the candidate large pages on the host machine are broken into small pages. This can degrade performance at a point in time when you don’t want performance degradation. Keep this in mind when deploying VMs with virtualized MMU enabled, make your decision based on these facts! Do performance testing on what effect overcommiting will have on your environment when virtualized MMU is enabled.“

Second Conclusion: Workloads being not compatible with TPS or ballooning:

- Java is the only known application that can have a performance hit when ballooning kicks in.

- When using V-MMU compatible CPU’s in combination with large pages a performance degradation can be expected when ballooning kicks in.

The memory gain by TPS is neglectable

How much memory would you expect to gain from using TPS? Keep in mind that TPS works at page level, not only between VMs with the same Guest OS, but across all Guest OSes. Not just over VMs but also within the same VM. You could theoretically already save memory through TPS with just one VM running on your host.

Looking at real world examples provided me by a number of people running ESX in production, it is clear that TPS saves a lot of memory. See the table below:

Third conclusion: Memory savings by TPS:

- As the table shows, memory savings thanks to TPS can be really huge and the price of memory saved by TPS is easily worth the extra buck for ESX licenses.

Overcommitment is dangerous

In the paragraphs above I think I made clear that there is no reason to NOT use TPS unless for specific applications. The only reason left for some people to not use TPS is that if you just keep on filling your host with VMs, it will soon start swapping (which is the bad form of overcommit) and they’re absolutely right. But may I ask this question: “Don’t you monitor what you are doing?”.

When I have a one liter bottle and I want to fill it with water, while filling it I usually keep an eye on (monitor) how much water I already pored into it and stop just before it’s full. I think this same practice can be used in many places, even in IT infrastructures.

Number of users on your Citrix hosts, I bet you monitor performance and start adding more hosts when certain limits will be reached. Or adding data to your disks, I bet you daily check to see if your disks aren’t near their limit. Creating snapshots of VMs in your virtual infrastructure, check your datastore if these snapshots aren’t filling up your space and please, do tell me you keep a sharp eye on those thin provisioned disk. Now, if you do all that and consider those techniques as save, I can’t see why overcommitment would be dangerous. Just monitor !!!

Fourth conclusion: Danger of overcommitting

- There is no danger, no hidden risks, just as long as you monitor what you are doing.

I would to thank my twitter friends how promptly responded with esxtop screenshots when I called out for help on twitter. Thanks guys.

And a big thank you to Duncan and Scott for providing great posts which helped me put together this post. Thanks.

Sources:

See full post at: Memory overcommit in production? YES YES YES

After reading verbeieren’s blog about vscsiStats, I was very interested to see what this tool could bring me. So I started to play with it and found that I needed some extra hints to get me started. Enough for another blogpost I thought.

With vscsiStats you can monitor the IO through the virtual SCSI controller of a VM (or multiple VMs). vscsiStats works from the console and outputs to the console. The data can be exported as a CSV file so you can import it into Excel or any other tool. The way vscsiStats works is that you first enable statistics collection on one or more VMs and then take samples at severall moments. Never forget to stop the collection when you’re done, because it will cause some performance degradation.

The vscsiStats command can be found in /usr/lib/vmware/bin and is not part of the path environment, which means that you either first change to this directory or refere to the full path everytime you use the command: /usr/lib/vmware/bin/vscsiStats.

Like said in the first part, you first have to start the collection of data. If you’re troubleshooting a single VM, it’s best to collect stats for just one VM. Before you can do that, you will have to find the Worldgroup ID of the VM. Strange thing is that when using the ‘normal’ method for this ( vm-support -x) you will find slightly different WID’s then what you need for vscsiStats. Better is to first run ./vscsiStats -l. This will give you a list of all available virtual machines and their disks. In my example I want to check my home exchange server called w2k3-ex01-64bit:

./vscsiStats -l

Virtual Machine worldGroupID: 1076,
Virtual Machine Display Name: w2k3-ex01-64bit {
Virtual SCSI Disk handleID: 8195
Virtual SCSI Disk handleID: 8196
Virtual SCSI Disk handleID: 8197
}

1076 is the WID I need. Now to start collecting data for just this VM run:

./vscsiStats -s -w 1076

The result will be something like:

vscsiStats: Starting Vscsi stats collection for worldGroup 1076, handleID 8195
Success.
vscsiStats: Starting Vscsi stats collection for worldGroup 1076, handleID 8196
Success.
vscsiStats: Starting Vscsi stats collection for worldGroup 1076, handleID 8197
Success.

The option -s is to start collection, -w determines the WID of the VM you want to monitor. After stats are running for lets say 5 minutes, you can get a sample with:

./vscsiStats -w 1076 -p all -c

Again -w is used for the WID. Then with -p you can select the stats you would like to use. You can choose between: all, ioLength, seekDistance, outstandingIOs, latency, interarrival. When using the stats to put them in a nice histogram or spreadsheet, you can use -c to print the output as csv and ofcourse you could use ‘ > /tmp/outputfile.csv’ to redirect the output to file.

Now after you have collected these values for a few times and you’re done with collecting, don’t forget to stop the collection.

./vscsiStats -x

You will now see the following output:
vscsiStats: Stopping all Vscsi stats collection for worldGroup 1076, handleID 8195
Success.
vscsiStats: Stopping all Vscsi stats collection for worldGroup 1076, handleID 8196
Success.
vscsiStats: Stopping all Vscsi stats collection for worldGroup 1076, handleID 8197
Success.

The tricky part is how to make some sensible info from it. I try to use excel to convert the data into a nice histogram and would love to have a macro for this. But untill I have written one, I have to do it by hand. Keep in mind that these are histograms when looking at the following piece of data:

44, 4095
1405, 4096
10, 8191
837, 8192

These numbers tell you that for 44 times, there was IO command with a blocklength of 4095, 1405 times of 4096, etc, etc. Be sure to interpret them correctly. Having too many blocks of 4095 and 8191 could point to a misaligned partition or disk in this case.

Hope this guide helped you working with the data. Should anyone have a nice excel macro to convert the data into histograms, please mail me or post it in the comments.

See full post at: Using vscsiStats (the full how-to)

I frequently read postings on different forums about what the best LUN size is and I thought I’d write a post about it on how I usually do it.

First of all, there is no ideal LUN size that suites all environments. But I think there is a general calculation you can use to come to the LUN size suited for your environment. And second, there is no technical difference between small or large LUNs. For the read/write performance it doesn’t matter if your VMDK is on a 100Gb LUN or 2TB LUN, it’s the total load on the LUN that matters.

The bases of my rule of thumb is that a LUN should hold no more then 30 VMDKs, more VMDKs could impact performance because of disk queuing. Keeping those 30 VMDKs in mind, you should then check what your average disk size is and please look at it by using your common sense. With that I mean, if you have 100 VMs that use around 15Gb each and 2 VMs with 1,5Tb, please leave those 2 out of your calculation. These exceptions you can handle later on.

Ok, you have determined the average disk size, let’s say 12Gb. Now multiply this by that magic 30 and you see that you would need 360Gb per LUN to accommodate 30 VMDKs. But that’s not all; we also need some spare room for VM swap space and for snapshots. Normally the VMs I use have 1Gb or 2Gb RAM assigned, which would give a 1-2Gb swap space for the VM. Because most VMs have more than one VMDK, I think its safe to state that 30Gb swap space per LUN is sufficient.

Most tricky part is the spare room you should reserve for snapshots. I try to keep snapshots active as short as possible. A week old snapshot is really long in my opinion. Of course, there can be reasons that you need it, but normal operations would not require running a snapshot for such a long time. So how big should the room for snapshot be? Let’s put it at 15%.

So to summarize and build the formula:

30 x (your average disk size) + 30Gb VM swap + 15% of (30 x your average disk size) = calculated LUN size.

And to put the cherry on top, take your calculated LUN size and round it up to the next “handy” number. For example, 444Gb I would round up to 500Gb, 689Gb I would round up to 750Gb and so.

With this formula I think you can make a basic calculation and get an idea of what COULD suite your environment. Its no hard rule, it something you have to feel comfortable with and maybe after some time you would tweak this to your own experiences.

Would love to hear your thoughts and comments on this!!!

See full post at: Storage: How to size your LUNs?

AP09 – Fast and easy disk workload characterization on ESX

(Richard McDougall)

 

When moving an I/O insensitive workload onto ESX, you’re always asking yourself: “Will it hold? Will it perform? Is there something I can tweak to make it perform better?”. In this session Richard showed some things to look at.

 

The key questions before moving to virtual are:

-         What is the I/O blocksize?

-         What is the spatial locality?

-         What is the I/O interarrival period?

-         What is the Queue Depth needed?

-         How much latency is allowed?

-         What are the read / write rations?

 

When knowing these figures for you physical environment, you can start testing in your virtual environment to see if your configuration can meet the demand. Richard is an SUN and Oracle guy, so a lot of examples are Oracle based, but that is no problem for understanding where its all about.

 

The tools he uses frequently to test performance are Filebench OLTP, OSDL dbt-2 and Windows large file copies.  In the virtual environment you can monitor what your vSCSI adapter of your VM is doing by looking at /usr/lib/vmware/bin/vscsistats -h. It can show you metrics like I/O size, R/W ratios and Outstanding I/Os. Very important data to have when discussing performance with your SAN admin.

 

There are some limitations in using vscsistats, it only shows histograms per virtual disk and it doesn’t give a complete I/O picture because you will have other VMs interfering and traffic from the ESX host itself. But, nevertheless still a great tool to start with when diving into performance characteristics. Be sure to check out these sites, on which more information can be found about performance:

http://www.virtualscoop.org and the VROOM blog on the VMware site.

 

Great session, although Richard sometimes went pretty fast and left no time for me to take notes. Well, there is always the online pdf in a couple of weeks.

See full post at: AP09 – Fast and easy disk workload characterization on ESX