Today, when performing a VMware Health Check, I came across a strange issue with a customer of mine. Well, issue is maybe a big word, but I discovered the VMware Tools version some of the VMs had running, were newer than the VMware Tools version from a newly installed VM. To make sure I double checked by removing the VMware Tools and reinstalling them, again the VMware Tools version was less than the tools version of the suspicious VMs.

The VMs with the very high VMware Tools version had version 8448 running. When checking the VMware Tools about page, it showed:  8.8.0 build 471268. The other ‘normal’ VMs had VMware Tools version 8300. The Windows VMware Tools about page showed 8.3.12 build 493255. The vSphere environment I was running this on had vCenter 4.1 and ESXi 4.1 update 2 build 502767.

After some help on twitter, it turned out these VMs had been created from a VM that was originally created in VMware Workstation !!! I removed the VMware Tools from the Windows VM and reinstalled them, since I’m not sure if there is a difference between the VMware Tools installed when the VM is on an ESXi host and the VMware Tools in VMware Workstation or Fusion. Upgrading them to the ESXi version didn’t work because, of course, the existing build number is higher than the version I would like to install.

I don’t have any real answers for you yet, but I’m trying to find out more whether there are potential issues when running the VMware Workstation VMware Tools in your ESXi VM. My first guess would be that it wouldn’t matter, otherwise VMware had probably done more checking or given other alarms when trying to upgrade the tools.

If you have all the answers, please tell me in the comment section.

On twitter I was pointed to this list: VMware Tools versions and Packages.

A big thank you to  Ewan Leith and Alexander for their twitter help.

See full post at: VMware Tools version number higher than expected

Today VMware released their Technical Preview of the VMware View PCoIP client for OSX. We’ve all been waiting for this since the release of VMware View 5 and today I finally got to test it on OSX. Of course I had to install it right away and run a test drive from work to my VMware View 5 lab at home. Even though I have only a small uplink to my ISP, I was able to run a Youtube movie at 480p without issues. Running it at 720p works fine to when windowed, but 720p full screen is a bit too much for the small upload I have and you can see some small hick-ups. But still a reasonable viewing experience.

From the VMware website :

Features In the VMware View Client for Mac OS X

• Support for Intel-based Macs
• PCoIP protocol optimized for VMware View 5
• Full screen support for Mac OS X Lion users
• Copy and Paste plain text between View 4.x virtual machines and Mac
• Copy and Paste text, formatted text, and graphics between View 5.x virtual machines and Mac
• Support optional RSA authentication
• Enhanced certificate checking
• Add up to 4 VMware View server shortcuts in VMware View Client

Installing the client

Installing the View Client with PCoIP is very easy. After downloading the DMG image, open it, and drag the application to the applications folder.

After opening the View Client, you’re presented with the connections screen in which you can add up to 4 View Servers.

Press the big cloud button and add your remote host. When the connection is made, you’ll see a security warning depending on the certificate you implemented on your VMware View Security or Connection Server. In the settings menu you can change the required action for the security level of your connections.

Next you’re presented a logon screen and after your logon completes you’ll see the desktops that are available for you. Do pay attention to the little pull down menu below the desktop, which will let you choose between RDP and PCoIP. Note: In this release ONLY the PCoIP protocol is supported. Connecting over RDP can only be done with the View 4.6 client for OSX.

Once you selected the desktop, you’ll be logged on to your desktop and you’re finished. Happy working!

From the release notes:

Known Issue: Unable to connect to Windows 7 based virtual machines using the RDP protocol. Workaround: Access your Windows 7 desktop with the higher performance PCoIP protocol or use the older VMware View 4.6 client with Microsoft Remote Desktop Connection to access Windows 7 based desktops with RDP.

Items Under Consideration for Future Major Client Releases Include:

• Use of Multiple displays/monitors
• Printing to locally attached printers
• Full screen support for Mac OS X Snow Leopard
• SmartCard authentication
• USB device connectivity

See full post at: VMware View Client with PCoIP for Mac OS X

When preparing your Virtual Infrastructure environment for vSphere 5 and vCenter 5 you also need to think of all the vCenter applications that are dependent on vCenter. One of  them is vCenter Heartbeat. To be able to run vCenter Heartbeat on vCenter 5 you will need to upgrade vCenter Heartbeat to version 6.4. Normally upgrading vCenter Heartbeat isn’t a big deal but there has been a change from 6.3 Update 1 to 6.4. Where you could normally use a configuration of “identical nodes”, you will now have to switch to “non-identical nodes” according to KB 1014435.

From the KB article: “Important: When performing an upgrade to vCenter Server Heartbeat 6.4 from 6.3 Update 1 with Identical nodes or from any previous version, you must first migrate from Identical nodes to Non-Identical nodes before performing the upgrade.”

Switching “identical” to “non-identical”

To switch your configuration from “identical” to “non-identical” nodes you can use KB 2002112 “Migrating vCenter Server Heartbeat 6.3 Update 1 and later from identical nodes to non-identical nodes”, but it needs some additions.

First there is the following note in the KB: “Note: This procedure reconfigures vCenter Server Heartbeat 6.3 Update 1 and later installed in a LAN environment where vCenter Server and SQL Server are protected but not installed on the same machine (remote SQL Server).”  However, this procedure DOES work for locally installed SQL Servers as well. In my testlab it even worked with SQL Express, even thought SQL Express is NOT SUPPORTED.

Second difference between real world and KB is that you will need to reconfigure your vCenter networking, but there is no mention of that in the KB. Below is the procedure I followed IN A LAB ENVIRONMENT when testing the upgrade for a customer of mine.

Test environment:

• Primary server: vCent01
• Public IP: 192.168.233.10
• Heartbeat interfaces: 10.0.0.1 and 10.0.5.1
• Secondary server: vCent01 (because it’s a clone of vCent01)
• Public IP: 192.168.233.10 (because it’s a clone of vCent01)
• Heartbeat interfaces: 10.0.0.2 and 10.0.5.2

When upgrading to heartbeat 6.4 and switching to “non-identical” nodes, you’ll need new server names and an extra public (passive) IP address for your primary and secondary vCenter.

• Primary server: vCent00 will be the new name
• Passive Public IP: 192.168.233.20
• Secondary server: vCent02 will be the new name
• Passive Public IP: 192.168.233.30

Following the first steps of KB 2002112, shutdown vCenter Heatbeat on both servers leaving the applications running. Do this through the VMware vCenter Heartbeat Management console.

On the Secondary Server part, follow steps 1 through 3.

Re-enter the “Service Name”. In my case this is “VCENT01”. Steps 4 and 5 tell you to assign the Principal Public IP and the passive IP in step 7. When entering the correct IP addresses, you will run into a problem later on after removing the secondary server from the domain and trying to re-add it. The Windows network stack is not update with the configuration change in IP you make here. To solve this set the IP on the network interface to 192.168.233.30. Then use this as the passive IP address and keep 192.168.233.10 as the public IP address.

Proceed with steps 8 through 11.  Step 12 explains how to rename the secondary server, in my case the new name will be vcent02. Use the last steps to rejoin the server to the domain. Then proceed to the primary server and also change its IP address at Windows level like we did with the secondary server and later on change the server name to vCent00. The primary server doesn’t need to be removed from the domain.

Upgrade VMware vCenter Heartbeat 6.3 update 1 to 6.4

After you completed these steps, make sure your VMware vCenter Heartbeat is running fine again. You might need to wait some time for the file sync to complete. After everything is running smooth again, the next step is the upgrade itself. This is a fairly straight forward procedure, outlined in KB 1014435 titled: “Upgrading vCenter Server Heartbeat from an earlier version to a later version”.

See full post at: Upgrading VMware vCenter Heartbeat 6.3 update 1 to 6.4

Today Veeam released Backup Academy. It is a FREE resource to get started with backup technologies targeted at IT professionals who are just getting started with data protection. Users that complete the video series can also do a Backup Academy Exam and get a certificate when they pass.

Although created by Veeam, Backup Academy is product neutral and it is already a great source of information with video’s from well know industry experts like David Davis, Eric Siebert, Elias Khnaser, Greg Shields and Rick Vanover. Even Veeam’s competitors are encouraged to add their content to the Backup Academy.

After regestering on the Backup Academy website you will find a number of video’s on real world practices today. You’ll be able to get more info on VMware and Hyper-V backup technologies targeted at System Administrators, Virtualization Administrators and System support staff.

Visit: BackupAcademy.com and follow on twitter @BckpAcademy

See full post at: Backup Academy

In response to my blog post on “vSphere 5 – How to run ESXi stateless with vSphere Auto Deploy“, I received a comment from Rob Ellison, who ran into issues when PXE booting through a Cisco ASA firewall. In a second comment he explained how he solved the issue. Turned out there was an issue with the frame size of the gPXE client. Below is his solution to this issue. Should you have any questions on this specific issue, feel free to contact Rob. Thank you Rob for this solution.

Rob Ellison, Managed Hosting Technical Architect
Twitter: @robellison
Google Plus: Rob Ellison

Confirm the ASA is dropping packets due to frame size:

show asp drop frame tcp-mss-exceeded
TCP data exceeded MSS (tcp-mss-exceeded)

Create a capture:

capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518

View the output after a failed boot:

show capture mss-capture
5 packets captured

1: 08:51:15.489278 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 516952 120427860>
2: 08:51:15.963420 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 516961 120427860>
3: 08:51:16.952099 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 516979 120427860>
4: 08:51:18.930097 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 517015 120427860>
5: 08:51:22.884140 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 517087 120427860>
5 packets shown

To resolve:

access-list vmware-auto-deploy permit tcp any host <auto-deploy-host>
class-map vmware-auto-deploy
match access-list vmware-auto-deploy
exit
tcp-map mss-map
exceed-mss allow
exit
policy-map vmware-auto-deploy
class vmware-auto-deploy
set connection advanced-options mss-map
exit
exit
service-policy vmware-auto-deploy interface <esxi server interface>

See full post at: vSphere 5 Auto Deploy PXE booting through Cisco ASA firewall

Last few weeks a lot of talk has been about the new VMware licensing for vSphere 5. Many reported how this would work against VMware’s principle of running as many VMs on one host as possible. After the dust had settled, people started checking their own situations and found that things weren’t as bad as they looked in the first place but for some the new licensing policy would still mean a substantial cost impact.

When reading all the comments, people weren’t complaining about the vRAM model, but mostly about the entitlements. A vSphere 5 Enterprise license would give you a 32GB vRAM entitlement per CPU and 48GB vRAM on Enterprise Plus. Many thought this was much too low.

Well, there is some great news. I picked up on a rumor which will make a lot of people happy. Personally, I never thought VMware would change the licensing policy and especially not in such a short time. A big company like VMware would need weeks and maybe months to change their plans, but I stand corrected. VMware used the customer feedback and changed the vRAM entitlements. My compliments!!!

The new policy:

• VMware vSphere 5 Essentials will give a 24GB vRAM entitlement
• VMware vSphere 5 Essentials Plus will give a 32GB vRAM entitlement
• Max vRAM in Essentials / Essentials Plus will be maxed at 192GB vRAM
• VMware vSphere 5 Standard vRAM entitlement has changed to 32GB ( <- my assumption)
• VMware vSphere 5 Enterprise vRAM entitlement will be doubled to 64GB
• VMware vSphere 5 Enterprise Plus vRAM entitlement will be doubled to 96GB

The amount of vRAM that counts against your vRAM license pool will be capped to 96GB per VM !!! In other words, even if you assign 256GB or the new 1TB limit of RAM to a VM, it will only count as 96GB for your license.

See: http://derek858.blogspot.com/2011/07/impending-vmware-vsphere-50-license.html

See full post at: VMware changes vRAM licensing on vSphere 5 after customer feedback on vTAX

I bet you all heard about the big news yesterday where VMware announced vSphere 5. Despite a lot of applauding for the great new features, there also a lot of noise on twitter about the new licensing for vSphere. VMware has decided to move to vRAM licensing without any maximums on the amount of RAM or the number of CPU cores. This PDF explains it all.

Biggest concern of the twitter community was that licenses would cost much much more for the new vSphere 5 than they did for the “old” vSphere 4. There are many theoretical scenario’s that can be thought of that show the new licensing model as much more expensive and existing customers wouldn’t upgrade to the new vSphere 5. I decided to call a number of my customers and ask them for the specs of their environment and then to a little math for them comparing the old and the new licenses.

Remember that if you are an existing customer with a valid support contract you get a FREE upgrade of your CPU license to vSphere 5. Current vSphere 4 advanced customers, will get vSphere 5 Enterprise license since there is no more advanced edition in vSphere 5.

Below you’ll see the math I’ve done and I was surprised to learn that NONE of them will require more licenses when upgrading to vSphere 5. Converting all their current CPU licenses one-to-one will license their current environment without extra cost. These customer have been running vSphere 4 for a number of years now and slowly switched from 16 GB to 32 GB to 96GB hosts.

I do admit though that vSphere will cost you more when you go above the 32GB per CPU (or 48GB per CPU for Enterprise Plus). But even then there are a number of things to keep in mind:

• New Standard license is $200 more expensive but now includes vMotion and Data Recovery compared to the old Standard license
• Old Advanced customers get a free upgrade to Enterprise which gives them Storage vMotion and DRS
• Don’t just compare physical RAM to vRAM but keep at least an 85% ratio.
• This is the time to take a closer look the difference between VM RAM usage and VM RAM assignment.

Example 1:

4 dual CPU hosts each with 56GB RAM (224GB total) and a total of 8 CPUs. Running 6 Citrix XenApp Servers with 26 GB RAM each and one ‘normal’ VM of 4GB RAM. Total vRAM assignment is 163GB.

Old ESXi 4 licensing with Enterprise license:
4 hosts x 2 CPU = 8 CPU licenses Enterprise Edition = 8 x $ 2875 = $ 23,000

New ESXi 5 licensing with Enterprise licenses:
4 hosts x 2 CPU = 8 CPU licenses Enterprise Edition = 8 x $2875 = $ 23,000. This entitles me to address 8 x 32GB = 256GB RAM.

Example 2:

8 dual CPU hosts each 49 GB and 2 hosts of 56GB ( 544GB total) and a total of 20 CPUs. Running a wide range of VMs with a total vRAM assignment of 442 GB.

Old ESXi 4 licensing with Enterprise license:
10 hosts x 2 CPU = 20 CPU licenses Enterprise Edition = 20 x $ 2875 = $ 57,500

New ESXi 5 licensing with Enterprise licenses:
10 hosts x 2 CPU = 20 CPU licenses Enterprise Edition = 20 x $2875 = $ 57,500. This entitles me to address 20 x 32GB = 640GB RAM.

Example 3:

18 dual CPU hosts varying in RAM sizes of 96GB and 48GB RAM but a total of 1350GB of physical RAM. I couldn’t get the exact figure of assigned RAM.

Old ESXi 4 licensing with Enterprise license:
18 hosts x 2 CPU = 36 CPU licenses Enterprise Edition = 36 x $ 2875 = $ 103,500

New ESXi 5 licensing with Enterprise licenses:
18 hosts x 2 CPU = 36 CPU licenses Enterprise Edition = 36 x $ 2875 = $ 103,500. This entitles me to address 36 x 32GB = 1152 GB RAM.

Since I don’t have the exact number of vRAM for this customer, I’m going to work with %. In my previous examples the amount of vRAM was between 70-80% of the total physical RAM available.  When we now look at the entitlement of 1152 GB RAM we see that this is 85.3% of physical RAM. Even they can just upgrade their current licenses to vSphere 5 without having to pay extra.

Example 4:
44 dual CPU hosts, totaling 3048 GB RAM. This customer too couldn’t give me the exact number of assigned vRAM.

Old ESXi 4 licensing with Enterprise license:
44 hosts x 2 CPU = 88 CPU licenses Enterprise Edition = 88 x $ 2875 = $ 253,000.

New ESXi 5 licensing with Enterprise licenses:
44 hosts x 2 CPU = 88 CPU licenses Enterprise Edition = 88 x $ 2875 = $ 253,000. This entitles me to address 88 x 32GB = 2816 GB RAM. The entitlement on vRAM is 92% of physical available RAM.

Example 5:
14 dual CPU hosts, totaling 788 GB RAM, 182 VMs using 482GB of vRAM.

Old ESXi 4 licensing with Enterprise license:
14 hosts x 2 CPU = 28 CPU licenses Enterprise Edition = 28 x $ 2875 = $ 80,500.

New ESXi 5 licensing with Enterprise licenses:
14 hosts x 2 CPU = 28 CPU licenses Enterprise Edition = 28 x $ 2875 = $ 80,500. This entitles me to address 28 x 32GB = 896 GB RAM. This is over the amount of physical RAM present.

Other posts on this subject:

• VMware is not evil – my opinion on their new licensing model by vTexan.
• vSphere 5.0 Licensing Causes Frenzy by Brian Norris
• A Look at VMware Licensing & Environment Growth by Bob Plankers

See full post at: vSphere 5 licensing with vRAM isn’t that bad at all

Today VMware released the long awaited VMware vSphere client for the iPad. Since VMworld 2010 the rumors where that the vSphere client for iPad would be released anytime soon, but unfortunately we had to wait until today before we could play with it. The VMware vSphere client for the iPad is FREE and great for some very basic admin tasks, but to be honest it’s no rocketscience. 

The vSphere client for iPad is not just an app that let’s you connect to your vCenter, you connect to the new and rebuild vCMA appliance which then makes a connection to vCenter. The vCMA can be downloaded here: http://labs.vmware.com/flings/vcma. VMware also released this instruction video to make setting up the vCMA a lot easier. Once you have setup the vCMA, you can connect the vSphere client for iPad to the vCMA (even through a firewall) and take control of your vSphere environment.

The vSphere client for iPad has a basic set of features that will enable you to do most common maintenance tasks for your vSphere environment. According to VMware, the vSphere client for iPad is not meant to be functionally equivalent with the existing Windows client but still offers the features to perform 80% of the most common vSphere admins’ tasks. These features are:

As a VMware admin, I’m not so sure if 80% of my work only exists out of the above tasks. My first impression of the vSphere client for iPad is that it looks very nice but doesn’t really help you in your daily tasks. If you get a call about a VM performing badly, you can search the VM and get a quick overview of the VMs performance, but it is not the in-depth info that will tell you exactly what is going on. This view will give you CPU, Memory, Disk and Network load, but doesn’t tell you what time frame and even more important nothing about ballooning, swapping or latency.

After you found the VM and checked the performance of it, the only “admin tasks” that are available to you are start, stop, suspend the VM (hard or through VMware tools) or view and restore a virtual machine’s snapshot; that’s it. In the snapshot option, a big “restore” button is presented to you and that is the only option you have and it won’t even ask for confirmation. First of all I think the name “restore” should be changed to be the equivalent of the vSphere Windows client, just to prevent any confusion. Secondly, even with a limited interface, I would have expected to at least be able to commit snapshots and not only have the more dangerous option to revert and go back in time.

Another option the vSphere client for iPad offers is to put a host in maintenance mode and reboot it. I don’t know about you but before I reboot a host or put a host in maintenance mode, I want to be very very sure that a reboot is going to solve my issue. To me rebooting an ESX host to maybe just solve an issue, is like defeat. Rebooting it to apply an update is fine to me, but using the “Windows way”, let’s reboot and see if my problems are gone, is a big no-no. Since the vSphere client for iPad has no diagnostics tools in it, I would never reboot a host based on the info from the iPad.

The maintenance mode option, is another one I would rarely use with such a limited toolset. Not sure about your environment, but with the customers I visit, putting a host in maintenance mode often has at least one VM that doesn’t VMotion away because someone left a CD connected. Since the vSphere client for iPad has no VMotion option or other option to help at this point, you would be stuck there and would have to go back to a Windows vSphere client.

Which leaves us with the diagnostics tools in the vSphere client for iPad. No, no… that’s not the page where you can see any alarms or warnings. Neither is it the page that shows you what services are stopped or running, or the page where you can find the log files of the host. The diagnostics tools is the page where you can ping the host or the VM. It has no option to enter an IP address, it will only ping the host or VM who’s focus you have currently or do a trace route to it.

Think you guessed it by now. The vSphere client for iPad is great for showing off at work. It looks really great and the UI is very intuitive, but ….. it has just way to few features to be really useful and I think because of that you will download the app, play with it and very seldom use it again. Since you installed the vCMA to get the vSphere client for iPad running, you should try pointing your browser to https://<ip of vCMA>/vim and see what features the vCMA offers. ( vCMA 1.0.0.41 needs http, vCMA 1.0.0.42 needs https)

Support for the vSphere client for iPad will only be given through the VMware communities:

Download the vSphere client for iPad: iTunes

VMware is the first vendor to release an application like this and I do think it is great that they dedicate resources to a project like this. For a first release it just works fine, is good looking, but is badly in need of more features. Let’s hope the next release will be followed soon after today.

See full post at: VMware vSphere client for iPad released too early

Just before I was heading home, my colleague asked me to assist with a VM he couldn’t power on anymore. Seemed a customer had tried to commit a snapshot and the job timed out. The customer then tried some other things and suddenly the VM was down. While trying to power on, there was an error telling us that disk1 could not be found.

Checking VM settings in VMX file and VMSD file

In the VM properties and in the vmx file, I found these references:

First harddisk: disk0 [DataStore4] FS05/FS05-000002.vmdk
Second harddisk: disk1 [DataStore4] FS05/FS05_1-000002.vmdk
Third harddisk: disk2 [DataStore4] FS05/FS05_2-000002.vmdk

These are clearly pointing to a snapshot. Strange thing was however that disk1 was 32GB in size instead of 1TB !!! That got me scared a little. In the vmsd file that normally keeps record of the snapshots I found the following references:

snapshot0.disk0.fileName = "FS05.vmdk"
snapshot0.disk0.node = "scsi0:0"
snapshot0.disk1.fileName = "FS05_1-000001.vmdk"
snapshot0.disk1.node = "scsi0:1"
snapshot0.disk2.fileName = "FS05_2-000001.vmdk"
snapshot0.disk2.node = "scsi0:2"

So there was the first contradiction. The vmx file is pointing at the *-000002.vmdk files while the vmsd file is pointing at the *-000001.vmdk files.

Finding an error report

Next step was to look for any mentioning of an error in the log files. Quickly I discovered that the vmware.log file reported that it is looking for the *-000002.vmdk files and that the FS05_1-000002.vmdk was not found:

Unable to find file FS05_1-000002.vmdk
Unable to find file FS05_1-000002.vmdk
DISK: OPEN scsi0:1 'FS05_1-000002.vmdk' persistent R[(null)]
Unable to find file FS05_1-000002.vmdk
DISKLIB-LINK  : "FS05_1-000002.vmdk" : failed to open (The system cannot find the file specified).
DISKLIB-CHAIN : "FS05_1-000002.vmdk" : failed to open (The system cannot find the file specified).
DISKLIB-LIB   : Failed to open 'FS05_1-000002.vmdk' with flags 0xa (The system cannot find the file specified).
Msg_Post: Error
[msg.disk.fileNotFound] VMware ESX Server cannot find the virtual disk "FS05_1-000002.vmdk". Please verify the path is valid and try again.
[msg.disk.noBackEnd] Cannot open the disk 'FS05_1-000002.vmdk' or one of the snapshot disks it depends on.
[msg.disk.configureDiskError] Reason: The system cannot find the file specified.----------------------------------------
Module DiskEarly power on failed.

Contents of the VM directory

I then looked at the vmdk files in the VM directory. First for the first disk, which is FS05:

FS05-flat.vmdk (32G)
FS05.vmdk (397 bytes)
FS05-000002-delta.vmdk (208M)
FS05-000002.vmdk (217 bytes)

The FS05–flat file is the one holding all the data, the FS05.vmdk file is the descriptor file, it’s an ASCII file that you can look into. The same goes for the FS05-000002.vmdk file that is a descriptor of the FS05-000002-delta.vmdk which contains the data of the snapshot.  Strange thing is that there is no FS05-000001.vmdk or delta.vmdk. It is not entirely impossible that the 000001 file is not present, for example when you would have made two snapshots from the same starting point, instead of each snapshot based on the previous one.

The next set of files is for the second disk FS05_1:

FS05_1-flat.vmdk (1.0T)
FS05_1.vmdk (403 bytes)
FS05_1-000001-delta.vmdk (13G)
FS05_1-000001.vmdk (223 bytes)
FS05_1-000002-delta.vmdk (402M)

Now here we have a bigger issue. The FS05_1-flat.vmdk containing 1TB of very important data (it always is when a customer calls), is “covered” by the FS05_1.vmdk. The FS05_000001-delta.vmdk also has its descriptor file FS05_1-000001.vmdk but the FS05_1-000002-delta.vmdk does not have a descriptor file.

The FS05_2 disk luckily looked quite complete:

FS05_2-flat.vmdk (20G)
FS05_2.vmdk (399 bytes)
FS05_2-000001-delta.vmdk (32M )
FS05_2-000001.vmdk (221 bytes)
FS05_2-000002-delta.vmdk (16M )
FS05_2-000002.vmdk (228 bytes)

About CID’s and ParentCIDs’

The descriptor vmdk file normally holds a CID and a ParentCID value, these values link the snapshots and flat vmdk together.  Let’s look at the CID’s in the FS05_2 vmdk’s and how they point to each other. In red you see the file name of the vmdk descriptor file.

Important is the pointer to the parentCID and the parentFileNameHint. In file FS05_2-000001.vmdk you see how the parentCID from the 000002 file matches the CID from the 000001 file. The last file to look at is the FS05_2.vmdk file. Since this is the vmdk file that describes the –flat.vmdk it holds some more info on the disk geometry of the vmdk, but this info is not important to us now. Again we only use the CID lines. You’ll notice the parentCID value is ffffffff. In all vmdk files that are related to the –flat vmdk I found the parentCID value being ffffffff, but I don’t know if this is required. Again we see that the value of parentCID of the 000001 matches the CID of the FS05_2.vmdk. This snapshot chain is OK. The FS05_2 files have been checked and confirmed that the chain is correct. Next step is to reconstruct the chain for the FS05_1 vmdks.

Reconstructing the vmdk

This actually is rather simple. Since we’re missing the FS05_1-000002.vmdk we can just copy the FS05_1-000001.vmdk and edit it to hold the right entries. Be aware that there are some other entries too in the FS05_1-000001.vmdk file and we need them, so a real full copy is needed of the descriptor. Below is the content of the FS05_1-000001.vmdk:

# Disk DescriptorFile
version=1
CID=79cda434
parentCID=7cec9dae
createType="vmfsSparse"
parentFileNameHint="FS05_1.vmdk"
# Extent description
RW 2147483648 VMFSSPARSE "FS05_1-000001-delta.vmdk"
# The Disk Data Base
#DDB

Now copy this to FS05_1-000002.vmdk and edit the lines I marked:

# Disk DescriptorFile
version=1
CID=458dea24 <--- Just pick any number
parentCID=79cda434  <--- make sure it links to the parent
createType="vmfsSparse"
parentFileNameHint="FS05_1-000001-delta.vmdk"  <--- edit this
# Extent description
RW 2147483648 VMFSSPARSE "FS05_1-000002-delta.vmdk"  <--- edit this
# The Disk Data Base
#DDB

The 458dea24 CID can be any number, you just make it up since it has no relation to the VMDK. This is what took a lot of time the first time I was playing with this. I thought I had to retrieve this info from the VMDK somehow, but from VMware Support I learned that this CID can be any number.

The last few steps

After the file has been edited, the chain of the vmdks is correct again. Now the last step is to get this thing up and running again. Remember that the vmsd file was also incorrect, but we can just delete the vmsd file, you did make a backup / copy? Next we remove the VM from the vCenter inventory and the re-add the VM. And as the last step take a new snapshot and then commit all snapshots. After the snapshot has been committed, check the VM settings to see if they are pointing to the correct parent files. If you’re confident that this is correct you’re done. Now power on the VM.

See full post at: Snapshot fails with incorrect CID info in the VDMK file

When I was writing an installation document on vSphere I thought and have some friends on twitter check the doc and comment on it. I received a lot of responses and decided to make a blog post out of it, so here it is.

vSphere default installation settings

When I’m out in the field installing vSphere 4, I want to make sure I install it the same way for each customer. Of course I also want to make sure I use the best practices, but a best practice will work in 90% of the cases and sometimes a best practice doesn’t work at all for your special case. What is important when applying these best practices: Keep thinking!!! But, if you make changes document them in the design and explain why the change is made, so that in a year when you’re with that customer upgrading to vSphere 5 (just kiddin’) you know why you changed the default.

Also remember that my way is not the only way. Especially in the partitioning section there is a lot of discussion on what is best and I think it is a discussion that will never end. Same goes for the HA settings weather to leave VMs powered on or shut them down as response to isolation. Duncan Epping wrote the HA deepdive guide and it is a must read for anyone configuring HA. See: http://www.yellow-bricks.com/vmware-high-availability-deepdiv/. Again, don’t just apply my settings without thinking if they would fit your environment.

Here we go!

How to configure a default vSphere 4 host.

- Use strong root password
- Create at least one local user account on the host

Partitioning of local disk during install:

- Primary

• /  – 5120MB
• Swap  – 1600MB

- Extended Partition:

• /var  – 4096MB
• /home – 2048MB
• /opt  – 2048MB
• /tmp  – 2048MB

- For the local datastore use the name:  <esxhostname>-local.
- http://www.yellow-bricks.com/2009/05/27/partitioning-your-esx-host-part-ii/
- When NOT using the full local disk to install ESX on, create at least a VMFS that a few GB bigger than the partitions you define. Otherwise vCenter will keep on reporting that your datastore is over 75% usage.

Time configuration:

- At least two ntp servers, 0.pool.europe.ntp.org, 1.pool.europe.ntp.org

Network configuration during install:

- Setting IP address, default GW, etc.
- Configure DNS during install
- Configure hostname during install. Difficult to change the linked certificates after name change!

Configuration decisions:

- Do not use host files to solve your HA problems. That was ESX 3.5
- Do not change the COS memory assignment (default = 300 MB) like you used to do in ESX 3.x. With vSphere 4, this value is automatically changed by vSphere based on the amount of RAM the host has.
- Rename the “Service Console” portgroup to “pg-cos” for simpler use in scripts
- Optional: Create a second service console named “pg-cos2″ for heartbeats, to avoid false positives and trigger an HA event
- Set default number of ports on a vSwitch to 120. It’s not the 120 that is important, but 56 can be too small when running many VMs on a host and gives strange VMotion problems.
- Unload VMFS2 drivers which is unfortunately still needed in vSphere.
- In BIOS of host enable NUMA, Intel-VT or AMD-V features and for Nehalam type cpu’s and higher also enable HyperThreading.
- Service Console network and VMotion network can be combined on one vSwitch.

• Portgroup pg-cos will have vmnic0 active, vmnic1 standby
• Portgroup pg-vmotion will have vmnic0 standby, vmnic1 active
• Portgroup pg-cos2 (optional) will have vmnic0 standby, vmnic1 active (haven’t tried it like this, may not let you put two service consoles on same vSwitch)

- Default loadbalancing on vSwitch is Virtual Port ID. This is the recommended setting.
- Create a vSwitch without physical nics for quarantine network. Add a portgroup call pg-quarantine
- Virtual Machine startup/shutdown -> Do not set at host level, only at HA cluster level
- iSCSI targets -> use static discovery as much as possible
- VMFS blocksize op 8MB
- Advanced settings: Only if NFS is being used there will be a few advanced settings (Check NetApp guides)

vSwitch settings:

- Security

• Promiscuous mode: Reject
• Mac address changes: Accept
• Forged Transmit: Accept

- Failover and Load balancing

• Port ID
• Network failure detection: Link status only
• Notify switches: Yes
• Failback: Yes

Cluster level settings:

- HA Configuration settings (Remember to look at Duncan Epping Deepdive HA section)

• Enable host monitoring = active
• Admission Control = Enable
• Admission Control policy = Percentage 25% (adapt to specific situation)
• Virtual Machine startup/shutdown -> Domain Controllers, SQL Server that holds the vCenter DB, vCenter at high level.
• VM monitoring not enabled unless specifically mentioned in design
• Isolation response is Shutdown
• When using iSCSI set the isolation response to “Power Off” and also create a secondary service console running on the same vSwitch as the iSCSI network to detect an iSCSI outage and avoid false positives.
• Add the das.failuredetectiontime at 60000 (60 secs) to avoid possible spanning tree protocol events at the switch level,  this is mitigated again by setting Cisco ports to portfast mode.

- DRS Configuration settings

• Fully automated with default threshold of 3 (Conservative =1 , aggressive = 5)
• Rules: Design should specify what VMs to keep together or apart
• Virtual Machine Options -> Exclude the vCenter server and the connected SQL Server from DRS. Set to “disabled”. Always place vCenter on the first ESX host in the cluster. When first host has to be set to maintenance mode, move to second host. For SQL server, place it on the second host and move to first host when in maintenance mode. Also see: http://www.gabesvirtualworld.com/how-to-quickly-recover-from-disaster/
• Virtual machine options under DRS – Make sure to exclude any VM that is using Microsoft clustering (MSCS or Windows Failover Clusters) in order to maintain support with both Microsoft and VMware.
• Power Management (DPM) -> configure ILO settings in routed network. ESX hosts broadcast the WOL package over VMotion network.

- Set VMware EVC mode according to design. Keep in mind to set EVC before running VMs in the cluster since it can only downgrade CPU’s / Hosts when VMs are powered off.
- Swapfile location remains default with the VM

Thank you for your input:
Mark Vaughn – http://blog.mvaughn.us/
Tom Howarth http://www.planetvm.net
Matt Liebowitz – http://blogs.kraftkennedy.com/
Gert van Gorp

See full post at: VMware vSphere 4 default installation settings