Normaly when installing a vSphere environment for new customers, I wait with entering the correct licenses until the environment is ready and can be moved to production. Until today I never ran into any issues with that since most of our customers buy Enterprise or Enterprise Plus edition. Today I discovered that after entering a vSphere Standard license, you might run into an issue with DRS where the affinity rules stay in effect.

As best practice we usually put the vCenter VM on the first host in the cluster using a VM to Host affinity rule in DRS. A few days after entering the vSphere Standard license which doesn’t allow DRS, I tried to move the vCenter VM to a different host, but this wasn’t allowed because a VM to Host affinity rule was still active. When checking the cluster settings, DRS was still disabled. When I enabled DRS, without clicking OK, I could see there still was a host-affinity rule defined in the settings.

I decided to remove the rule and the host-group and VM-group.

As expected, when clicking OK I received an error telling me that DRS was not a valid option for this license.

I did hope however that the deletion of the rules would still have been applied. Unfortunately not. When opening the cluster settings, enable DRS again, I could still see the rule and the groups were present. Only way to get rid of this was to enter a NFR license key for the Enterprise (or Plus) edition, enable DRS, remove the VM- and host-groups, remove the rule, click OK to see the changes applied. Go back to Cluster settings, make sure they’re all gone. Disable DRS and remove the NFR license.

Sounds like a little bug to me and I therefore create a support call with VMware for this.

PS: Frank Denneman hinted me this is probably only for “MUST RUN” DRS rules.

See full post at: vCenter DRS rules bug when downgrading license

Today, when performing a VMware Health Check, I came across a strange issue with a customer of mine. Well, issue is maybe a big word, but I discovered the VMware Tools version some of the VMs had running, were newer than the VMware Tools version from a newly installed VM. To make sure I double checked by removing the VMware Tools and reinstalling them, again the VMware Tools version was less than the tools version of the suspicious VMs.

The VMs with the very high VMware Tools version had version 8448 running. When checking the VMware Tools about page, it showed:  8.8.0 build 471268. The other ‘normal’ VMs had VMware Tools version 8300. The Windows VMware Tools about page showed 8.3.12 build 493255. The vSphere environment I was running this on had vCenter 4.1 and ESXi 4.1 update 2 build 502767.

After some help on twitter, it turned out these VMs had been created from a VM that was originally created in VMware Workstation !!! I removed the VMware Tools from the Windows VM and reinstalled them, since I’m not sure if there is a difference between the VMware Tools installed when the VM is on an ESXi host and the VMware Tools in VMware Workstation or Fusion. Upgrading them to the ESXi version didn’t work because, of course, the existing build number is higher than the version I would like to install.

I don’t have any real answers for you yet, but I’m trying to find out more whether there are potential issues when running the VMware Workstation VMware Tools in your ESXi VM. My first guess would be that it wouldn’t matter, otherwise VMware had probably done more checking or given other alarms when trying to upgrade the tools.

If you have all the answers, please tell me in the comment section.

On twitter I was pointed to this list: VMware Tools versions and Packages.

A big thank you to  Ewan Leith and Alexander for their twitter help.

See full post at: VMware Tools version number higher than expected

Ever had it happen in your environment? Committing a snapshot on a VM fails and you can’t get your VM to boot again because there are disks missing. This could happen because when initiating Delete or DeleteAll snapshots, the snapshot details are deleted from Snapshot Manager, then the snapshot files are consolidated and merged to another snapshot file or to the virtual machine parent disk. If the consolidation fails, there were no snapshots shown in the Snapshot Manager, but the snapshot files were still being used on the datastore. This can cause the datastore to run out of space.

Unfortunately today this happened for me too, but it has been the first time this happened in a vSphere 5 environment. And therefore it was also the first time I noticed a new feature in VMware vSphere 5 / vCenter 5: “consolidate snapshots”. When a VM has snaphots present on the datastore but they don’t seem to be linked to the VM anymore, vSphere 5 will detect this and report an error on the summary tab: “Virtual machine disks consolidation is needed”.

I quickly found there is a new option now under the snapshot menu. Right click the VM, select Snapshot and select the new option “Consolidate”. Now vSphere will start a new consolidation.

A great new option that probably can save quite some work for vSphere admins. See these KB’s for more info:

• Consolidating snapshots in vSphere 5
• Consolidating snapshots

Unfortunately in my case, this didn’t work and I immediately got a general I/O error, even though the datastore had enough space left. I eventually had to revert to my last Veeam backup because that was more recent that going back in time to the day the snapshot was made.

See full post at: New vSphere5 feature: Virtual Machine disks consolidation is needed

In my previous post on VMware vSphere5 Auto deploy I explained how to setup an auto deploy environment. This is great for initial setup and to deploy your hosts, but what when an update is released by VMware? The easiest thing is to just delete the deploy rules and recreate them which is ok for a small environment but not in a large environment. In this post I will show you how to adjust your existing rules.

Finding the new builds

There are two ways of getting the new builds for your ESXi image. The well known way is to go to the VMware download pages, logon, access your download for vSphere and download the zip image. However, there is an easier way. Logon to your vCenter where you have your Software Depot. Open the VMware PowerShell command line, connect to vCenter and add the VMware HostUpdate URL to the EsxSoftwareDepot. Then retrieve a list of the available images:

Connect-viserver –server w2k8-vc5.vanzanten.local
Add-EsxSoftwareDepot -DepotUrl https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Get-EsxImageProfile

The result should look like this:

Name                           Vendor          Last Modified   Acceptance Level
----                           ------          -------------   ----------------
ESXi-5.0.0-20110904001-notools VMware, Inc.    8/26/2011 1:... PartnerSupported
ESXi-5.0.0-469512-no-tools     VMware, Inc.    8/19/2011 1:... PartnerSupported
ESXi-5.0.0-20110904001-stan... VMware, Inc.    8/26/2011 1:... PartnerSupported
ESXi-5.0.0-469512-standard     VMware, Inc.    8/19/2011 1:... PartnerSupported

To see if these are really newer than the existing images list the current deployrules and the images that are being deployed.

Get-DeployRule
Name        : CL01-GA
PatternList : {ipv4=192.168.0.140-192.168.0.141}
ItemList    : {ESXi-5.0.0-469512-standard, CL01, HostProfile001}

This will list all deploy rules created, not necisserally active rules. Looking at the result I see that the “ESXi-5.0.0-20110904001-standard” image is newer and I would like to deploy this to my hosts. To see what has changed between the existing image and the new image I want to deploy, I do a comparison:

Compare-EsxImageProfile ESXi-5.0.0-469512-standard ESXi-5.0.0-20110904001-standard

This shows that the modules VMware_locker_tools-light_5.0.0-0.3.474610 and VMware_bootbank_esx-base_5.0.0-0.3.474610 have changed between these two builds.

Adding the new build

I’ve determined that “ESXi-5.0.0-20110904001-standard” is the new build and I want to update my existing rule named “CL01-Beta”. This can be done through the following command:

Copy-DeployRule -DeployRule CL01-Beta -ReplaceItem "ESXi-5.0.0-20110904001-standard"

After running this command, you should see the start of the image download:

Downloading esx-base 5.0.0-0.3.474610
Download finished, uploading to AutoDeploy...
Upload finished.
Downloading tools-light 5.0.0-0.3.474610
Download finished, uploading to AutoDeploy...
Upload finished.

Even though the above looked good, I did get an error immediately after:

WARNING: The supplied Image Profile does not contain the "vmware-fdm" software package, which is required for the vSphere HA feature.  If this Image Profile is to be used with hosts in a vSphere HA cluster, you should add the vmware-fdm package to the image profile.  The vmware-fdm package can be retrieved from the software depot published by this vCenter Server at the following URL:

http://192.168.0.41:80/vSphere-HA-depot

You can use the Add-ESXSoftwarePackage cmdlet to add the package to the image profile and then update any hosts or rules that were using the older version of the profile.

Panic !!! Well, luckily the VMware documentation offered a solution to this, although I still don’t know why this happens, but this is the way to solve it:

Add-EsxSoftwareDepot http://192.168.0.41/vSphere-HA-depot
Add-EsxSoftwarePackage -ImageProfile "ESXi-5.0.0-20110904001-standard" -SoftwarePackage "vmware-fdm"

The module is added to the image profile now. Check what the deploy ruleset looks like now. First the deployrules available and next the active deployrules:

Get-deployRule
Get-deployRuleSet

Should your rules still contain the old image, probably because you ran into the same error as I did, you might want to run the copy-command again:

Copy-DeployRule -DeployRule CL01-Beta -ReplaceItem "ESXi-5.0.0-20110904001-standard"

One last check that needs to be performed is to see if your hosts are still compliant. On initial boot the Auto Deploy server parses the host attributes against the rules engine to determine the Image Profile, Host Profile, and vCenter Cluster/Folder and saves this info. On subsequent reboots, it uses this saved info. When copying the rule and updating the Image Profile, the hosts of that rule are no longer compliant (compliance here is dealing with Auto Deploy and not the Host Profile compliance shown in vCenter) because the image listed in the rule no longer matches the image the Auto Deploy server has saved for the host. To fix this you need to run a Test/Repair for all hosts.

To check for compliance:

Get-VMHost * | Test-DeployRuleSetCompliance 

VMHost               ItemList
------               --------
autoesx02.vanzant... {}
autoesx03.vanzant... {}
autoesx01.vanzant... {Non-compliant: ESXi-5.0.0-441354-standard != E...

The results above show that host autoesx01 is not compliant. To fix this:

Get-VMHost * | Test-DeployRuleSetCompliance | Repair-DeployRuleSetCompliance

And of course check the result again by running the check again. If things look OK now, it is time to reboot the ESXi host to test the new image.

See full post at: Updating your ESXi host using VMware vSphere 5 Auto deploy

When preparing your Virtual Infrastructure environment for vSphere 5 and vCenter 5 you also need to think of all the vCenter applications that are dependent on vCenter. One of  them is vCenter Heartbeat. To be able to run vCenter Heartbeat on vCenter 5 you will need to upgrade vCenter Heartbeat to version 6.4. Normally upgrading vCenter Heartbeat isn’t a big deal but there has been a change from 6.3 Update 1 to 6.4. Where you could normally use a configuration of “identical nodes”, you will now have to switch to “non-identical nodes” according to KB 1014435.

From the KB article: “Important: When performing an upgrade to vCenter Server Heartbeat 6.4 from 6.3 Update 1 with Identical nodes or from any previous version, you must first migrate from Identical nodes to Non-Identical nodes before performing the upgrade.”

Switching “identical” to “non-identical”

To switch your configuration from “identical” to “non-identical” nodes you can use KB 2002112 “Migrating vCenter Server Heartbeat 6.3 Update 1 and later from identical nodes to non-identical nodes”, but it needs some additions.

First there is the following note in the KB: “Note: This procedure reconfigures vCenter Server Heartbeat 6.3 Update 1 and later installed in a LAN environment where vCenter Server and SQL Server are protected but not installed on the same machine (remote SQL Server).”  However, this procedure DOES work for locally installed SQL Servers as well. In my testlab it even worked with SQL Express, even thought SQL Express is NOT SUPPORTED.

Second difference between real world and KB is that you will need to reconfigure your vCenter networking, but there is no mention of that in the KB. Below is the procedure I followed IN A LAB ENVIRONMENT when testing the upgrade for a customer of mine.

Test environment:

• Primary server: vCent01
• Public IP: 192.168.233.10
• Heartbeat interfaces: 10.0.0.1 and 10.0.5.1
• Secondary server: vCent01 (because it’s a clone of vCent01)
• Public IP: 192.168.233.10 (because it’s a clone of vCent01)
• Heartbeat interfaces: 10.0.0.2 and 10.0.5.2

When upgrading to heartbeat 6.4 and switching to “non-identical” nodes, you’ll need new server names and an extra public (passive) IP address for your primary and secondary vCenter.

• Primary server: vCent00 will be the new name
• Passive Public IP: 192.168.233.20
• Secondary server: vCent02 will be the new name
• Passive Public IP: 192.168.233.30

Following the first steps of KB 2002112, shutdown vCenter Heatbeat on both servers leaving the applications running. Do this through the VMware vCenter Heartbeat Management console.

On the Secondary Server part, follow steps 1 through 3.

Re-enter the “Service Name”. In my case this is “VCENT01”. Steps 4 and 5 tell you to assign the Principal Public IP and the passive IP in step 7. When entering the correct IP addresses, you will run into a problem later on after removing the secondary server from the domain and trying to re-add it. The Windows network stack is not update with the configuration change in IP you make here. To solve this set the IP on the network interface to 192.168.233.30. Then use this as the passive IP address and keep 192.168.233.10 as the public IP address.

Proceed with steps 8 through 11.  Step 12 explains how to rename the secondary server, in my case the new name will be vcent02. Use the last steps to rejoin the server to the domain. Then proceed to the primary server and also change its IP address at Windows level like we did with the secondary server and later on change the server name to vCent00. The primary server doesn’t need to be removed from the domain.

Upgrade VMware vCenter Heartbeat 6.3 update 1 to 6.4

After you completed these steps, make sure your VMware vCenter Heartbeat is running fine again. You might need to wait some time for the file sync to complete. After everything is running smooth again, the next step is the upgrade itself. This is a fairly straight forward procedure, outlined in KB 1014435 titled: “Upgrading vCenter Server Heartbeat from an earlier version to a later version”.

See full post at: Upgrading VMware vCenter Heartbeat 6.3 update 1 to 6.4

Today Veeam released Backup Academy. It is a FREE resource to get started with backup technologies targeted at IT professionals who are just getting started with data protection. Users that complete the video series can also do a Backup Academy Exam and get a certificate when they pass.

Although created by Veeam, Backup Academy is product neutral and it is already a great source of information with video’s from well know industry experts like David Davis, Eric Siebert, Elias Khnaser, Greg Shields and Rick Vanover. Even Veeam’s competitors are encouraged to add their content to the Backup Academy.

After regestering on the Backup Academy website you will find a number of video’s on real world practices today. You’ll be able to get more info on VMware and Hyper-V backup technologies targeted at System Administrators, Virtualization Administrators and System support staff.

Visit: BackupAcademy.com and follow on twitter @BckpAcademy

See full post at: Backup Academy

In response to my blog post on “vSphere 5 – How to run ESXi stateless with vSphere Auto Deploy“, I received a comment from Rob Ellison, who ran into issues when PXE booting through a Cisco ASA firewall. In a second comment he explained how he solved the issue. Turned out there was an issue with the frame size of the gPXE client. Below is his solution to this issue. Should you have any questions on this specific issue, feel free to contact Rob. Thank you Rob for this solution.

Rob Ellison, Managed Hosting Technical Architect
Twitter: @robellison
Google Plus: Rob Ellison

Confirm the ASA is dropping packets due to frame size:

show asp drop frame tcp-mss-exceeded
TCP data exceeded MSS (tcp-mss-exceeded)

Create a capture:

capture mss-capture type asp-drop tcp-mss-exceeded packet-length 1518

View the output after a failed boot:

show capture mss-capture
5 packets captured

1: 08:51:15.489278 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 516952 120427860>
2: 08:51:15.963420 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 516961 120427860>
3: 08:51:16.952099 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 516979 120427860>
4: 08:51:18.930097 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 517015 120427860>
5: 08:51:22.884140 802.1Q vlan#99 P0 10.x.x.x.6519 > <ip of auto-deploy>.6501: P 888399370:888400818(1448) ack 2273403046 win 8192 <nop,nop,timestamp 517087 120427860>
5 packets shown

To resolve:

access-list vmware-auto-deploy permit tcp any host <auto-deploy-host>
class-map vmware-auto-deploy
match access-list vmware-auto-deploy
exit
tcp-map mss-map
exceed-mss allow
exit
policy-map vmware-auto-deploy
class vmware-auto-deploy
set connection advanced-options mss-map
exit
exit
service-policy vmware-auto-deploy interface <esxi server interface>

See full post at: vSphere 5 Auto Deploy PXE booting through Cisco ASA firewall

A great new feature of vSphere 5 is the possibility to run ESXi stateless. Long, long time ago when ESX 3.0 was hip, we would all install ESX on the local harddisk (or SAN disk). With ESX 3.5, the first ESXi version was released but only few were using it. With 4.x ESXi really got a large install base and more and more people were moving to installing ESXi on USB or SD card. Now with vSphere 5 and ESXi as the only hypervisor (no more ESX), we don’t need to install ESXi at all.

When running ESXi stateless, the host will PXE boot and load an ESXi image into memory. There is no more need to have local disks, SD card or USB on your host. Another advantage is that you can now switch the ESXi version the host is running by just rebooting or refresh the host configuration with a reboot. Adding new ESXi hosts to your cluster has become a task of just a few minutes. Let’s have a look at how to configure auto deploy. This blog post will take you through the steps of setting up the required infrastructure and prepare an image and deploy it to a host.

Important: When running stateless, your ESXi host cannot boot without all vCenter Auto Deploy components being active. Changes are big that your vCenter Auto Deploy is running in a virtual machine. Should your whole environment power down, you will have a problem getting everything running again. Personally I’m thinking of having an extra USB install of ESXi 5 at hand at all times. Just in case. The bigger the environment the smaller the chance this will happen. But especially for some home-lab environments this might be something to think about.

Infrastructure

When using Auto Deploy a host will use DHCP to get an IP address and learn if there is a PXE boot server. The PXE boot image is downloaded and the pre-boot environment is ran on the host. It will now contact the Auto Deploy server which will now check the rule engine. Based on the rules in the rule engine an image is now offered to the host and a host profile is applied to the host.

An image is based on a VMware ESXi deployment image combined with optional extra VIBs (VMware Image Bundles). You can download a “ready-to-deploy” image of ESXi from the VMware download site, just like you would download your normal ESXi install files. For example the latest ESXi version I could download in the beta, was build 441354. The normal ESXi install to download would be: VMware-VMvisor-Installer-5.0.0-441354.x86_64.iso, but on the download page there also is a: VMware-ESXi-5.0.0-441354-depot.zip. This is an ESXi image suited for auto deploy, containing the ESXi VIB. It is also possible to combine VIBs into a new image if needed, for example when you want to add extra drivers or special VIB supplied by the hardware manufacturer.

Another component is the Image Builder PowerCLI. Using PowerCLI commands a rule set is created and together with a host profile connected to a host or number of hosts.

To make all this possible you need to have the following in your environment:

• Hosts that are capable of running ESXi 5
• TFPT boot server
• DHCP / DNS server
• Windows server to run Auto Deploy or vSphere vCenter Appliance
• vSphere PowerCLI

The vCenter Auto Deploy manual will show you how to enable the vSphere vCenter Appliance for Auto Deploy. In this post I will use a Windows server to act as TFTP and Auto Deploy server.

Installing TFTP server

To keep things simple I downloaded the WinAgents TFTP server. It is a 30 day trial. Installation is a simple next, next, finish. Create a directory on your Windows server, for example: E:\tftproot and make this your TFTP root directory.

Set up DHCP

To be able to PXE boot, you need to set some options in your DHCP scope. Open the scope on your DHCP server and add the following options:

066 – Boot server host name:   <ip of your TFTP / PXE boot server>
067 – Boot file name: undionly.kpxe.vmw-hardwired

Set up vSphere Auto Deploy

On the vCenter installation media you’ll find the Auto Deploy software. It can be installed on your vCenter server but you can also run it on a separate Windows server and have it point to your vCenter server. I kept everything, including the TFTP server, on my vCenter host. Installation again is a simple next, next, finish in which you only need to supply the IP of the vCenter server and the correct credentials. After the installation has finished you can install a plug-in in your vSphere Infrastructure Client. You can see the Auto Deploy icon on the home tab.

Install TFTP boot image

The last step is now to add a boot image to your TFTP server root directory. In the vSphere client click the Auto Deploy plug-in and see the following page:

Now choose to download the TFTP Boot Zip and extract this ZIP file into your TFTP root directory. With this, the first part is ready. To test if things are working, you can boot an ESXi host or just for test boot a VM. Make sure that the host or VM is using PXE boot. You should see that it is assigned an IP address and it starts loading a TFTP image.

Next you’ll see that although a TFTP image was loaded, there was no ESXi image associated with this host. This screen also displays a number of Machine attributes we could use later on when tying a rule set to a host.

Installing vSphere PowerCLI

Well this should be a piece of cake since I expect many of you to have done this many times before. Download the vSphere 5 PowerCLI and install it on the server on which you will be working with your images. To test if your VMware PowerCLI is working, start the VMware vSphere PowerCLI command prompt and run: Get-DeployCommand. This should give you a list of all the commands you’ll need to work with Auto Deploy. After this you’re done and all requirements for vSphere Auto Deploy have been installed.

Preparing your first image

When installing vSphere Auto Deploy, you’re asked for a directory for your SoftwareDepot. This might be a bit confusing since it actually is never used by YOU, only by the Auto Deploy software for internal ‘housekeeping’. My advice is to also create a directory called ‘VIB-downloads’ in which you store the VIBs and images you want to deploy.

First we’ll deploy the basic VMware ESXi 5.0 image to a new host without any further configuration. We’ll attach the image to the host based on mac-address and want the host to show up in your vCenter in a folder named ‘Staging’. Since it has no configuration, I don’t want it to show up in a cluster yet.

All the info we need:

• IP address of the host and the DNS hostname
• Folder name: ‘Staging’
• Mac address: 001a92b8da77
• Image name: VMware-ESXi-5.0.0-441354-depot.zip (as downloaded from VMware site)
• Image name: ESXi-5.0.0-441354-standard (after being added to the depot)

Important: Since we’ll be using DHCP to assign a ‘fixed’ IP and hostname to the host, you need to create a DHCP reservation for the mac-address. In your DHCP scope create the reservation and use the proper hostname. In DNS create an A-record for this hostname and IP address and make sure you give it a PTR / reverse lookup record too.

Adding a new image to your depot

This is what confused me during my first steps with vSphere Auto Deploy. I was doing all sorts of things trying to add the VMware-ESXi-5.0.0-441354-depot.zip to the software depot somehow. I copied the whole zip into it, but that didn’t work, unzipping and copying didn’t work either. Until I learned that you don’t do anything with the software depot directory. Place the downloaded image ‘VIB-downloads’ and then run the command:

Add-EsxSoftwareDepot “E:\VIB-downloads\VMware-ESXi-5.0.0-441354-depot.zip”

You should now see how the image is inserted into the software depot. After this is finished, you can run the command Get-EsxImageProfile, to see what images are present in your depot. In my example you can see I now added two versions of ESXi beta. Each image contains two flavors, a no-tools and a standard version. I usually use the standard version to deploy.

Get-EsxImageProfile

Deploying your first host

We now have an image ready to deploy, when a host reboots he’ll pick up the TFTP image and will ask the vSphere Auto Deploy server for an image. Next step is to create the rule to match a host to an image. When creating rules, you should know there are two rule sets, a ‘working-set’ and an ‘active-set’. The ‘working-set’ is like a depot of rules, the ‘active-set’ are the rules that are really available to hosts. It is a bit confusing sometimes, especially since the commands to attach / create rules are the same. I would have preferred a different naming or just use an extra attribute like “active” and “in-active”. Think that would have made it easier to understand. Also, I was having or actually still have problems editing rules. I just can’t seem to get that work so instead of editing them I delete and re-create them. When I get it to work I’ll update this blog post.

As I was saying, we’re now going to create a rule to connect the image to the host. This is where the ‘New-DeployRule’ command comes in.

New-DeployRule –Name “PreStaging” –Item “ESXi-5.0.0-441354-standard”, “Staging” –Pattern “mac=00:1a:92:b8:da:77”

The new rule that has just been created in called “PreStaging”, it will make sure that the image called “ESXi-5.0.0-441354-standard” ( Get-EsxImageProfile ) will be deployed to a host with the mac address of 00:1a:92:b8:da:77 and will be placed in the “Staging” folder in vCenter.

With the command Get-DeployRule you can see the rule you just created. This is a rule in the ‘working set’. To make the rule part of the ‘active set’ you need to add it like this:

Add-DeployRule -DeployRule “PreStaging”

To check the rules in the ‘active set’ you run the Get-DeployRuleSet command. You’re all set now to boot your host and see how it installs and then shows up in your vCenter.

Using a host profile

After your host has shown in vCenter you’ll see that it is not configured. If you would configure it now and reboot it, all changes will be lost. The way to save your configuration settings is to create a host profile. A great feature which I found particularly useful when trouble shooting host profiles is “Enable / disable Profile Configuration”. This gives you the option to only partly apply a host profile.

You’ll notice that when working with the host profiles and get more advanced configurations, you’re going to run into problems where settings can’t be applied the way you want it and errors without great detail will be thrown at you. For now I’ll show you how to work with a simple host profile.

Configure the new host like normally. Create the vSwitches, attach the datastores, make sure the NTP settings are correct, etc, etc. Since you’re running diskless hosts, it is important to correctly setup syslogging and the core dump location.

For setting up syslogging, I would like to refer you to this post from Jason Boch: “Configure a vCenter 5.0 integrated Syslog server” and use his post to configure syslogging for your ESXi host.

For redirecting the core dump, it is advised to install the vCenter Coredump tool. Just like the syslog tool, the Coredump utility can also be found in the vCenter tools directory. Again install it with a simple next, next, finish. Your server can now also receive coredumps in case an ESXi host would encounter a purple screen of death (PSOD). After configuring the vCenter Coredump utility, configure your ESXi host to use the coredump server. To do this, go into the configuration screen of your host, go to the security profile and enable SSH. Now logon to the ESXi console using your favorite SSH client and run the following commands:

esxcli system coredump network set --interface-name vmk0 --server-ipv4 192.168.0.40 --server-port 6500
esxcli system coredump network set --enable true
esxcli system coredump network get

The last line will show you if the new setting has been enabled. Now logout of your ESXi host and switch back to your vSphere Client.

Important: Don’t forget to disable SSH again, because we’ll be generating a profile and wouldn’t want SSH to be enabled on all new hosts running from this profile.

Now go to the “Host and Clusters” view in your vSphere client and select the host you have just prepared. Right click the host and select “Create Profile from host”. Give the profile a name, for example ‘Profile-Cluster01”. Now attach the profile to this host using the Host Profiles section in the vSphere client and have the profile checked to be compliant.

Auto deploy the host with the host profile

The first rule we created made sure that the host with certain mac address would be connected to the standard image and would be put in the “Staging” folder.

New-DeployRule –Name “PreStaging” –Item “ESXi-5.0.0-441354-standard”, “Staging” –Pattern “mac=00:1a:92:b8:da:77”

Now we are going to create a new rule. The new role will move the host into the production cluster called “CL01” and the host profile “Profile-Cluster01” will be used. Because we’re pretty sure this works without testing (yeah right), I’m not going to do this based on mac address, but on IP address range. You should use the IP range that you use in the DHCP scope for the ESXi hosts and create a reservation in the DHCP scope for each host and also create a DNS record. The new rule would be created like this:

New-DeployRule –Name “Prod-CL01” –Item “ESXi-5.0.0-441354-standard”, “CL01”, “Profile-Cluster01” –Pattern “ipv4=192.168.0.100-192.168.0.110”

In the above rule, ‘Prod-CL01’ is the name of the rule, ‘CL01’ is the name of the cluster, ‘Profile-Cluster01’ is the name of the host profile and ipv4 is the DHCP range.

In the ‘working-set’ we now have two rules (“PreStaging” and “Prod-CL01”) and in the ‘active-set’ the “PreStaging” rule is active. Using the remove command, we can remove the “PreStaging” rule from the ‘active-set’ and next we add the “Prod-CL01” to the ‘active-set’ and double check what we have done:

Remove-DeployRule -DeployRule “PreStaging”

Add-DeployRule -DeployRule “Prod-CL01”

Get-DeployRuleSet

And we’re done !!! You’ll see that if you now reboot your hosts, they will come back and will be added to the CL01 cluster fully participating as a normal host.

See full post at: vSphere 5 – How to run ESXi stateless with vSphere Auto Deploy

I bet you all heard about the big news yesterday where VMware announced vSphere 5. Despite a lot of applauding for the great new features, there also a lot of noise on twitter about the new licensing for vSphere. VMware has decided to move to vRAM licensing without any maximums on the amount of RAM or the number of CPU cores. This PDF explains it all.

Biggest concern of the twitter community was that licenses would cost much much more for the new vSphere 5 than they did for the “old” vSphere 4. There are many theoretical scenario’s that can be thought of that show the new licensing model as much more expensive and existing customers wouldn’t upgrade to the new vSphere 5. I decided to call a number of my customers and ask them for the specs of their environment and then to a little math for them comparing the old and the new licenses.

Remember that if you are an existing customer with a valid support contract you get a FREE upgrade of your CPU license to vSphere 5. Current vSphere 4 advanced customers, will get vSphere 5 Enterprise license since there is no more advanced edition in vSphere 5.

Below you’ll see the math I’ve done and I was surprised to learn that NONE of them will require more licenses when upgrading to vSphere 5. Converting all their current CPU licenses one-to-one will license their current environment without extra cost. These customer have been running vSphere 4 for a number of years now and slowly switched from 16 GB to 32 GB to 96GB hosts.

I do admit though that vSphere will cost you more when you go above the 32GB per CPU (or 48GB per CPU for Enterprise Plus). But even then there are a number of things to keep in mind:

• New Standard license is $200 more expensive but now includes vMotion and Data Recovery compared to the old Standard license
• Old Advanced customers get a free upgrade to Enterprise which gives them Storage vMotion and DRS
• Don’t just compare physical RAM to vRAM but keep at least an 85% ratio.
• This is the time to take a closer look the difference between VM RAM usage and VM RAM assignment.

Example 1:

4 dual CPU hosts each with 56GB RAM (224GB total) and a total of 8 CPUs. Running 6 Citrix XenApp Servers with 26 GB RAM each and one ‘normal’ VM of 4GB RAM. Total vRAM assignment is 163GB.

Old ESXi 4 licensing with Enterprise license:
4 hosts x 2 CPU = 8 CPU licenses Enterprise Edition = 8 x $ 2875 = $ 23,000

New ESXi 5 licensing with Enterprise licenses:
4 hosts x 2 CPU = 8 CPU licenses Enterprise Edition = 8 x $2875 = $ 23,000. This entitles me to address 8 x 32GB = 256GB RAM.

Example 2:

8 dual CPU hosts each 49 GB and 2 hosts of 56GB ( 544GB total) and a total of 20 CPUs. Running a wide range of VMs with a total vRAM assignment of 442 GB.

Old ESXi 4 licensing with Enterprise license:
10 hosts x 2 CPU = 20 CPU licenses Enterprise Edition = 20 x $ 2875 = $ 57,500

New ESXi 5 licensing with Enterprise licenses:
10 hosts x 2 CPU = 20 CPU licenses Enterprise Edition = 20 x $2875 = $ 57,500. This entitles me to address 20 x 32GB = 640GB RAM.

Example 3:

18 dual CPU hosts varying in RAM sizes of 96GB and 48GB RAM but a total of 1350GB of physical RAM. I couldn’t get the exact figure of assigned RAM.

Old ESXi 4 licensing with Enterprise license:
18 hosts x 2 CPU = 36 CPU licenses Enterprise Edition = 36 x $ 2875 = $ 103,500

New ESXi 5 licensing with Enterprise licenses:
18 hosts x 2 CPU = 36 CPU licenses Enterprise Edition = 36 x $ 2875 = $ 103,500. This entitles me to address 36 x 32GB = 1152 GB RAM.

Since I don’t have the exact number of vRAM for this customer, I’m going to work with %. In my previous examples the amount of vRAM was between 70-80% of the total physical RAM available.  When we now look at the entitlement of 1152 GB RAM we see that this is 85.3% of physical RAM. Even they can just upgrade their current licenses to vSphere 5 without having to pay extra.

Example 4:
44 dual CPU hosts, totaling 3048 GB RAM. This customer too couldn’t give me the exact number of assigned vRAM.

Old ESXi 4 licensing with Enterprise license:
44 hosts x 2 CPU = 88 CPU licenses Enterprise Edition = 88 x $ 2875 = $ 253,000.

New ESXi 5 licensing with Enterprise licenses:
44 hosts x 2 CPU = 88 CPU licenses Enterprise Edition = 88 x $ 2875 = $ 253,000. This entitles me to address 88 x 32GB = 2816 GB RAM. The entitlement on vRAM is 92% of physical available RAM.

Example 5:
14 dual CPU hosts, totaling 788 GB RAM, 182 VMs using 482GB of vRAM.

Old ESXi 4 licensing with Enterprise license:
14 hosts x 2 CPU = 28 CPU licenses Enterprise Edition = 28 x $ 2875 = $ 80,500.

New ESXi 5 licensing with Enterprise licenses:
14 hosts x 2 CPU = 28 CPU licenses Enterprise Edition = 28 x $ 2875 = $ 80,500. This entitles me to address 28 x 32GB = 896 GB RAM. This is over the amount of physical RAM present.

Other posts on this subject:

• VMware is not evil – my opinion on their new licensing model by vTexan.
• vSphere 5.0 Licensing Causes Frenzy by Brian Norris
• A Look at VMware Licensing & Environment Growth by Bob Plankers

See full post at: vSphere 5 licensing with vRAM isn’t that bad at all

When trying to update a number of hosts in the ESXi cluster of a customer I first ran into the issue that scanning the host for updates would result in the following error: “Could not scan esx12 for patches”. Searching through the VMware KB I learned that this is probably due to a corrupted scratch partition on the ESXi host, see: KB1020283. This scratch location is, amongst other things, used to store the staged updates for the ESXi host.

Diving into the ESXi host I noticed there was no scratch partition at all. This is strange in my case, since I was sure that after re-installing the hosts a few months ago, my colleague had set the scratch location of all ESXi hosts manually. When double checking this myself I noticed that the scratch location was empty. See: host – configuration tab – advanced settings – ScratchConfig. After kicking my colleague I set the scratch location manually and to double check if it was set correctly I noticed the setting was empty again. While setting the correct location I only got a “Completed successfully” message so it was very strange the setting was empty again.

Since I couldn’t find anything in the VMware KB about not being able to set this value, I tried setting the value through a PowerShell command, but that wouldn’t stick either. My last resort was a reboot which normally feels a bit like being defeated. Nonetheless, after the reboot, the scratch location was still empty, but changing the value worked and sticked. Another reboot is then needed to activate this new scratch location. After this reboot I was able to rescan the host for patches and remediate the host.

Well, this of course solved only half of the problem, because this left me with 8 hosts in this cluster to change the setting. Think you know what happened next, indeed: “PowerShell to the rescue”

VMware KB 1033696 “Creating a persistent scratch location for ESXi” has a good explanation of how to set the scratch location, including the powershell script to do it. The most important commands:

Setting the scratch location:

Get-VMHost -Name esx12 | Set-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/ESX_VMFS00_SCRATCH/esx12/"

Viewing current scratch location setting:

Get-VMHost -Name * | Get-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation"

Viewing current scratch location used:

Get-VMHost -Name * | Get-VMHostAdvancedConfiguration -Name "ScratchConfig.CurrentScratchLocation"

Watch these last two carefully, one is the scratch location that is configured, the other is the location currently in use! If these are not equal the host is probably still waiting for a reboot. The first command only sets the location for one host at a time.

See full post at: Errors during remediation and scratchconfig issues