vSphere 5 – How to run ESXi stateless with vSphere Auto Deploy

A great new feature of vSphere 5 is the possibility to run ESXi stateless. Long, long time ago when ESX 3.0 was hip, we would all install ESX on the local harddisk (or SAN disk). With ESX 3.5, the first ESXi version was released but only few were using it. With 4.x ESXi really got a large install base and more and more people were moving to installing ESXi on USB or SD card. Now with vSphere 5 and ESXi as the only hypervisor (no more ESX), we don’t need to install ESXi at all.

When running ESXi stateless, the host will PXE boot and load an ESXi image into memory. There is no more need to have local disks, SD card or USB on your host. Another advantage is that you can now switch the ESXi version the host is running by just rebooting or refresh the host configuration with a reboot. Adding new ESXi hosts to your cluster has become a task of just a few minutes. Let’s have a look at how to configure auto deploy. This blog post will take you through the steps of setting up the required infrastructure and prepare an image and deploy it to a host.

My series on VMware vSphere 5 Auto Deploy:

vSphere 5 – How to run ESXi stateless with vSphere Auto Deploy
vSphere 5 Auto Deploy PXE booting through Cisco ASA firewall
Updating your ESXi host using VMware vSphere 5 Auto deploy
My first Auto Deploy design for real production environment

Important: When running stateless, your ESXi host cannot boot without all vCenter Auto Deploy components being active. Changes are big that your vCenter Auto Deploy is running in a virtual machine. Should your whole environment power down, you will have a problem getting everything running again. Personally I’m thinking of having an extra USB install of ESXi 5 at hand at all times. Just in case. The bigger the environment the smaller the chance this will happen. But especially for some home-lab environments this might be something to think about.

Infrastructure

When using Auto Deploy a host will use DHCP to get an IP address and learn if there is a PXE boot server. The PXE boot image is downloaded and the pre-boot environment is ran on the host. It will now contact the Auto Deploy server which will now check the rule engine. Based on the rules in the rule engine an image is now offered to the host and a host profile is applied to the host.

An image is based on a VMware ESXi deployment image combined with optional extra VIBs (VMware Image Bundles). You can download a “ready-to-deploy” image of ESXi from the VMware download site, just like you would download your normal ESXi install files. For example the latest ESXi version I could download in the beta, was build 441354. The normal ESXi install to download would be: VMware-VMvisor-Installer-5.0.0-441354.x86_64.iso, but on the download page there also is a: VMware-ESXi-5.0.0-441354-depot.zip. This is an ESXi image suited for auto deploy, containing the ESXi VIB. It is also possible to combine VIBs into a new image if needed, for example when you want to add extra drivers or special VIB supplied by the hardware manufacturer.

Another component is the Image Builder PowerCLI. Using PowerCLI commands a rule set is created and together with a host profile connected to a host or number of hosts.

To make all this possible you need to have the following in your environment:

  • Hosts that are capable of running ESXi 5
  • TFPT boot server
  • DHCP / DNS server
  • Windows server to run Auto Deploy or vSphere vCenter Appliance
  • vSphere PowerCLI

The vCenter Auto Deploy manual will show you how to enable the vSphere vCenter Appliance for Auto Deploy. In this post I will use a Windows server to act as TFTP and Auto Deploy server.

Installing TFTP server

To keep things simple I downloaded the WinAgents TFTP server. It is a 30 day trial. Installation is a simple next, next, finish. Create a directory on your Windows server, for example: E:\tftproot and make this your TFTP root directory.

Set up DHCP

To be able to PXE boot, you need to set some options in your DHCP scope. Open the scope on your DHCP server and add the following options:

066 – Boot server host name:   <ip of your TFTP / PXE boot server>
067 – Boot file name: undionly.kpxe.vmw-hardwired

Set up vSphere Auto Deploy

On the vCenter installation media you’ll find the Auto Deploy software. It can be installed on your vCenter server but you can also run it on a separate Windows server and have it point to your vCenter server. I kept everything, including the TFTP server, on my vCenter host. Installation again is a simple next, next, finish in which you only need to supply the IP of the vCenter server and the correct credentials. After the installation has finished you can install a plug-in in your vSphere Infrastructure Client. You can see the Auto Deploy icon on the home tab.

Install TFTP boot image

The last step is now to add a boot image to your TFTP server root directory. In the vSphere client click the Auto Deploy plug-in and see the following page:

Now choose to download the TFTP Boot Zip and extract this ZIP file into your TFTP root directory. With this, the first part is ready. To test if things are working, you can boot an ESXi host or just for test boot a VM. Make sure that the host or VM is using PXE boot. You should see that it is assigned an IP address and it starts loading a TFTP image.

Next you’ll see that although a TFTP image was loaded, there was no ESXi image associated with this host. This screen also displays a number of Machine attributes we could use later on when tying a rule set to a host.

Installing vSphere PowerCLI

Well this should be a piece of cake since I expect many of you to have done this many times before. Download the vSphere 5 PowerCLI and install it on the server on which you will be working with your images. To test if your VMware PowerCLI is working, start the VMware vSphere PowerCLI command prompt and run: Get-DeployCommand. This should give you a list of all the commands you’ll need to work with Auto Deploy. After this you’re done and all requirements for vSphere Auto Deploy have been installed.

Preparing your first image

When installing vSphere Auto Deploy, you’re asked for a directory for your SoftwareDepot. This might be a bit confusing since it actually is never used by YOU, only by the Auto Deploy software for internal ‘housekeeping’. My advice is to also create a directory called ‘VIB-downloads’ in which you store the VIBs and images you want to deploy.

First we’ll deploy the basic VMware ESXi 5.0 image to a new host without any further configuration. We’ll attach the image to the host based on mac-address and want the host to show up in your vCenter in a folder named ‘Staging’. Since it has no configuration, I don’t want it to show up in a cluster yet.

All the info we need:

  • IP address of the host and the DNS hostname
  • Folder name: ‘Staging’
  • Mac address: 001a92b8da77
  • Image name: VMware-ESXi-5.0.0-441354-depot.zip (as downloaded from VMware site)
  • Image name: ESXi-5.0.0-441354-standard (after being added to the depot)

Important: Since we’ll be using DHCP to assign a ‘fixed’ IP and hostname to the host, you need to create a DHCP reservation for the mac-address. In your DHCP scope create the reservation and use the proper hostname. In DNS create an A-record for this hostname and IP address and make sure you give it a PTR / reverse lookup record too.

Adding a new image to your depot

This is what confused me during my first steps with vSphere Auto Deploy. I was doing all sorts of things trying to add the VMware-ESXi-5.0.0-441354-depot.zip to the software depot somehow. I copied the whole zip into it, but that didn’t work, unzipping and copying didn’t work either. Until I learned that you don’t do anything with the software depot directory. Place the downloaded image ‘VIB-downloads’ and then run the command:

Add-EsxSoftwareDepot “E:\VIB-downloads\VMware-ESXi-5.0.0-441354-depot.zip”

You should now see how the image is inserted into the software depot. After this is finished, you can run the command Get-EsxImageProfile, to see what images are present in your depot. In my example you can see I now added two versions of ESXi beta. Each image contains two flavors, a no-tools and a standard version. I usually use the standard version to deploy.

Get-EsxImageProfile

Deploying your first host

We now have an image ready to deploy, when a host reboots he’ll pick up the TFTP image and will ask the vSphere Auto Deploy server for an image. Next step is to create the rule to match a host to an image. When creating rules, you should know there are two rule sets, a ‘working-set’ and an ‘active-set’. The ‘working-set’ is like a depot of rules, the ‘active-set’ are the rules that are really available to hosts. It is a bit confusing sometimes, especially since the commands to attach / create rules are the same. I would have preferred a different naming or just use an extra attribute like “active” and “in-active”. Think that would have made it easier to understand. Also, I was having or actually still have problems editing rules. I just can’t seem to get that work so instead of editing them I delete and re-create them. When I get it to work I’ll update this blog post.

As I was saying, we’re now going to create a rule to connect the image to the host. This is where the ‘New-DeployRule’ command comes in.

New-DeployRule –Name “PreStaging” –Item “ESXi-5.0.0-441354-standard”, “Staging” –Pattern “mac=00:1a:92:b8:da:77”

The new rule that has just been created in called “PreStaging”, it will make sure that the image called “ESXi-5.0.0-441354-standard” ( Get-EsxImageProfile ) will be deployed to a host with the mac address of 00:1a:92:b8:da:77 and will be placed in the “Staging” folder in vCenter.

With the command Get-DeployRule you can see the rule you just created. This is a rule in the ‘working set’. To make the rule part of the ‘active set’ you need to add it like this:

Add-DeployRule -DeployRule “PreStaging”

To check the rules in the ‘active set’ you run the Get-DeployRuleSet command. You’re all set now to boot your host and see how it installs and then shows up in your vCenter.

Using a host profile

After your host has shown in vCenter you’ll see that it is not configured. If you would configure it now and reboot it, all changes will be lost. The way to save your configuration settings is to create a host profile. A great feature which I found particularly useful when trouble shooting host profiles is “Enable / disable Profile Configuration”. This gives you the option to only partly apply a host profile.

You’ll notice that when working with the host profiles and get more advanced configurations, you’re going to run into problems where settings can’t be applied the way you want it and errors without great detail will be thrown at you. For now I’ll show you how to work with a simple host profile.

Configure the new host like normally. Create the vSwitches, attach the datastores, make sure the NTP settings are correct, etc, etc. Since you’re running diskless hosts, it is important to correctly setup syslogging and the core dump location.

For setting up syslogging, I would like to refer you to this post from Jason Boch: “Configure a vCenter 5.0 integrated Syslog server” and use his post to configure syslogging for your ESXi host.

For redirecting the core dump, it is advised to install the vCenter Coredump tool. Just like the syslog tool, the Coredump utility can also be found in the vCenter tools directory. Again install it with a simple next, next, finish. Your server can now also receive coredumps in case an ESXi host would encounter a purple screen of death (PSOD). After configuring the vCenter Coredump utility, configure your ESXi host to use the coredump server. To do this, go into the configuration screen of your host, go to the security profile and enable SSH. Now logon to the ESXi console using your favorite SSH client and run the following commands:

esxcli system coredump network set --interface-name vmk0 --server-ipv4 192.168.0.40 --server-port 6500
esxcli system coredump network set --enable true
esxcli system coredump network get

The last line will show you if the new setting has been enabled. Now logout of your ESXi host and switch back to your vSphere Client.

Important: Don’t forget to disable SSH again, because we’ll be generating a profile and wouldn’t want SSH to be enabled on all new hosts running from this profile.

Now go to the “Host and Clusters” view in your vSphere client and select the host you have just prepared. Right click the host and select “Create Profile from host”. Give the profile a name, for example ‘Profile-Cluster01”. Now attach the profile to this host using the Host Profiles section in the vSphere client and have the profile checked to be compliant.

Auto deploy the host with the host profile

The first rule we created made sure that the host with certain mac address would be connected to the standard image and would be put in the “Staging” folder.

New-DeployRule –Name “PreStaging” –Item “ESXi-5.0.0-441354-standard”, “Staging” –Pattern “mac=00:1a:92:b8:da:77”

Now we are going to create a new rule. The new role will move the host into the production cluster called “CL01” and the host profile “Profile-Cluster01” will be used. Because we’re pretty sure this works without testing (yeah right), I’m not going to do this based on mac address, but on IP address range. You should use the IP range that you use in the DHCP scope for the ESXi hosts and create a reservation in the DHCP scope for each host and also create a DNS record. The new rule would be created like this:

New-DeployRule –Name “Prod-CL01” –Item “ESXi-5.0.0-441354-standard”, “CL01”, “Profile-Cluster01” –Pattern “ipv4=192.168.0.100-192.168.0.110”

In the above rule, ‘Prod-CL01’ is the name of the rule, ‘CL01’ is the name of the cluster, ‘Profile-Cluster01’ is the name of the host profile and ipv4 is the DHCP range.

In the ‘working-set’ we now have two rules (“PreStaging” and “Prod-CL01”) and in the ‘active-set’ the “PreStaging” rule is active. Using the remove command, we can remove the “PreStaging” rule from the ‘active-set’ and next we add the “Prod-CL01” to the ‘active-set’ and double check what we have done:

Remove-DeployRule -DeployRule “PreStaging”

Add-DeployRule -DeployRule “Prod-CL01”

Get-DeployRuleSet

And we’re done !!! You’ll see that if you now reboot your hosts, they will come back and will be added to the CL01 cluster fully participating as a normal host.

My series on VMware vSphere 5 Auto Deploy:
vSphere 5 – How to run ESXi stateless with vSphere Auto Deploy
vSphere 5 Auto Deploy PXE booting through Cisco ASA firewall
Updating your ESXi host using VMware vSphere 5 Auto deploy

29 thoughts on “vSphere 5 – How to run ESXi stateless with vSphere Auto Deploy

  1. Good article, I have a question though.  How would you address automating the assignment of a static vMotion address per host?

  2. check out tftpd for your tftp server. little trick all us people use for upgrading cisco routers. best free tftp server you can get for windows. and it’s free.

    Also, thanks for the spoonfed tut. it was great.

  3. Since ESXi 4.1 there is no more need for separate VMotion address. But if you work with static IPs you could edit the host profile and have “questions” in it. When you apply the profile with the questions included, the host will show up in maintenance mode the very first time. After you answered the questions, it will save these in “answer files”. Which are stored in vCenter. Then after each reboot vCenter will apply the answers automatically.

  4. Very thorough writeup. The only thing i disslike in complete picture is vCenter + AutoDeploy in a virtual machine on ESXi host. Enterprise organizations would not permit running around with USB stick and booting up one host registering a vCenter VM on it (unless you already know where you vCenter is) and enabling HA and steteless environment. Downtimes still hapen no matter how big the customer is.
    So, vCenter as a physical machine again? Why not? We also also have to make sure the DNS, DC and other dependencies are physicaly available. Or to provide a separete management HA cluster for all those VMs but with no stateless?

  5. Tried to locate Depot Zip file for ESXi 5.0 on vmware website. Can you please provide the URL to download the offline bundle file

  6. Do not want to install AD, DNS, DHCP then just install the Microsoft 208r2 Transport sever by itself and still deploy .Boot PXE with Transport Server.
     Includes a PXE provider when you install the Transport Server role service. You can use Transport Server to network boot, multicast data, or both as part of an advanced configuration.
    Transport Server is a stand-alone server…
     That is, when you use Transport Server for network booting and multicasting, your environment does not need Active Directory Domain Services (AD DS) or Domain Name System (DNS).

  7. Just came across this browsing through the internet.  Can you elaborate on the lack of need for seperate VMotion address?

  8. Since ESXi 4.1 you don’t need a seperate vmkernel interface and/or VMotion IP address. The management address and VMotion address can be set to the same vmkernel interface. So, there is no need anymore for special VMotion IP. If you still want to separate VMotion traffic from Management traffic you should assign different vmkernel interfaces and IP’s but I don’t see the need for it.

  9. Gabe – I am at VMworld SFO and ran into some SysOp guys who need to see this post so thanks. On the vmkernel “question”, for HA I think separate IP’s are required before it will enable. Catch me if you are here.

  10. Gabrie I have 2 questions:

    1) As suggested by our consultants, I always installed my ESXi hosts using the manufacter iso images because they already have all agents. For example for Dell I download the ESXi image from dell site.
    Is possibile to use these images?

    2) Supposing that you need to change an advanced attribute on all hosts, do you confirm to me that is necessary to create a new host profile and re-deploy?

    3) You spoken about problems working with host profiles advanced configurations (you wrote: you’re goint to run into problems whre settings can’t be applied….)
    Do you have some news about release 6? All of them solved?
    My question is just because I would switch to autodeploy using esxi 6 but before to proceed I would know your opinion and experience.

Comments are closed.