When preparing your Virtual Infrastructure environment for vSphere 5 and vCenter 5 you also need to think of all the vCenter applications that are dependent on vCenter. One of  them is vCenter Heartbeat. To be able to run vCenter Heartbeat on vCenter 5 you will need to upgrade vCenter Heartbeat to version 6.4. Normally upgrading vCenter Heartbeat isn’t a big deal but there has been a change from 6.3 Update 1 to 6.4. Where you could normally use a configuration of “identical nodes”, you will now have to switch to “non-identical nodes” according to KB 1014435.

From the KB article: “Important: When performing an upgrade to vCenter Server Heartbeat 6.4 from 6.3 Update 1 with Identical nodes or from any previous version, you must first migrate from Identical nodes to Non-Identical nodes before performing the upgrade.”

Switching “identical” to “non-identical”

To switch your configuration from “identical” to “non-identical” nodes you can use KB 2002112 “Migrating vCenter Server Heartbeat 6.3 Update 1 and later from identical nodes to non-identical nodes”, but it needs some additions.

First there is the following note in the KB: “Note: This procedure reconfigures vCenter Server Heartbeat 6.3 Update 1 and later installed in a LAN environment where vCenter Server and SQL Server are protected but not installed on the same machine (remote SQL Server).”  However, this procedure DOES work for locally installed SQL Servers as well. In my testlab it even worked with SQL Express, even thought SQL Express is NOT SUPPORTED.

Second difference between real world and KB is that you will need to reconfigure your vCenter networking, but there is no mention of that in the KB. Below is the procedure I followed IN A LAB ENVIRONMENT when testing the upgrade for a customer of mine.

Test environment:

• Primary server: vCent01
• Public IP: 192.168.233.10
• Heartbeat interfaces: 10.0.0.1 and 10.0.5.1
• Secondary server: vCent01 (because it’s a clone of vCent01)
• Public IP: 192.168.233.10 (because it’s a clone of vCent01)
• Heartbeat interfaces: 10.0.0.2 and 10.0.5.2

When upgrading to heartbeat 6.4 and switching to “non-identical” nodes, you’ll need new server names and an extra public (passive) IP address for your primary and secondary vCenter.

• Primary server: vCent00 will be the new name
• Passive Public IP: 192.168.233.20
• Secondary server: vCent02 will be the new name
• Passive Public IP: 192.168.233.30

Following the first steps of KB 2002112, shutdown vCenter Heatbeat on both servers leaving the applications running. Do this through the VMware vCenter Heartbeat Management console.

On the Secondary Server part, follow steps 1 through 3.

Re-enter the “Service Name”. In my case this is “VCENT01”. Steps 4 and 5 tell you to assign the Principal Public IP and the passive IP in step 7. When entering the correct IP addresses, you will run into a problem later on after removing the secondary server from the domain and trying to re-add it. The Windows network stack is not update with the configuration change in IP you make here. To solve this set the IP on the network interface to 192.168.233.30. Then use this as the passive IP address and keep 192.168.233.10 as the public IP address.

Proceed with steps 8 through 11.  Step 12 explains how to rename the secondary server, in my case the new name will be vcent02. Use the last steps to rejoin the server to the domain. Then proceed to the primary server and also change its IP address at Windows level like we did with the secondary server and later on change the server name to vCent00. The primary server doesn’t need to be removed from the domain.

Upgrade VMware vCenter Heartbeat 6.3 update 1 to 6.4

After you completed these steps, make sure your VMware vCenter Heartbeat is running fine again. You might need to wait some time for the file sync to complete. After everything is running smooth again, the next step is the upgrade itself. This is a fairly straight forward procedure, outlined in KB 1014435 titled: “Upgrading vCenter Server Heartbeat from an earlier version to a later version”.

See full post at: Upgrading VMware vCenter Heartbeat 6.3 update 1 to 6.4

Zerto http://www.Zerto.com/ (Zero RTO) is a startup company that I first heard about when they were presenting at the Techfield Day in Boston. All information was under embargo, which didn’t allow us delegates to publish about Zerto until June 22^nd and so I had to wait with this post until now. The presentation done by Chen Burshan – Director Product Management and Gil Levonai – VP Products got me really excited about the product and I started playing with the beta. Below is an explanation of what Zerto does and some of my own experiences with it.

Full storage agnostic virtual machine replication

What Zerto promises is full replication of your virtual machines between different sites, in depended of the brand of storage and even on high latency WAN connections.

Zerto is an Israel based company and was founded by Ziv and Oded Kedem who also founded Kashya, which was acquired by EMC and is now the foundation of the EMC RecoverPoint product. After selling their company and learning from the good and bad about traditional BC/ DR solutions, Ziv and Oded used their knowledge to start working on Zerto. When designing Zerto the main thought behind the product was that replication should no longer be done at storage level, but should move to the next level. Zerto will prove today that you can move replication into the hypervisor and still keep all your functions of storage and hypervisor. Zerto is meant to be an enterprise solution, meaning it will be able to handle many virtual machines and work across geographically dispersed data centers.

In short what Zerto will offer is Virtual replication:

• virtualisation aware
• software only
• Tier-1 replication, RTO’s of seconds
• Enterprise class replication
• Purpose built for virtual environments

How does Zerto replicate

On each ESX/ESXi host with virtual machines that need to be replicated a virtual replication appliance will be running on the ESX/ESXi host.

The VRA uses the VMware APIs, which allows it to see all data coming through the IO stack and will then replicate it to the secondary site. It will only see the data in the IO stack and not interfere with the process of writing to disk. Should the VRA fail, ESX/ESXi will continue to operate normally and VMs will not notice any delays.

The virtual appliance makes sure the data is replicated to the virtual appliance on the ESX/ESXi host at the target host. This target can be locally or on a remote site. The data to be replicated will be compressed before sending it to the target using built-in WAN compression & throttling techniques. It can handle WAN disconnects or degradation automatically.

When replicating over WAN, it could take quite some time to finish the first sync between source and target if the source site is a few Tera Bytes in size. To save a lot of time, it is possible to first do a restore from your regular backup on the target site and then have Zerto perform a sync, saving you a lot of time.

The journaling

Source virtual machines can have thin or thick VMDK’s or a physical or virtual RDM and at the target site the target data can be in different places than the source. For example a database server on the source site has its VMDKs on different datastores but grouped in one datastore at the target site. At the target site all received data will be written into the target and a journal will be kept in a VDMK.

In case of a disaster on the source site, it is of course possible to switch to the current status of the target site, otherwise it wouldn’t be a disaster recovery product. However the journaling system used, makes it possible to go back in time and restore the status of one or two hours ago, maybe longer if you have the disk capacity for this. In many cases, real DR is not because of natural disaster, but mostly logical errors, therefore it is ideal to be able to go back further than just the latest status.

The journal holds all data changes and will take up extra disk space that you should plan for when determining how far back in time you want to go. An IO intensive application will of course generate a lot of data and therefore use quite some journaling space.

The journaling system also enables you to test-run a virtual machine on the target site, while still keeping replication running. While starting the replicated virtual machine for test purposes, replication data coming from the source site is still written into the target VMDK. Not only can the replicated virtual machine be tested for correct failover, it can also be used to create a small test environment for other purposes.

When replication fails

As explained above, on the source site every write that goes through the SCSI stack of the virtual machine is split into the memory of the virtual appliance and will then be replicated to the target. However when the link between source and target fails, the virtual appliance will start using a bitmapping technique to remember all the changes to be able to start replication as soon as the link is up again.

To prevent running out of memory in the virtual appliance the bitmap technique will store less detail the longer the link is down. What happens is this:

• Under normal operation the virtual appliance writes changes at the smallest block size to memory. If blocks 2, 5, 9, 13, 17, 20 have changed this is written into memory and removed from memory as soon as these blocks have been replicated.
• However, when the link fails and the number of changes to remember in memory keep growing, the virtual appliance decides to store less detail. In our example, the virtual appliance could decide to only remember that something changed from block 2 to 9 and 13 to 20.
• If the link stays down even longer, the virtual appliance might go even further and now only remembers that block 2 through 20 has changed.

While storing less and less details if the link stays down for a long time, the amount of data to be replicated after the outage will increase.

Zerto Virtual Replication architecture

The architecture of the Zerto Virtual Replication environment is actually rather simple. From the bottom we have the Virtual Replication Appliance. This is a debian based virtual machine that is able to see all SCSI writes by the VM and splits the write into its memory. Then it will try to replicate the blocks over to the Virtual Replication Appliance on the replication site, using TCP port 4005. It will also talk to the Zerto Virtual Manager over TCP port 4005 to keep it updated on all the stuff the VRA is doing.

The Zerto Virtual Manager is a small piece of Windows software that can be installed on the vCenter server, but can also be installed on a separate server, just as long as it can talk to vCenter. The Zerto virtual manager will monitor replication, manage site links, protection groups, protected VMs, etc. It needs to talk to the Zerto Virtual Manager on the target site and does this over TCP port 9081.
The last component is the vSphere Virtual Infrastructure client that runs the Zerto plugin and enables you to manage both protected and replication site from the same interface. It will run over HTTP port 9080.

Installation and protecting VMs in a few simple steps

The installation of the Zerto environment was actually very easy. I will walk you through it with a number of screenshots, just be aware that this isn’t a full step by step walk through.

First you install the Zerto Virtual Manager on a Windows server. This is a very simple next, next, finish process. The only options you have to enter are the IP of your vCenter and the user account to be used to access vCenter.

After installation is complete you run the same install on the replication site. Next step to pair both sites. You can also choose to first install the VRAs but I went for pairing the sites. For this you go into the vSphere VI Client and at cluster or datacenter level click the Zerto tab. In this tab you see a fancy interface with very clear instructions. Either you click the big “Install VRAs” button or you click the “Pair…” button to pair your protection and replication site.  Although the interface looks good, I’m not always happy with it. It requires adobe flash and doesn’t always give the proper response when you click a button. I want to see the button change when I click to prevent me from clicking it multiple times.

Pairing the sites is very easy. Enter the IP address of the Zerto virtual manager of the replication site and click PAIR. That’s it. Next we’ll install the VRA, the virtual machine that captures all the writes and does the actual replication. In the screen where you saw the option to pair the sites, you also have the option to install the VRA. Press the button and the next screen will ask you where to install the VRA to. You’ll see a list of ESX/ESXi hosts, datastores and networks to choose from. The VRA can use a DHCP address or fixed IP. After you selected the desired settings, you push the button “Install selected VRAs” and you wait. In the vCenter task list you’ll the progress of the installation.

Installation progress:

After this the Zerto framework is ready. All we need now is VMs to replicate. Replication is done based on Virtual Protection Groups (VPG). Each protection group contains one or more VM that you want to have the same set of replication and recovery settings. To create a VPG just click the “New VPG” button, add a VM and walk through the recovery settings, things like RPO Threshold, Maintain history, Max Journal size, what host or cluster to replicate to, which datastores to use, which networks, etc, etc.

Once this is done, your replication should start and can be seen in the Zerto tab at cluster level or for the specific VM at VM level.

Summary

As you see, this is a very easy way to setup replication between two sites and the fact that it doesn’t matter what storage you are using will not only be appealing to many SMB customers but also large enterprises. Playing with the beta of Zerto has shown me that this is a very nice product even though I couldn’t really stress test it in my lab environment. But I surely will do some more testing in a larger environment.

Some last notes I’ve written down during the Techfield Day event but haven’t mentioned yet in this post:

• Fully support vMotion, HA, vApp
• Group Policy and configuration
• VSS support
• HA boot orders, vApps, host affinity rules replicated as well
• Version 1 will be all vmware. On road map (months, max 1 yr) other hypervisors. Not sure what other hypervisors. They will be able to replicate between DIFFERENT hypervisors !!!!
• Protect both sites: Symmetrical replication.  You can replicate both ways at the same time, within the same LUN.
• Working on multi-tenancy.

Be aware that the Tech Field Day event is fully sponsored by the companies we visit, including flight and hotel, but we are in no way obligated to write about the sponsors.

See full post at: Zerto Replication and Disaster Recovery the easy way

At the Dutch VMUG, together with my colleague Jos Vanaubel, I presented about how to do cheap disaster recovery using only PowerShell. At the London VMUG I held the same presentation again and promised to upload the scripts to my blog. Well, here it is in PDF format:

The scripts can be downloaded by clicking this link -> Cheap Disaster Recovery Scripts

Before we can failover to DR site, we need the data from the production site. Therefore run the “preparation” scripts weekly or daily, depending on the number of changes. These are your backups. When failover needs to occur, first run failover step 1 and 2, then start the import section. Last step is to boot the vms in failover step 3.

Use the scripts in this order (in the zip file an Excel sheet is included with this info):

A special word of thanks to the guys who helped me with these scripts:
Alan Renouf – http://www.virtu-al.net
Luc Dekens – http://www.lucd.info/
Arnim van Lieshout – http://www.van-lieshout.com
Maish Saidel-Keesing - http://technodrone.blogspot.com/

If you have improvements or additions to the scripts, let me know, I’d be happy to post them here.

See full post at: Cheap disaster recovery

Just before I was heading home, my colleague asked me to assist with a VM he couldn’t power on anymore. Seemed a customer had tried to commit a snapshot and the job timed out. The customer then tried some other things and suddenly the VM was down. While trying to power on, there was an error telling us that disk1 could not be found.

Checking VM settings in VMX file and VMSD file

In the VM properties and in the vmx file, I found these references:

First harddisk: disk0 [DataStore4] FS05/FS05-000002.vmdk
Second harddisk: disk1 [DataStore4] FS05/FS05_1-000002.vmdk
Third harddisk: disk2 [DataStore4] FS05/FS05_2-000002.vmdk

These are clearly pointing to a snapshot. Strange thing was however that disk1 was 32GB in size instead of 1TB !!! That got me scared a little. In the vmsd file that normally keeps record of the snapshots I found the following references:

snapshot0.disk0.fileName = "FS05.vmdk"
snapshot0.disk0.node = "scsi0:0"
snapshot0.disk1.fileName = "FS05_1-000001.vmdk"
snapshot0.disk1.node = "scsi0:1"
snapshot0.disk2.fileName = "FS05_2-000001.vmdk"
snapshot0.disk2.node = "scsi0:2"

So there was the first contradiction. The vmx file is pointing at the *-000002.vmdk files while the vmsd file is pointing at the *-000001.vmdk files.

Finding an error report

Next step was to look for any mentioning of an error in the log files. Quickly I discovered that the vmware.log file reported that it is looking for the *-000002.vmdk files and that the FS05_1-000002.vmdk was not found:

Unable to find file FS05_1-000002.vmdk
Unable to find file FS05_1-000002.vmdk
DISK: OPEN scsi0:1 'FS05_1-000002.vmdk' persistent R[(null)]
Unable to find file FS05_1-000002.vmdk
DISKLIB-LINK  : "FS05_1-000002.vmdk" : failed to open (The system cannot find the file specified).
DISKLIB-CHAIN : "FS05_1-000002.vmdk" : failed to open (The system cannot find the file specified).
DISKLIB-LIB   : Failed to open 'FS05_1-000002.vmdk' with flags 0xa (The system cannot find the file specified).
Msg_Post: Error
[msg.disk.fileNotFound] VMware ESX Server cannot find the virtual disk "FS05_1-000002.vmdk". Please verify the path is valid and try again.
[msg.disk.noBackEnd] Cannot open the disk 'FS05_1-000002.vmdk' or one of the snapshot disks it depends on.
[msg.disk.configureDiskError] Reason: The system cannot find the file specified.----------------------------------------
Module DiskEarly power on failed.

Contents of the VM directory

I then looked at the vmdk files in the VM directory. First for the first disk, which is FS05:

FS05-flat.vmdk (32G)
FS05.vmdk (397 bytes)
FS05-000002-delta.vmdk (208M)
FS05-000002.vmdk (217 bytes)

The FS05–flat file is the one holding all the data, the FS05.vmdk file is the descriptor file, it’s an ASCII file that you can look into. The same goes for the FS05-000002.vmdk file that is a descriptor of the FS05-000002-delta.vmdk which contains the data of the snapshot.  Strange thing is that there is no FS05-000001.vmdk or delta.vmdk. It is not entirely impossible that the 000001 file is not present, for example when you would have made two snapshots from the same starting point, instead of each snapshot based on the previous one.

The next set of files is for the second disk FS05_1:

FS05_1-flat.vmdk (1.0T)
FS05_1.vmdk (403 bytes)
FS05_1-000001-delta.vmdk (13G)
FS05_1-000001.vmdk (223 bytes)
FS05_1-000002-delta.vmdk (402M)

Now here we have a bigger issue. The FS05_1-flat.vmdk containing 1TB of very important data (it always is when a customer calls), is “covered” by the FS05_1.vmdk. The FS05_000001-delta.vmdk also has its descriptor file FS05_1-000001.vmdk but the FS05_1-000002-delta.vmdk does not have a descriptor file.

The FS05_2 disk luckily looked quite complete:

FS05_2-flat.vmdk (20G)
FS05_2.vmdk (399 bytes)
FS05_2-000001-delta.vmdk (32M )
FS05_2-000001.vmdk (221 bytes)
FS05_2-000002-delta.vmdk (16M )
FS05_2-000002.vmdk (228 bytes)

About CID’s and ParentCIDs’

The descriptor vmdk file normally holds a CID and a ParentCID value, these values link the snapshots and flat vmdk together.  Let’s look at the CID’s in the FS05_2 vmdk’s and how they point to each other. In red you see the file name of the vmdk descriptor file.

Important is the pointer to the parentCID and the parentFileNameHint. In file FS05_2-000001.vmdk you see how the parentCID from the 000002 file matches the CID from the 000001 file. The last file to look at is the FS05_2.vmdk file. Since this is the vmdk file that describes the –flat.vmdk it holds some more info on the disk geometry of the vmdk, but this info is not important to us now. Again we only use the CID lines. You’ll notice the parentCID value is ffffffff. In all vmdk files that are related to the –flat vmdk I found the parentCID value being ffffffff, but I don’t know if this is required. Again we see that the value of parentCID of the 000001 matches the CID of the FS05_2.vmdk. This snapshot chain is OK. The FS05_2 files have been checked and confirmed that the chain is correct. Next step is to reconstruct the chain for the FS05_1 vmdks.

Reconstructing the vmdk

This actually is rather simple. Since we’re missing the FS05_1-000002.vmdk we can just copy the FS05_1-000001.vmdk and edit it to hold the right entries. Be aware that there are some other entries too in the FS05_1-000001.vmdk file and we need them, so a real full copy is needed of the descriptor. Below is the content of the FS05_1-000001.vmdk:

# Disk DescriptorFile
version=1
CID=79cda434
parentCID=7cec9dae
createType="vmfsSparse"
parentFileNameHint="FS05_1.vmdk"
# Extent description
RW 2147483648 VMFSSPARSE "FS05_1-000001-delta.vmdk"
# The Disk Data Base
#DDB

Now copy this to FS05_1-000002.vmdk and edit the lines I marked:

# Disk DescriptorFile
version=1
CID=458dea24 <--- Just pick any number
parentCID=79cda434  <--- make sure it links to the parent
createType="vmfsSparse"
parentFileNameHint="FS05_1-000001-delta.vmdk"  <--- edit this
# Extent description
RW 2147483648 VMFSSPARSE "FS05_1-000002-delta.vmdk"  <--- edit this
# The Disk Data Base
#DDB

The 458dea24 CID can be any number, you just make it up since it has no relation to the VMDK. This is what took a lot of time the first time I was playing with this. I thought I had to retrieve this info from the VMDK somehow, but from VMware Support I learned that this CID can be any number.

The last few steps

After the file has been edited, the chain of the vmdks is correct again. Now the last step is to get this thing up and running again. Remember that the vmsd file was also incorrect, but we can just delete the vmsd file, you did make a backup / copy? Next we remove the VM from the vCenter inventory and the re-add the VM. And as the last step take a new snapshot and then commit all snapshots. After the snapshot has been committed, check the VM settings to see if they are pointing to the correct parent files. If you’re confident that this is correct you’re done. Now power on the VM.

See full post at: Snapshot fails with incorrect CID info in the VDMK file

When you build your virtual infrastructure with only ESXi hosts that you also lock down for security reasons, you might be in for a little surprise when you want to get your VI up and running again after major maintenance or a failure. First thing to do after the virtual infrastructure has been down, is to get vCenter up and running again. In a previous post “How to quickly recover from disaster” I already explained the idea of running vCenter always on the first host in your cluster. In case of failure you don’t have to search where DRS left your vCenter VM, you just connect the VI Client to the first host and start the vCenter VM. Don’t forget you need your Active Directory and SQL database before starting vCenter.

With an all ESXi environment and locked down hosts, you cannot use the VI Client to connect and start the necessary VMs, at least that is what many think, but this isn’t completely true. The Locked down mode does NOT prevent direct VI Client connections as many think, it does however prevent direct VI Client connetions made with the root-user account. The same goes for PowerCLI, vCLI, vMA, or any of the other public APIs. In locked down mode the root user has no direct access. You can however create an extra user on your ESXi install and assign this user administrator rights. Then, after enabling locked down mode, you can still make a direct VI Client connection to your ESXi box and perform some admin tasks like starting VMs.

Another option would be to just get access to the console of the ESXi host using ILO, KVM, DRAC or similar techniques and disable lockdown mode. After disabling lockdown mode, you can then again make root access using the VI Client.

To summarize:
- Lockdown mode for ESXi does prevent root access using VI Client, PowerCLI, vMA, API’s etc.
- Lockdown mode for ESXi does NOT prevent other users accessing the ESXi host using above mentioned tools. Just be sure to first create this user.
- Procedure in an enterprise to create that local user on all ESXi hosts, would be to use (for example) PowerShell to create that admin user and then enable the lockdown mode.

See full post at: About ESXi lockdown mode

Recently I had the opportunity to review StorMagic’s SvSAN software. Their current release available from their website is only compatible with VMware ESX 3.5, since my home lab is already running VMware vSphere I received the latest beta 4.1.913 which does work with vSphere. In this review I will not write on absolute performance since my home lab will probably hit its limits sooner than the reviewed software does.

What StorMagic does

With StorMagic it is possible to use the local storage (your internal hard disks) of your VMware vSphere hosts as shared iSCSI storage. Other hosts can connect to the targets as if it was a normal iSCSI NAS or SAN. With the free StorMagic version, you can have up to 2Tb of storage managed by StorMagic. Licenses for 4 Tb, 8 Tb and unlimited storage are also available. The advantage of sharing your local storage is that your vSphere environment can now use the extra features like VMware High Availability, VMware DRS and VMware VMotion and you don’t have to spend money on buying a separate iSCSI NAS or SAN.

A disadvantage of having your shared storage on one vSphere host is that this vSphere host will now become your most important part in your Virtual Infrastructure and downtime of the host, is downtime of your storage and therefore downtime for all the VMs on this storage. Features like VMotion and HA are meant to reduce the downtime in your environment and it wouldn’t be wise to now increase your downtime (risk) by putting all your VMs on shared storage that is kept up by just one host.

To reduce this risk StorMagic offers an extra license called StorMagic SvSAN High Availability. With this license the shared storage can be mirrored to a second host. Should one of the hosts fail, the other host will immediately take over and your VMs that are running on the shared storage will keep on running. Well, most of them, because the VMs running in memory of the failing host will of course crash, but VMware HA will be able to get them up and running from the shared storage.

How a virtual infrastructure can be built with StorMagic

In the following figures is shown how local storage is used to create shared iSCSI storage. In the first figure you see the VMware vSphere host which has 4 local disks attached. The first local disk is used for the installation of vSphere, which by default (new in vSphere) creates a local vmfs volume on which the COS (Service Console) is installed. This local VMFS will also be used to install the StorMagic SvSAN appliance on, which needs a 256MB virtual disk and a larger 20GB virtual disk. The remaining three disks will be used to create a Raid5 volume that will be used for as shared storage in StorMagic. There can be two choices in how to use this Raid5 volume. If the host has a Raid5 capable controller, the disks can be joined into a Raid5 volume at hardware level and presented as one disk to StorMagic or StorMagic can group the three local disks into a Raid5 volume and will do the Raid calculations.

The second figure shows how the shared storage is connected to by the other VMware vSphere hosts in the cluster. Not showing in this figure is that the vSphere host that holds the shared storage can and in most cases will connect to the shared storage in the same way the other hosts do.

To make this shared storage redundant a second host will be running StorMagic SvSAN and mirror the shared storage. All vSphere hosts using the shared storage will get a second iSCSI target connection which points to the mirror. Remember, an extra license is needed for StorMagic HA (mirror).

In the virtual infrastructure client, you can clearly see how a vSphere host connects to the iSCSI target over two paths, one to the shared storage on the first host, one to the second host. In case of failure the other path will be chosen. In my lab environment the SvSAN iSCSI Targets have IP address 192.168.1.242 and 192.168.1.243.

Installation of StorMagic

To be short, it’s a breeze. On the host that will run StorMagic an OVF is imported and asks for default settings like IP address, appliance name, etc and then it is already up and running. To create shared storage an iSCSI target has to be created. Each iSCSI Target is built from a storage pool and can be the size of that pool or smaller. You could have a 100GB pool and create a 25GB target from it or even multiple targets out of that pool. A storage pool is made out of devices, which are in essential the local disks. This can be virtual disks (VDMKs) or local physical disks.

When using virtual disks (VMDKs) do keep in mind that this will add another virtualization layer and will lead to some performance loss, therefore it is advised to use disks that are not yet used by vSphere as VMFS stores but use empty disks that only will be used as RDM for the SvSAN.

Essentially creating an iSCSI target is done in three steps. First add the available disks into a device, this can be in a Raid5 config or a simple JBOD (Just a Bunch Of Disks) config. Then create a pool from these devices and as last step create a target out of a pool. When creating a mirrored volume, a special wizard is available to combine a pool from the first host with a pool from the second host into a mirrored iSCSI target. This wizard will create the master iSCSI target on one host, create the mirror on the second host and the best feature of this wizard is that all the iSCSI configuration settings for each vSphere host are done by the wizard. It will create the connectors to the master and the mirrored volume on all hosts in the cluster, not only the hosts that run the SvSAN appliance.

Integration with vCenter

An important part of the StorMagic SvSAN is the integration with vCenter. First there is the StorMagic plug-in that very nicely integrates into your virtual infrastructure client by adding an extra tab called “StorMagic” at host level just next to “Storage Views”. By clicking on it all the tools to manage your SvSAN are available and a nice extra is that it is actually just a web page inside the virtual infrastructure client which of course can also be viewed with your favorite browser. I have to admit that although the integration is very nice, I often used the web browser since this is much quicker than the virtual infrastructure client when you have to switch hosts often. Maybe this will change once Windows 7 is officially supported for the virtual infrastructure client and I don’t have to use any hacks to get it running. Definitely not a StorMagic issue.

Secondly there is the “Neutral host” service that is installed on the vCenter Server. This “Neutral host” service is used to determine which SvSAN is still alive and to prevent a split brain scenario in case of network failure. Running this service on the vCenter Server also creates a little ‘gotcha’, because when vCenter is running as a VM using the shared storage offered by SvSAN, it can’t play the role of “Neutral host”. Advice from StorMagic is to not run vCenter in a VM but that to me is politically a big no-no. In my opinion vCenter has to run in a VM on high available storage unless there are really big issues and not doing this just because of a simple service that has to run on the vCenter Server is not enough for me. I discussed this with a StorMagic support engineer and learned that they felt the same and are already looking at a different way of doing this, though no guarantees can be given for the final release.

Failing over to the mirror

When working with techniques like virtual SAN’s an administrator has to really know his stuff. Know which features kick in at what time. For example in my test lab I have three vSphere hosts called esx01, esx02 and esx03. On host esx02 and esx03 the SvSAN appliance is running in a mirrored configuration. On each host there will be one VM running using the shared storage offered by esx02 (mirrored to esx03). What will happen when I pull the power from host esx02?

Without host esx02, there is no SvSAN running anymore on esx02, so the master of the mirror will fail, but the SvSAN on esx03 will take over and keep the shared storage available. Since host esx02 is down, the VM running in esx02 will also be down, but VMware’s HA feature will restart the VM on host esx01 or esx03, using the shared storage on esx03. The VMs on esx01 and esx03 will keep on running, provided the failover to the mirror is handled fast enough by the StorMagic SvSAN. To show how this failover works I posted a video on youtube where I show the configuration and then demo how the VM on esx01 keeps on running, although there is a “freeze” for almost a minute. It feels a bit long but when checking the Windows event logs there is no mention from Windows about losing its disks and the application I had running inside the VM didn’t report any error either.

Features not mentioned

In my review I couldn’t cover all features. Not only because it would make the review way too long but also because I don’t have all the equipment. It is for example possible to connect the SvSAN and the physical SAN’s StorMagic sells (called SM Series) and create mirrors between both these products or manage storage on the physical SAN from within the SvSAN. For a more extensive list of all the features, do visit their website at www.StorMagic.com and get the Product Brief.

Cost and licensing

StorMagic SvSAN comes in 4 editions:

To make your 2 TB SvSAN Highly Available, the add-on feature HA will cost $ 995 per host. HA is included in the 4, 8 and unlimited TB license.

Conclusion

Working with StorMagic’s SvSAN is very simple and straightforward. When I first laid my hands on it, I had some trouble grasping the concept but with excellent help from a StorMagic Support engineer I quickly knew what I wanted and how to get it done. The software is robust and didn’t fail on me even after really stressing it by pulling the plug many times and have mirrors get out of sync and re-sync and break them again. It all kept running without issues. The integration of the GUI into vCenter makes it very easy to use and leaves very few things to wish for. Everything you would want to do is available from the interface and I saw new features coming with every beta release. I like StorMagic’s SvSAN very much and will certainly recommend it.

Is StorMagic suited for your organization or is it cost effective to use StorMagic SvSAN in your environment? This is something I can’t answer for you and you will have to find out yourself. I suggest you download the free 2TB license and give it a try. Take Excel and make some nice calculations to find out if using local disks plus StorMagic is cheaper than buying a small NAS or SAN which can offer same features of which the mirroring is a very strong selling point.

See full post at: StorMagic SvSAN with High Availability mirroring

LAB02: Site Recovery Manager (SRM)

(Drill Instructor Dan Anderson and Tim Meyers)

This Lab was great. When entering class and finding your seat, there was this huge American dude standing in front of class. When taking us through the first powerpoint slides, he was constantly walking from left to right and back again, remembering us that “We’re talking smokin’ craters here!!!” when talking about what happened to our production site. And he made also very sure that we wouldn’t press the “RUN fail over now” button, which would mess up the lab. Yeah, don’t mess with the instructor.

I’ve seen a number of presentations of SRM already, which all turned out to be rather dull slides about what the product could do, but I wanted to get some hands on and this was very well possible during this lab. We got to configure two sites, create the replication and do a test fail over. It gave me a really good idea about what SRM is going to offer to me and even more to the customers I work for.

With SRM you define a number of protection groups that include the items you want to fail over, things like VMs (wow really???), resource pools, datastores, networks. On your primary site you will have your Virtual Center with SRM. SRM can be installed on the VC server or on a separate machine, it can use the same SQL server as your Virtual Center, just make sure it uses a different database, because it will happily clean your VC database on install if you by accident give the VC database name. On your secondary or recovery site, you will also install the same combination (VC and SRM). SRM heavily relies on the storage replication by your SAN. You need to make sure that the LUNs you want to fail over on your primary site are always in sync with the LUNs on the secondary site. It is essential for your recovery.

An important part of the replication and fail over mechanism is the “Storage Replication Adapter”. This is not a hardware like adapter, but a software plug in for SRM that should be created by your storage vendor. What SRM globally does when a fail over is initiated:

- Stop the running VMs on your primary site, because the disaster happening might not be a total disaster yet and you might have VMs that are still running. You want to shut them cleanly first.

- Then SRM tells the Storage Replication Adapter to break the replication of the LUNs from primary to secondary site and make the secondary site LUNs available to the ESX hosts on the secondary site.

- Next it will start the VMs on the ESX hosts on the secondary site, using the replicated LUNs.

Important to know is that SRM will not manage your LUNs, it will not talk to your storage, the Storage Replication Adapter will do this. SRM just kicks off this SRA script and tell it to do its job. This is why your storage vendor should write the Storage Replication Adapter. They know best how to talk to their SAN. So for your its essential to ring your storage vendor tomorrow and ask them when they will be ready for SRM. Without their support, nothing is gonna happen !!! Let them know you want it.

It was a great lab although there was not really anything I hadn’t heard already in earlier presentations, but getting a feel of it was great. I was even able to draw their attention to the fact that the small printed text in green and red, was very difficult to read for a colorblind guy like me. Love to see if they listened and changed this in the final product.

See full post at: LAB02: Site Recovery Manager (SRM)