Today I visited the VKernel stand where I met Bryan Semple. He had some great news. Next week VKernel will release a new free tool called the VKernel vScope Explorer which will be great for every vSphere admin. It will give you the ability to show any problems in your environment for up to 8000 VMs. Although this free version is ment to be an overview and will not show you everything in deep technical details, it does give you very good info on the problems a VM is having. VKernel vScopre Explorer is built on the same engine of the full product and will give you similar results, it is almost a light version of the new VMware vCOMS only from VKernel instead of VMware.

Check their website for the official release.

See full post at: Free vScope Explorer from VKernel

Zerto http://www.Zerto.com/ (Zero RTO) is a startup company that I first heard about when they were presenting at the Techfield Day in Boston. All information was under embargo, which didn’t allow us delegates to publish about Zerto until June 22^nd and so I had to wait with this post until now. The presentation done by Chen Burshan – Director Product Management and Gil Levonai – VP Products got me really excited about the product and I started playing with the beta. Below is an explanation of what Zerto does and some of my own experiences with it.

Full storage agnostic virtual machine replication

What Zerto promises is full replication of your virtual machines between different sites, in depended of the brand of storage and even on high latency WAN connections.

Zerto is an Israel based company and was founded by Ziv and Oded Kedem who also founded Kashya, which was acquired by EMC and is now the foundation of the EMC RecoverPoint product. After selling their company and learning from the good and bad about traditional BC/ DR solutions, Ziv and Oded used their knowledge to start working on Zerto. When designing Zerto the main thought behind the product was that replication should no longer be done at storage level, but should move to the next level. Zerto will prove today that you can move replication into the hypervisor and still keep all your functions of storage and hypervisor. Zerto is meant to be an enterprise solution, meaning it will be able to handle many virtual machines and work across geographically dispersed data centers.

In short what Zerto will offer is Virtual replication:

• virtualisation aware
• software only
• Tier-1 replication, RTO’s of seconds
• Enterprise class replication
• Purpose built for virtual environments

How does Zerto replicate

On each ESX/ESXi host with virtual machines that need to be replicated a virtual replication appliance will be running on the ESX/ESXi host.

The VRA uses the VMware APIs, which allows it to see all data coming through the IO stack and will then replicate it to the secondary site. It will only see the data in the IO stack and not interfere with the process of writing to disk. Should the VRA fail, ESX/ESXi will continue to operate normally and VMs will not notice any delays.

The virtual appliance makes sure the data is replicated to the virtual appliance on the ESX/ESXi host at the target host. This target can be locally or on a remote site. The data to be replicated will be compressed before sending it to the target using built-in WAN compression & throttling techniques. It can handle WAN disconnects or degradation automatically.

When replicating over WAN, it could take quite some time to finish the first sync between source and target if the source site is a few Tera Bytes in size. To save a lot of time, it is possible to first do a restore from your regular backup on the target site and then have Zerto perform a sync, saving you a lot of time.

The journaling

Source virtual machines can have thin or thick VMDK’s or a physical or virtual RDM and at the target site the target data can be in different places than the source. For example a database server on the source site has its VMDKs on different datastores but grouped in one datastore at the target site. At the target site all received data will be written into the target and a journal will be kept in a VDMK.

In case of a disaster on the source site, it is of course possible to switch to the current status of the target site, otherwise it wouldn’t be a disaster recovery product. However the journaling system used, makes it possible to go back in time and restore the status of one or two hours ago, maybe longer if you have the disk capacity for this. In many cases, real DR is not because of natural disaster, but mostly logical errors, therefore it is ideal to be able to go back further than just the latest status.

The journal holds all data changes and will take up extra disk space that you should plan for when determining how far back in time you want to go. An IO intensive application will of course generate a lot of data and therefore use quite some journaling space.

The journaling system also enables you to test-run a virtual machine on the target site, while still keeping replication running. While starting the replicated virtual machine for test purposes, replication data coming from the source site is still written into the target VMDK. Not only can the replicated virtual machine be tested for correct failover, it can also be used to create a small test environment for other purposes.

When replication fails

As explained above, on the source site every write that goes through the SCSI stack of the virtual machine is split into the memory of the virtual appliance and will then be replicated to the target. However when the link between source and target fails, the virtual appliance will start using a bitmapping technique to remember all the changes to be able to start replication as soon as the link is up again.

To prevent running out of memory in the virtual appliance the bitmap technique will store less detail the longer the link is down. What happens is this:

• Under normal operation the virtual appliance writes changes at the smallest block size to memory. If blocks 2, 5, 9, 13, 17, 20 have changed this is written into memory and removed from memory as soon as these blocks have been replicated.
• However, when the link fails and the number of changes to remember in memory keep growing, the virtual appliance decides to store less detail. In our example, the virtual appliance could decide to only remember that something changed from block 2 to 9 and 13 to 20.
• If the link stays down even longer, the virtual appliance might go even further and now only remembers that block 2 through 20 has changed.

While storing less and less details if the link stays down for a long time, the amount of data to be replicated after the outage will increase.

Zerto Virtual Replication architecture

The architecture of the Zerto Virtual Replication environment is actually rather simple. From the bottom we have the Virtual Replication Appliance. This is a debian based virtual machine that is able to see all SCSI writes by the VM and splits the write into its memory. Then it will try to replicate the blocks over to the Virtual Replication Appliance on the replication site, using TCP port 4005. It will also talk to the Zerto Virtual Manager over TCP port 4005 to keep it updated on all the stuff the VRA is doing.

The Zerto Virtual Manager is a small piece of Windows software that can be installed on the vCenter server, but can also be installed on a separate server, just as long as it can talk to vCenter. The Zerto virtual manager will monitor replication, manage site links, protection groups, protected VMs, etc. It needs to talk to the Zerto Virtual Manager on the target site and does this over TCP port 9081.
The last component is the vSphere Virtual Infrastructure client that runs the Zerto plugin and enables you to manage both protected and replication site from the same interface. It will run over HTTP port 9080.

Installation and protecting VMs in a few simple steps

The installation of the Zerto environment was actually very easy. I will walk you through it with a number of screenshots, just be aware that this isn’t a full step by step walk through.

First you install the Zerto Virtual Manager on a Windows server. This is a very simple next, next, finish process. The only options you have to enter are the IP of your vCenter and the user account to be used to access vCenter.

After installation is complete you run the same install on the replication site. Next step to pair both sites. You can also choose to first install the VRAs but I went for pairing the sites. For this you go into the vSphere VI Client and at cluster or datacenter level click the Zerto tab. In this tab you see a fancy interface with very clear instructions. Either you click the big “Install VRAs” button or you click the “Pair…” button to pair your protection and replication site.  Although the interface looks good, I’m not always happy with it. It requires adobe flash and doesn’t always give the proper response when you click a button. I want to see the button change when I click to prevent me from clicking it multiple times.

Pairing the sites is very easy. Enter the IP address of the Zerto virtual manager of the replication site and click PAIR. That’s it. Next we’ll install the VRA, the virtual machine that captures all the writes and does the actual replication. In the screen where you saw the option to pair the sites, you also have the option to install the VRA. Press the button and the next screen will ask you where to install the VRA to. You’ll see a list of ESX/ESXi hosts, datastores and networks to choose from. The VRA can use a DHCP address or fixed IP. After you selected the desired settings, you push the button “Install selected VRAs” and you wait. In the vCenter task list you’ll the progress of the installation.

Installation progress:

After this the Zerto framework is ready. All we need now is VMs to replicate. Replication is done based on Virtual Protection Groups (VPG). Each protection group contains one or more VM that you want to have the same set of replication and recovery settings. To create a VPG just click the “New VPG” button, add a VM and walk through the recovery settings, things like RPO Threshold, Maintain history, Max Journal size, what host or cluster to replicate to, which datastores to use, which networks, etc, etc.

Once this is done, your replication should start and can be seen in the Zerto tab at cluster level or for the specific VM at VM level.

Summary

As you see, this is a very easy way to setup replication between two sites and the fact that it doesn’t matter what storage you are using will not only be appealing to many SMB customers but also large enterprises. Playing with the beta of Zerto has shown me that this is a very nice product even though I couldn’t really stress test it in my lab environment. But I surely will do some more testing in a larger environment.

Some last notes I’ve written down during the Techfield Day event but haven’t mentioned yet in this post:

• Fully support vMotion, HA, vApp
• Group Policy and configuration
• VSS support
• HA boot orders, vApps, host affinity rules replicated as well
• Version 1 will be all vmware. On road map (months, max 1 yr) other hypervisors. Not sure what other hypervisors. They will be able to replicate between DIFFERENT hypervisors !!!!
• Protect both sites: Symmetrical replication.  You can replicate both ways at the same time, within the same LUN.
• Working on multi-tenancy.

Be aware that the Tech Field Day event is fully sponsored by the companies we visit, including flight and hotel, but we are in no way obligated to write about the sponsors.

See full post at: Zerto Replication and Disaster Recovery the easy way

At Techfield day, VMware presented to us delegates their Mobile Virtualization Platform (MVP) and as expected all these geeks where very interested to see what VMware had to offer. Unfortunately the overall feeling was that the product as demonstrated to us was a bit disappointing. Of course it is still no released product, but even then there are some points to worry about, but first let’s have a look at what VMware MVP really is.

Taken from the VMware MVP website ( http://www.vmware.com/mobile):

“Employees want to use their personal smartphone for work and are pushing IT to support those devices. This trend—sometimes called “Consumerization of IT”—challenges IT requirements for security, compliance and ease of management. VMware MVP enables enterprises to embrace this trend, by allowing IT to safely support employee owned devices. With VMware MVP, enterprises can get the security and ease of management they require, while reducing CAPEX.With VMware MVP, a personal profile and a corporate profile can securely and simultaneously run on the same device in isolated containers. Corporate applications and data are securely isolated from an employee’s personal profile.With VMware MVP, employees can connect their own mobile devices to the corporate network from a corporate profile that is provisioned and managed by the corporate IT group.”

Architecture

First important message to all those hypervisor geeks that thought of running multiple VMs on their smartphones giving them total freedom, that is not what VMware MVP is aiming for. VMware MVP want’s to create a secure environment in which you can run your business applications and uses a hybride hypervisor model to create this environment. VMware MVP will create a single secure VM environment on top of the smartphone’s native OS by inserting a MVP module into the kernel of the native OS, called the MVPkm.

See this blog for more details on how exactly MVPkm hooks into the native OS and an in-depth explanation of the pro’s and con’s of this choice. http://www.ok-labs.com/blog/entry/vmware-mvp-how-it-works/ (Author: Gernot Heiser).

VMware MVP

The take away from Gernot Heiser’s post is this:

• The hybride hypervisor model is better than the type 2 hypervisor performance wise, but it is no type 1 hypervisor either since it has no exclusive control of the hardware.
• The hybride approach’s biggest disadvantage is that it creates a new attack surface and adds nothing to the security of the guest apps.

If the architecture doesn’t provide the additional security how do we get a secure environment then? Well, VMware MVP will only run OS’es inside the VM that have been provided and adjusted (secured) by VMware. You cannot just download the latest build of the smartphone’s OS and run it inside the MVP VM.

The plan that VMware has is that the enterprise admin will download the image supplied by VMware, add the corporate applications and then deploy them to user’s phone. This deployed VM is locked and the user cannot install any other apps into it. Well actually, the admin is supposed to lock this image and prevent further installation of new apps by the user. He could leave the image unlocked but that would defeat the whole purpose of having a ‘secure image’ with approved apps only.

Biggest drawback of this approach is that as a corporate admin you will have to wait with updates not until the vendor releases them, but for VMware to receive that update and adopt it to fit in the VMware MVP. Questions that where asked but could not answered at the Techfield Day demo:

• Will VMware add extra security to the image supplied by the vendor?
• What is VMware aiming at for the update cycles, how fast are they planning to release their image after the vendor releases it? Will they only release big updates (3.3 to 3.4 upgrade for example or also 3.4 to 3.4.1 updates?
• Will Apple allow running iOS inside a VM on top of android OS? And the same question of course also goes for RIM (Black Berry). I wonder what their motivation is in working together with VMware in this as they all have their own solutions to secure their devices. Specifically with Apple I could imagine that they want their iOS running on other hardware device than an iPhone, since “user experience” would be different than on a real iPhone.

Current status

It was at VMworld 2008 that VMware first talked about their ideas on Mobile Virtualization Platform and got a lot of people very excited. Now we’re 2.5 years down the road and what have we got?

• MVP will only run on android right now and inside the VM it will also run android only
• MVP will only support a single SIM, which means you still have to walk around with two smartphones if you want to keep a separate private and business number
• The management framework that would deploy these images to the users smartphones could not be demoed yet since it is not ready and no comment could be made about  what stage it is in.

All these unanswered questions give a lot of concerns. Personally I think VMware shouldn’t even try to release it if only Android on Android is supported. They should at least have one other OS they can run it on and run inside the VM to prevent that after release, all that is remembered by people who evaluate the product that it only runs on android. It will be very hard to invite those people again to evaluate the product when in a big update a new OS is added to the list.

My biggest concern is that the security gap they are trying to fix won’t be fixed by adding a hypervisor into the equation, but should be done by really securing those business apps. There are some real good management tools available that can protect a smartphone, lock down applications and secure your e-mail. Have a look at GOOD ( http://www.Good.com) for example. Others have already mentioned it would be better to go down the road of application virtualization instead of a hypervisor.

I’m wondering how long it will take for VMware to finally release a version 1 product and if it will ever be released. What they have shown us now wasn’t much and looked like a lot of work still needs to be done. I see a parallel here with the VMware Client Hypervisor (CVP) that eventually got canceled and merged into VMware View’s offline mode. Maybe, eventually VMware MVP will be merged with ThinApp and make mobile application virtualization possible.

Be aware that the Tech Field Day event is fully sponsored by the companies we visit, including flight and hotel, but we are in no way obligated to write about the sponsors.

See full post at: Thoughts on VMware Mobile Virtualization Platform

On Friday the Field Tech Day delegates of the Gestalt IT event paid a visit to Cisco where they were treated on a very good session on the VCE Vblock. This session was brought to us by “the other” Scott Lowe and Ed Saipetch. Apart from doing a very good presentation they also showed they could fight like lions against the comments of the delegates, resulting in the best session of this Tech Field Day – Boston 2010. Although Vblock is a great piece of machinery the feeling amongst the delegates was almost unanimous about Vblock being very hard to sell and offer little extra value over a self-built configuration using the same components. Writing this blog post took me quite some time reading several guides to better understand the Vblock and during this investigation I changed my mind on whether Vblock is a good or bad idea several times. I hope the following helps you make up your mind.

Vblock, what is it?

Let me explain a bit more on what the Vblock really is, actually it’s fairly simple: the Vblock is a complete Virtual Infrastructure package built on EMC Clariion CX4 series or the EMC Symmetrix V-Max for the storage layer, connected over Cisco Nexus 1000V and Cisco Multilayer Directional Switches (MDS) to a Cisco Unified Computing Systems (UCS) blade center running VMware vSphere 4 . By using a fixed combination of components VCE (a consortium of VMware, Cisco and EMC) is able to guarantee performance, capacity and availability SLA for a known number of virtual machines.

(The Cisco Nexus 7000 in the diagram is not a Vblock component. EMC Ionix is optional and available at additional cost.)

The unique selling points of a Vblock according to VCE are:

• Pretested
• Fully Integrated
• Ready to Go
• Ready to Grow

Vblock Type 1 and Vblock type 2

A Vblock comes in two flavors, type 1 and type 2. Scott Lowe did mention that a type 0 is being constructed at this moment but specs have not been made available yet. When asking my good friend Google, he (or she) told me to expect the smaller type 0 in summer 2010, but that is all unconfirmed info.

A type 1 Vblock will be able to host up to a 1000 VMs and a type 2 Vblock will host up to 2000 VMs. If you hit the limits of a Vblock you just extend your Vblock with another Vblock, which can be of any type (1 or 2). All these Vblocks together can be managed as one. When deciding on what size of Vblock you need it is important to NOT think about RAM, CPU cycles or IOPS needed, but only think of number of VMs you want to run (more not this later) and buy Vblocks accordingly.

No upgrades

Now I do see your head frown on “Buy another Vblock if I hit the limits? Can’t I just upgrade a Vblock with more memory for example?” Well, technically you can; you could add more blades or add more memory to your blades but then it isn’t a Vblock anymore and you lose your one point of support. You don’t lose all support of course, since each component still has full support by either EMC, Cisco or VMware, but the VCE consortium just won’t be able to support you anymore. There are only very limited changes you’re allowed to make to the system to stay within the supported configuration.

This wouldn’t be that much of an issue if resources would have been used to their max, but the maximum values for the UCS Vblock blades are not even near their hardware limits when looking at the Vblock 1 config. Have a look at the exact specs according to the “Vblock Infrastructure Packages Reference Architecture” guide:

A UCS B-200 M1 blade can hold 96GB RAM according to the specs but will only be filled to half their maximum possible configuration in a type 1 Vblock which always uses 6 blades per chassis at only half of their possible configuration max plus 2 blades maxed out at 96 GB RAM. Why not max out those first 6 blades as well? If I start with the minimum config of 2 chassis of each 8 blades (6+2), a ratio of 1:4 VMs per core, would max out at 512 VMs (32 VMs per blade). Now when I go over those 512 VMs, according to the Vblock principle I would need to add another chassis. Such a chassis would then give me 256 VMs extra. However, when working with 96GB blades instead of the 48GB blades, I could run up to 768 VMs on the first to chassis but this is no longer a supported configuration.

The balance

This is where the balanced design of the Vblock comes into play. According VCE the supported configurations guarantee there is always a good balance between CPU, RAM and IOPS. An increase RAM will enable you to run more VMs but will also ask for more CPU cycles and demand more IOPS from your storage system. With a Vblock each type or combination of types will always make sure this balance remains intact. Sounds good. The Vblock’s fixed configuration will protect you from creating a bottleneck when changing a configuration.

The bottleneck

What I don’t understand though is where the bottleneck is in a Vblock type 1 to use only 48GB? When starting with 2 chassis there is plenty of memory that could be added before adding a 3^rd chassis. CPU shouldn’t be the problem, since the Vblock type 2 blades are the same B-200 blades, all running 96GB RAM and are able to host more VMs per blade than the Vblock type 1.  Would storage be the bottleneck? Actually, I doubt that, since adding a 3^rd or 4^th chassis would put more VMs on the storage and ask more IOPS from the storage, which the Vblock can deliver according to the specs. Then why would the balance be gone when adding more memory? I have no answer on that, I can only say that where 4 chassis with each 6x 48GB + 2x 96GB blades will give me 1920GB RAM, a non-supported config with 3 chassis of 8x 96GB blades would give me 2304GB RAM and thus save me buying that 4rd chassis.

From a VMware view, there is the question on how the vCenter cluster design will be made, will a two chassis configuration span one cluster or will each chassis be a cluster of its own? Both scenarios have their potential design problems. To learn more on this read Duncan Epping’s “HA Deep dive” Section on HA-Admission control. http://www.yellow-bricks.com/vmware-high-availability-deepdiv/#HA-admission.

When buying a Rolls, you don’t ask for the price

But maybe I’m too focused on the details. A Vblock can hold a lot of VMs and when buying capacity for that many VMs you don’t care about these details, like when buying a Rolls Royce. If you have to ask for the price, you can’t afford one. I’m convinced that the Vblock in both type 1 and type 2 is a carefully selected configuration which is able to deliver really great performance, but it is just not for us mortals to buy. The Vblock will be bought by CEO’s of big companies during dinner with the sales people from VCE, where all they discuss is how many VMs they want to run. The sales man from VCE then says: “Sure, 15.000 VMs is no problem for us, just sign on the dotted line to order 2 Vblocks type 2. Now, what’s for desert?”

Disclaimer: My trip, hotel and food during this event is paid for by the sponsors of the event. However, I’m not obliged to blog about it or only write positive posts

Links to other Tech Field Day posts on the Vblock:

• StorageNerve: GestaltIT Tech Field Day 2010: VBlocks Presentation
• EtherealMind.com: VCE vBlock – Look Mum! No cables

See full post at: [Gestalt] vBlock, great product, just not for you

As you may have seen on the Gestalt IT website, I have been invited to join this great event and today I’m on my way to Boston. I’m looking forward to meeting all the delegates some I’ve know for some time, others I will be meeting for the first time this week. The program that Stephen Foskett has setup for us is intense, we’ll be hearing a lot of new things and discussing with the bright minds that came up with these great products. We will be paying a visit to Cisco, EMC, Data Robotics, Hewlett-Packard and last but not least vkernel. Should you have any questions about their products or their vision you want me to ask, feel free to post them in the comment section.

On my blog and on twitter I will try to cover the event and talk about all the interesting things that are going to be told. On twitter the hashtag will be #TechFieldDate and on my blog I will mark the post title with [Gestalt].

Disclaimer: My trip, hotel and food during this event is paid for by the sponsors of the event. However, I’m not obliged to blog about it or only write positive posts.

See full post at: [Gestalt] On my way to Gestalt IT event