When visiting customers I noticed that the VMware admin often doesn’t really know how much free space is left on the datastores and wether action is needed to free up space or get extra capacity. When I show them the datastore overview in the VI Client that shows the warnings and alerts for the datastores, it turns out that they don’t often look at this page because it gives too many false alarms. These false alarms happen because the default values right out of the box are set at 75% for a warning to trigger and 85% for an alarm to trigger. In most cases this is much too soon as the next image shows you.

In general I use as a rule of thumb a required free space per datastore of 40GB. Usually these 40GB can accomodate enough snapshot space during nightly backups and storage migrations to prevent the datastore from filling up. Those 40GB free space I would like to monitor through my vCenter alarms which is very easy if all datastores would be equal in size, you could than edit the default alarm to reflect the new % settings. Unfortunately in most environments, not all datastores are equal in size and you would have to change the alarm on each datastore.

There is an easier way, you can create folders and assign alarms to folders and move datastores of the same size into these folders. This will give you a much better view of your datastores and alarms. The alarms you see are now “real” alarms that really require your attention. See how the next image gives you much less alarms and only shows warnings and alarms on datastores that really are below 40GB.

I should warn you though… my friend Duco Jaspars tipped me about this KB article: “Defining alarms on datastore folders can cause vCenter Server 4.x to becomes unresponsive“, although I have not yet ran into this issue with the customers were I implemented this.

In the next version of vCenter I do hope VMware makes a change and enables the use of GB on this alarm instead of just a percentage. You could than just set a GB value and it would be equal to all datastores. In the current situation you have to work with % of the total size of the datastore and especially with large datastores this can be quite a challenge, especially since you can only enter integers and no decimals. See the table below:

As you can see you’ll have to be a little creative with the numbers to get the warnings the way you want them.

See full post at: How to make vCenter datastore alarms useful

End of 2009, beginning of 2010 I put an Iomega StorCenter IX4 to the test and published the results in these two blogposts:
Putting your storage to the test – Part 1 iSCSI on Iomega StorCenter IX4-200D and part 2. Recently Iomega sent me their new StorCenter PX4-300R for testing purpose and I was happy to run some Iometer stress tests on it. I was very anxious to learn if the PX4 would be able to outperform the “good old” IX4.

What’s in the box?

The PX4-300R is a rack mountable model of 1U height and comes with rails and the necessary power and UTP cable. When I read the several websites where the unit is sold, it normally comes without disks. The unit I received had four Hitachi 2.0TB HDS723020BLA642 disks in it configured in RAID5, giving me 5.4TB of disk space. Setup is easy, you plug the UTP cable and power cable, boot the unit and check your dhcp server to see what address it picked up. Then there is a web interface that gives you all the management options you need.

Performance

Despite many features the PX4 holds, I was most interested in the performance of this thing. In the 5.4TB volume I created a big NFS volume and a 500GB iSCSI volume. Connected my vSphere5 ESXi hosts to it and moved my performance test VM on to the volume.  This Windows 2003 VM is running on HW level 7 with 1 OS disk and a 20GB test disk (all VMDKs of course). The first set of test where run with a non-aligned disk, it was formatted in Windows 2003. The aligned tests where run with a disk formatted by Windows 2008 and then reattached to the Windows 2003 server.  For testing I used Iometer with the same config file I ran my IX4 tests, it creates a 4GB dummy file on the disk and performs the reads and writes on that dummy file.

The results

When looking at the results, it clearly shows the PX4-300R is running circles around the IX4. It also shows that for the PX4-300R iSCSI performs a little better than NFS, which was completely different for the IX4 that showed a clear advantage for NFS over iSCSI.

Max Throughput-100%Read

Config IOPS MB/sec PX4 iSCSI Aligned 3.500 114 PX4 iSCSI Non-Aligned 3.486 114 IX4 iSCSI Non-Aligned 1.761 55 PX4 NFS Aligned 3.372 110 PX4 NFS Non-Aligned 3.363 110 IX4 NFS Non-Aligned 3.475 108

Max Throughput-50%Read

Config IOPS MB/sec PX4 iSCSI Aligned 5.267 172 PX4 iSCSI Non-Aligned 5.142 168 IX4 iSCSI Non-Aligned 705 22 PX4 NFS Aligned 4.678 153 PX4 NFS Non-Aligned 4.655 152 IX4 NFS Non-Aligned 684 21

Random-8K-70% Read

Config IOPS MB/sec PX4 iSCSI Aligned 520 4 PX4 iSCSI Non-Aligned 485 3,9 IX4 iSCSI Non-Aligned 64 0,5 PX4 NFS Aligned 145 1,1 PX4 NFS Non-Aligned 129 1 IX4 NFS Non-Aligned 90 0,7

RealLife-60%Rand-65%Read

Config IOPS MB/sec PX4 iSCSI Aligned 645 5,2 PX4 iSCSI Non-Aligned 604 4,9 IX4 iSCSI Non-Aligned 89 0,7 PX4 NFS Aligned 189 1,5 PX4 NFS Non-Aligned 179 1,4 IX4 NFS Non-Aligned 112 0,8

Performance recap

When looking at the charts it is clear that the PX4-300R performs much better than the IX4, sometimes up to 7 times faster. It is good to see that iSCSI and NFS perform almost equal on read performance but surprised to see that NFS can’t keep up with iSCSI in ‘real-world’ tests.

Partition alignment

Running the tests doesn’t take that much time so I decided to also check if partition alignment would really make a difference by formatting one disk with Windows 2003 and one disk using Windows 2008. In Windows 2008 partitions are automatically aligned when formatted. As you can see from the results there is a clear difference between aligned and non-aligned partitions. For heavy read the difference isn’t that big in percentage, but still a 100 IOPS, for the other tests you can see a difference of between 5-7%. That’s free IOPS you get back when making sure you align your partitions.

Conclusion

For the workload I would like to use it for, which is running as my homelab shared storage and store some video’s and foto’s, this would make a great NAS. It holds a number of other features as well, like automatic RSYNC of files to other rsync enabled devices, syncing with cloud services and creating a personal cloud. But I don’t care so much for them personally. What I find important is that it feels fast, almost like I’m working on enterprise storage. Of course it can’t compete with enterprise storage but with the few VMs running in my lab, this is great stuff. The PX4 is more expensive than the IX4, but then again it performs up to 7 times faster in some tests. For a complete overview of the different products, see the Iomega Network Storage overview page.

Test tools

If you want to perform these tests your self, you can download this Gabes-PerfTest.zip which includes iometer and the test configuration file. Would be great if you post your results in the comments.

See full post at: Performance test of Iomega StorCenter PX4-300R

Last evening I had a WebEx session with Tintri in which they told me about their “VM only” storage appliance VMstore and I must admit that I’m impressed with what they have to offer. I have not yet had the opportunity to test this appliance, all info in this blog post is from the WebEx session and documentation provided by Tintri.

What is Tintri VMstore?

It’s an easy to install storage box that comes in only one configuration: 8.5TB of usable data. In the box is a mix of SATA disks of flash disks. The storage is offered to your VMware environment as one big NFS datastore. By moving data back and forth from SATA to flash, VMstore will eliminate storage performance bottlenecks.

What’s under the hood?

The idea of the VMstore is that you no longer carve your storage into different volumes, LUNs, raid-configs, etc. You have just one big volume that is presented as one single datastore to your VMware infrastructure. Having just one single datastore and no LUNs with different performance characteristics, eliminates a lot of storage configuration and management.

What VMstore actually is doing is moving your data from slow rotating disks into super fast flash storage. Moving ALL of your data to flash would be very costly, so they use the flash storage as cache, but a rather big cache. Contrary to other vendors, VMstore uses the flash for read and write, not just read.

To make optimal use of the flash cache, all data that is moved into cache is compressed and deduped. Where other storage vendors use 64K blocks of data to move into cache, VMstore uses only 8K blocks, making it possible to more precisely address the data that should be moved to cache. Tintri says their hitting cache for 97% of all IOPS in production environment.

Of course the flash and 16 SATA disks are protected by RAID, which is a RAID6 level, but for your storage workload, you don’t need different RAID levels.

Auto-alignment

Another technique they are using, which will be announced soon, is auto-alignment. Yes, that is correct; VMstore will automatically align all those VMDK’s that you place on the VMstore. This is a feature I would welcome very much, not even for all the performance gains it would bring to VMstore, but for all those VMs that are still on my to-do list that need re-alignment. Maybe I can ‘test’ a VMstore appliance for a week and storage VMotion all my VMs back and forth between my current storage and the VMstore.

Silver, Gold, Platinum

Since there is just one big volume there is no option to differentiate between Silver, Gold or Platinum performance levels. The only influence you have on the performance of a VM (or single VMDK of a VM) is to pin it to the flash cache. Say a VM with a database running inside, is running for a few days and the most used parts of that VMDK have been moved into flash, you can now pin the VMDK into the flash storage. From now on the data blocks of this VMDK that were in flash, will remain in flash even if in normal use VMstore would start moving those blocks back to the SATA disks. Any extra blocks of this VMDK that are moved from SATA to flash, will also be kept in flash for as long as the VMDK is pinned.

Managing VMstore

The goal was to create storage that would need hardly any management and indeed, all the management you have on the VMstore is decisions on whether to pin or not pin a VM into your flash cache and maybe some day replace a disk.

VMstore has a very intuitive web interface in which you can quickly see how your storage is performing. Again, performance is key here, so the view that shows you how much capacity is left, is telling you about “Performance reserves”.

Seeing latency at VM level

A very powerful tool is seeing the latency at VM or VMDK level. In just a few clicks you can see how your VM is performing. Normally you had to first check at storage level what LUN was having high latency, then find out which VMs are running on it and try to figure out which one is the one with the high latency. No more need for that, just open the VMstore web interface.

Competition

VMstore is aiming at enterprise customers, since you need to have a certain workload on your storage before you’re running into performance bottlenecks caused by storage configurations. A small environment with just a few IOPS and looking for a lot of room to store data is not the customer that will benefit from a VMstore.

To give you an idea what Tintri is aiming for: They claim a VMstore can outperform an EMC Clariion with 250 spindles. Right now Tintri is testing the VMstore with 65/35 R/W workloads and claims to be able to hit a 50.000 IOPS.

A VMstore with 8.5TB storage should sell for around $65,000 – $68,000 list price.

Any drawbacks?

After listening to the presentation and discussing some topics, there remain some points that should be improved I think.

• There is just one controller (dual nic though) for the current box. You can choose for a RJ45 connection or 10Gbit connection, but it is still just one controller that connects the VMstore to your VMware infrastructure. This seems a big point for Enterprise ready storage. The 2^nd generation Vmstore, which will be presented at Vmworld, will contain two controllers.

• Another Enterprise feature that is missing right now and will probably available in the next release is replication. Right now there is no replication at all. Plans for Tintri are to add a-sync replication in the next release.

• In the current release there is no support for VMware VAAI yet, which means especially when offloading storage workloads from the hypervisor to the storage, you would gain some extra performance. However you won’t use VAAI that often during normal operation and the performance bennefit isn’t that big. In vSphere 5 VAAI for NFS will be introduced and Trinti is planning to include this in their next release.

• I’m not sure yet on the concept of just one model: 8.5TB. If you run out of space, you need to buy a second 8.5TB box. Think data growth within the company has to be really huge to justify buying 8.5TB at once.

• And then there of course is the point of real world performance. How will the VMstore handle a lot a random reads and writes? When will workloads be generating cache misses and how will the SATA disks perform in this scenario. We’ll have to wait till we get more real life data from customers.

Overall I very much liked what I saw. Of course I can’t comment on performance at all, but the presentation convinced me that VMstore will lower the cost of implementing and managing your storage, if VM storage is the only storage you need.

The view on latency at VM and VMDK level and the auto-alignment are fantastic. The complete absence of difficult storage management is a big big plus for the VMstore.  I think with the coming new version of the VMstore, it will be a real Enterprise ready device.

See full post at: Tintri VMstore – VM only storage appliance

When trying to update a number of hosts in the ESXi cluster of a customer I first ran into the issue that scanning the host for updates would result in the following error: “Could not scan esx12 for patches”. Searching through the VMware KB I learned that this is probably due to a corrupted scratch partition on the ESXi host, see: KB1020283. This scratch location is, amongst other things, used to store the staged updates for the ESXi host.

Diving into the ESXi host I noticed there was no scratch partition at all. This is strange in my case, since I was sure that after re-installing the hosts a few months ago, my colleague had set the scratch location of all ESXi hosts manually. When double checking this myself I noticed that the scratch location was empty. See: host – configuration tab – advanced settings – ScratchConfig. After kicking my colleague I set the scratch location manually and to double check if it was set correctly I noticed the setting was empty again. While setting the correct location I only got a “Completed successfully” message so it was very strange the setting was empty again.

Since I couldn’t find anything in the VMware KB about not being able to set this value, I tried setting the value through a PowerShell command, but that wouldn’t stick either. My last resort was a reboot which normally feels a bit like being defeated. Nonetheless, after the reboot, the scratch location was still empty, but changing the value worked and sticked. Another reboot is then needed to activate this new scratch location. After this reboot I was able to rescan the host for patches and remediate the host.

Well, this of course solved only half of the problem, because this left me with 8 hosts in this cluster to change the setting. Think you know what happened next, indeed: “PowerShell to the rescue”

VMware KB 1033696 “Creating a persistent scratch location for ESXi” has a good explanation of how to set the scratch location, including the powershell script to do it. The most important commands:

Setting the scratch location:

Get-VMHost -Name esx12 | Set-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation" -Value "/vmfs/volumes/ESX_VMFS00_SCRATCH/esx12/"

Viewing current scratch location setting:

Get-VMHost -Name * | Get-VMHostAdvancedConfiguration -Name "ScratchConfig.ConfiguredScratchLocation"

Viewing current scratch location used:

Get-VMHost -Name * | Get-VMHostAdvancedConfiguration -Name "ScratchConfig.CurrentScratchLocation"

Watch these last two carefully, one is the scratch location that is configured, the other is the location currently in use! If these are not equal the host is probably still waiting for a reboot. The first command only sets the location for one host at a time.

See full post at: Errors during remediation and scratchconfig issues

Zerto http://www.Zerto.com/ (Zero RTO) is a startup company that I first heard about when they were presenting at the Techfield Day in Boston. All information was under embargo, which didn’t allow us delegates to publish about Zerto until June 22^nd and so I had to wait with this post until now. The presentation done by Chen Burshan – Director Product Management and Gil Levonai – VP Products got me really excited about the product and I started playing with the beta. Below is an explanation of what Zerto does and some of my own experiences with it.

Full storage agnostic virtual machine replication

What Zerto promises is full replication of your virtual machines between different sites, in depended of the brand of storage and even on high latency WAN connections.

Zerto is an Israel based company and was founded by Ziv and Oded Kedem who also founded Kashya, which was acquired by EMC and is now the foundation of the EMC RecoverPoint product. After selling their company and learning from the good and bad about traditional BC/ DR solutions, Ziv and Oded used their knowledge to start working on Zerto. When designing Zerto the main thought behind the product was that replication should no longer be done at storage level, but should move to the next level. Zerto will prove today that you can move replication into the hypervisor and still keep all your functions of storage and hypervisor. Zerto is meant to be an enterprise solution, meaning it will be able to handle many virtual machines and work across geographically dispersed data centers.

In short what Zerto will offer is Virtual replication:

• virtualisation aware
• software only
• Tier-1 replication, RTO’s of seconds
• Enterprise class replication
• Purpose built for virtual environments

How does Zerto replicate

On each ESX/ESXi host with virtual machines that need to be replicated a virtual replication appliance will be running on the ESX/ESXi host.

The VRA uses the VMware APIs, which allows it to see all data coming through the IO stack and will then replicate it to the secondary site. It will only see the data in the IO stack and not interfere with the process of writing to disk. Should the VRA fail, ESX/ESXi will continue to operate normally and VMs will not notice any delays.

The virtual appliance makes sure the data is replicated to the virtual appliance on the ESX/ESXi host at the target host. This target can be locally or on a remote site. The data to be replicated will be compressed before sending it to the target using built-in WAN compression & throttling techniques. It can handle WAN disconnects or degradation automatically.

When replicating over WAN, it could take quite some time to finish the first sync between source and target if the source site is a few Tera Bytes in size. To save a lot of time, it is possible to first do a restore from your regular backup on the target site and then have Zerto perform a sync, saving you a lot of time.

The journaling

Source virtual machines can have thin or thick VMDK’s or a physical or virtual RDM and at the target site the target data can be in different places than the source. For example a database server on the source site has its VMDKs on different datastores but grouped in one datastore at the target site. At the target site all received data will be written into the target and a journal will be kept in a VDMK.

In case of a disaster on the source site, it is of course possible to switch to the current status of the target site, otherwise it wouldn’t be a disaster recovery product. However the journaling system used, makes it possible to go back in time and restore the status of one or two hours ago, maybe longer if you have the disk capacity for this. In many cases, real DR is not because of natural disaster, but mostly logical errors, therefore it is ideal to be able to go back further than just the latest status.

The journal holds all data changes and will take up extra disk space that you should plan for when determining how far back in time you want to go. An IO intensive application will of course generate a lot of data and therefore use quite some journaling space.

The journaling system also enables you to test-run a virtual machine on the target site, while still keeping replication running. While starting the replicated virtual machine for test purposes, replication data coming from the source site is still written into the target VMDK. Not only can the replicated virtual machine be tested for correct failover, it can also be used to create a small test environment for other purposes.

When replication fails

As explained above, on the source site every write that goes through the SCSI stack of the virtual machine is split into the memory of the virtual appliance and will then be replicated to the target. However when the link between source and target fails, the virtual appliance will start using a bitmapping technique to remember all the changes to be able to start replication as soon as the link is up again.

To prevent running out of memory in the virtual appliance the bitmap technique will store less detail the longer the link is down. What happens is this:

• Under normal operation the virtual appliance writes changes at the smallest block size to memory. If blocks 2, 5, 9, 13, 17, 20 have changed this is written into memory and removed from memory as soon as these blocks have been replicated.
• However, when the link fails and the number of changes to remember in memory keep growing, the virtual appliance decides to store less detail. In our example, the virtual appliance could decide to only remember that something changed from block 2 to 9 and 13 to 20.
• If the link stays down even longer, the virtual appliance might go even further and now only remembers that block 2 through 20 has changed.

While storing less and less details if the link stays down for a long time, the amount of data to be replicated after the outage will increase.

Zerto Virtual Replication architecture

The architecture of the Zerto Virtual Replication environment is actually rather simple. From the bottom we have the Virtual Replication Appliance. This is a debian based virtual machine that is able to see all SCSI writes by the VM and splits the write into its memory. Then it will try to replicate the blocks over to the Virtual Replication Appliance on the replication site, using TCP port 4005. It will also talk to the Zerto Virtual Manager over TCP port 4005 to keep it updated on all the stuff the VRA is doing.

The Zerto Virtual Manager is a small piece of Windows software that can be installed on the vCenter server, but can also be installed on a separate server, just as long as it can talk to vCenter. The Zerto virtual manager will monitor replication, manage site links, protection groups, protected VMs, etc. It needs to talk to the Zerto Virtual Manager on the target site and does this over TCP port 9081.
The last component is the vSphere Virtual Infrastructure client that runs the Zerto plugin and enables you to manage both protected and replication site from the same interface. It will run over HTTP port 9080.

Installation and protecting VMs in a few simple steps

The installation of the Zerto environment was actually very easy. I will walk you through it with a number of screenshots, just be aware that this isn’t a full step by step walk through.

First you install the Zerto Virtual Manager on a Windows server. This is a very simple next, next, finish process. The only options you have to enter are the IP of your vCenter and the user account to be used to access vCenter.

After installation is complete you run the same install on the replication site. Next step to pair both sites. You can also choose to first install the VRAs but I went for pairing the sites. For this you go into the vSphere VI Client and at cluster or datacenter level click the Zerto tab. In this tab you see a fancy interface with very clear instructions. Either you click the big “Install VRAs” button or you click the “Pair…” button to pair your protection and replication site.  Although the interface looks good, I’m not always happy with it. It requires adobe flash and doesn’t always give the proper response when you click a button. I want to see the button change when I click to prevent me from clicking it multiple times.

Pairing the sites is very easy. Enter the IP address of the Zerto virtual manager of the replication site and click PAIR. That’s it. Next we’ll install the VRA, the virtual machine that captures all the writes and does the actual replication. In the screen where you saw the option to pair the sites, you also have the option to install the VRA. Press the button and the next screen will ask you where to install the VRA to. You’ll see a list of ESX/ESXi hosts, datastores and networks to choose from. The VRA can use a DHCP address or fixed IP. After you selected the desired settings, you push the button “Install selected VRAs” and you wait. In the vCenter task list you’ll the progress of the installation.

Installation progress:

After this the Zerto framework is ready. All we need now is VMs to replicate. Replication is done based on Virtual Protection Groups (VPG). Each protection group contains one or more VM that you want to have the same set of replication and recovery settings. To create a VPG just click the “New VPG” button, add a VM and walk through the recovery settings, things like RPO Threshold, Maintain history, Max Journal size, what host or cluster to replicate to, which datastores to use, which networks, etc, etc.

Once this is done, your replication should start and can be seen in the Zerto tab at cluster level or for the specific VM at VM level.

Summary

As you see, this is a very easy way to setup replication between two sites and the fact that it doesn’t matter what storage you are using will not only be appealing to many SMB customers but also large enterprises. Playing with the beta of Zerto has shown me that this is a very nice product even though I couldn’t really stress test it in my lab environment. But I surely will do some more testing in a larger environment.

Some last notes I’ve written down during the Techfield Day event but haven’t mentioned yet in this post:

• Fully support vMotion, HA, vApp
• Group Policy and configuration
• VSS support
• HA boot orders, vApps, host affinity rules replicated as well
• Version 1 will be all vmware. On road map (months, max 1 yr) other hypervisors. Not sure what other hypervisors. They will be able to replicate between DIFFERENT hypervisors !!!!
• Protect both sites: Symmetrical replication.  You can replicate both ways at the same time, within the same LUN.
• Working on multi-tenancy.

Be aware that the Tech Field Day event is fully sponsored by the companies we visit, including flight and hotel, but we are in no way obligated to write about the sponsors.

See full post at: Zerto Replication and Disaster Recovery the easy way

At the Dutch VMUG, together with my colleague Jos Vanaubel, I presented about how to do cheap disaster recovery using only PowerShell. At the London VMUG I held the same presentation again and promised to upload the scripts to my blog. Well, here it is in PDF format:

The scripts can be downloaded by clicking this link -> Cheap Disaster Recovery Scripts

Before we can failover to DR site, we need the data from the production site. Therefore run the “preparation” scripts weekly or daily, depending on the number of changes. These are your backups. When failover needs to occur, first run failover step 1 and 2, then start the import section. Last step is to boot the vms in failover step 3.

Use the scripts in this order (in the zip file an Excel sheet is included with this info):

A special word of thanks to the guys who helped me with these scripts:
Alan Renouf – http://www.virtu-al.net
Luc Dekens – http://www.lucd.info/
Arnim van Lieshout – http://www.van-lieshout.com
Maish Saidel-Keesing - http://technodrone.blogspot.com/

If you have improvements or additions to the scripts, let me know, I’d be happy to post them here.

See full post at: Cheap disaster recovery

Recently I had the opportunity to review StorMagic’s SvSAN software. Their current release available from their website is only compatible with VMware ESX 3.5, since my home lab is already running VMware vSphere I received the latest beta 4.1.913 which does work with vSphere. In this review I will not write on absolute performance since my home lab will probably hit its limits sooner than the reviewed software does.

What StorMagic does

With StorMagic it is possible to use the local storage (your internal hard disks) of your VMware vSphere hosts as shared iSCSI storage. Other hosts can connect to the targets as if it was a normal iSCSI NAS or SAN. With the free StorMagic version, you can have up to 2Tb of storage managed by StorMagic. Licenses for 4 Tb, 8 Tb and unlimited storage are also available. The advantage of sharing your local storage is that your vSphere environment can now use the extra features like VMware High Availability, VMware DRS and VMware VMotion and you don’t have to spend money on buying a separate iSCSI NAS or SAN.

A disadvantage of having your shared storage on one vSphere host is that this vSphere host will now become your most important part in your Virtual Infrastructure and downtime of the host, is downtime of your storage and therefore downtime for all the VMs on this storage. Features like VMotion and HA are meant to reduce the downtime in your environment and it wouldn’t be wise to now increase your downtime (risk) by putting all your VMs on shared storage that is kept up by just one host.

To reduce this risk StorMagic offers an extra license called StorMagic SvSAN High Availability. With this license the shared storage can be mirrored to a second host. Should one of the hosts fail, the other host will immediately take over and your VMs that are running on the shared storage will keep on running. Well, most of them, because the VMs running in memory of the failing host will of course crash, but VMware HA will be able to get them up and running from the shared storage.

How a virtual infrastructure can be built with StorMagic

In the following figures is shown how local storage is used to create shared iSCSI storage. In the first figure you see the VMware vSphere host which has 4 local disks attached. The first local disk is used for the installation of vSphere, which by default (new in vSphere) creates a local vmfs volume on which the COS (Service Console) is installed. This local VMFS will also be used to install the StorMagic SvSAN appliance on, which needs a 256MB virtual disk and a larger 20GB virtual disk. The remaining three disks will be used to create a Raid5 volume that will be used for as shared storage in StorMagic. There can be two choices in how to use this Raid5 volume. If the host has a Raid5 capable controller, the disks can be joined into a Raid5 volume at hardware level and presented as one disk to StorMagic or StorMagic can group the three local disks into a Raid5 volume and will do the Raid calculations.

The second figure shows how the shared storage is connected to by the other VMware vSphere hosts in the cluster. Not showing in this figure is that the vSphere host that holds the shared storage can and in most cases will connect to the shared storage in the same way the other hosts do.

To make this shared storage redundant a second host will be running StorMagic SvSAN and mirror the shared storage. All vSphere hosts using the shared storage will get a second iSCSI target connection which points to the mirror. Remember, an extra license is needed for StorMagic HA (mirror).

In the virtual infrastructure client, you can clearly see how a vSphere host connects to the iSCSI target over two paths, one to the shared storage on the first host, one to the second host. In case of failure the other path will be chosen. In my lab environment the SvSAN iSCSI Targets have IP address 192.168.1.242 and 192.168.1.243.

Installation of StorMagic

To be short, it’s a breeze. On the host that will run StorMagic an OVF is imported and asks for default settings like IP address, appliance name, etc and then it is already up and running. To create shared storage an iSCSI target has to be created. Each iSCSI Target is built from a storage pool and can be the size of that pool or smaller. You could have a 100GB pool and create a 25GB target from it or even multiple targets out of that pool. A storage pool is made out of devices, which are in essential the local disks. This can be virtual disks (VDMKs) or local physical disks.

When using virtual disks (VMDKs) do keep in mind that this will add another virtualization layer and will lead to some performance loss, therefore it is advised to use disks that are not yet used by vSphere as VMFS stores but use empty disks that only will be used as RDM for the SvSAN.

Essentially creating an iSCSI target is done in three steps. First add the available disks into a device, this can be in a Raid5 config or a simple JBOD (Just a Bunch Of Disks) config. Then create a pool from these devices and as last step create a target out of a pool. When creating a mirrored volume, a special wizard is available to combine a pool from the first host with a pool from the second host into a mirrored iSCSI target. This wizard will create the master iSCSI target on one host, create the mirror on the second host and the best feature of this wizard is that all the iSCSI configuration settings for each vSphere host are done by the wizard. It will create the connectors to the master and the mirrored volume on all hosts in the cluster, not only the hosts that run the SvSAN appliance.

Integration with vCenter

An important part of the StorMagic SvSAN is the integration with vCenter. First there is the StorMagic plug-in that very nicely integrates into your virtual infrastructure client by adding an extra tab called “StorMagic” at host level just next to “Storage Views”. By clicking on it all the tools to manage your SvSAN are available and a nice extra is that it is actually just a web page inside the virtual infrastructure client which of course can also be viewed with your favorite browser. I have to admit that although the integration is very nice, I often used the web browser since this is much quicker than the virtual infrastructure client when you have to switch hosts often. Maybe this will change once Windows 7 is officially supported for the virtual infrastructure client and I don’t have to use any hacks to get it running. Definitely not a StorMagic issue.

Secondly there is the “Neutral host” service that is installed on the vCenter Server. This “Neutral host” service is used to determine which SvSAN is still alive and to prevent a split brain scenario in case of network failure. Running this service on the vCenter Server also creates a little ‘gotcha’, because when vCenter is running as a VM using the shared storage offered by SvSAN, it can’t play the role of “Neutral host”. Advice from StorMagic is to not run vCenter in a VM but that to me is politically a big no-no. In my opinion vCenter has to run in a VM on high available storage unless there are really big issues and not doing this just because of a simple service that has to run on the vCenter Server is not enough for me. I discussed this with a StorMagic support engineer and learned that they felt the same and are already looking at a different way of doing this, though no guarantees can be given for the final release.

Failing over to the mirror

When working with techniques like virtual SAN’s an administrator has to really know his stuff. Know which features kick in at what time. For example in my test lab I have three vSphere hosts called esx01, esx02 and esx03. On host esx02 and esx03 the SvSAN appliance is running in a mirrored configuration. On each host there will be one VM running using the shared storage offered by esx02 (mirrored to esx03). What will happen when I pull the power from host esx02?

Without host esx02, there is no SvSAN running anymore on esx02, so the master of the mirror will fail, but the SvSAN on esx03 will take over and keep the shared storage available. Since host esx02 is down, the VM running in esx02 will also be down, but VMware’s HA feature will restart the VM on host esx01 or esx03, using the shared storage on esx03. The VMs on esx01 and esx03 will keep on running, provided the failover to the mirror is handled fast enough by the StorMagic SvSAN. To show how this failover works I posted a video on youtube where I show the configuration and then demo how the VM on esx01 keeps on running, although there is a “freeze” for almost a minute. It feels a bit long but when checking the Windows event logs there is no mention from Windows about losing its disks and the application I had running inside the VM didn’t report any error either.

Features not mentioned

In my review I couldn’t cover all features. Not only because it would make the review way too long but also because I don’t have all the equipment. It is for example possible to connect the SvSAN and the physical SAN’s StorMagic sells (called SM Series) and create mirrors between both these products or manage storage on the physical SAN from within the SvSAN. For a more extensive list of all the features, do visit their website at www.StorMagic.com and get the Product Brief.

Cost and licensing

StorMagic SvSAN comes in 4 editions:

To make your 2 TB SvSAN Highly Available, the add-on feature HA will cost $ 995 per host. HA is included in the 4, 8 and unlimited TB license.

Conclusion

Working with StorMagic’s SvSAN is very simple and straightforward. When I first laid my hands on it, I had some trouble grasping the concept but with excellent help from a StorMagic Support engineer I quickly knew what I wanted and how to get it done. The software is robust and didn’t fail on me even after really stressing it by pulling the plug many times and have mirrors get out of sync and re-sync and break them again. It all kept running without issues. The integration of the GUI into vCenter makes it very easy to use and leaves very few things to wish for. Everything you would want to do is available from the interface and I saw new features coming with every beta release. I like StorMagic’s SvSAN very much and will certainly recommend it.

Is StorMagic suited for your organization or is it cost effective to use StorMagic SvSAN in your environment? This is something I can’t answer for you and you will have to find out yourself. I suggest you download the free 2TB license and give it a try. Take Excel and make some nice calculations to find out if using local disks plus StorMagic is cheaper than buying a small NAS or SAN which can offer same features of which the mirroring is a very strong selling point.

See full post at: StorMagic SvSAN with High Availability mirroring

I frequently read postings on different forums about what the best LUN size is and I thought I’d write a post about it on how I usually do it.

First of all, there is no ideal LUN size that suites all environments. But I think there is a general calculation you can use to come to the LUN size suited for your environment. And second, there is no technical difference between small or large LUNs. For the read/write performance it doesn’t matter if your VMDK is on a 100Gb LUN or 2TB LUN, it’s the total load on the LUN that matters.

The bases of my rule of thumb is that a LUN should hold no more then 30 VMDKs, more VMDKs could impact performance because of disk queuing. Keeping those 30 VMDKs in mind, you should then check what your average disk size is and please look at it by using your common sense. With that I mean, if you have 100 VMs that use around 15Gb each and 2 VMs with 1,5Tb, please leave those 2 out of your calculation. These exceptions you can handle later on.

Ok, you have determined the average disk size, let’s say 12Gb. Now multiply this by that magic 30 and you see that you would need 360Gb per LUN to accommodate 30 VMDKs. But that’s not all; we also need some spare room for VM swap space and for snapshots. Normally the VMs I use have 1Gb or 2Gb RAM assigned, which would give a 1-2Gb swap space for the VM. Because most VMs have more than one VMDK, I think its safe to state that 30Gb swap space per LUN is sufficient.

Most tricky part is the spare room you should reserve for snapshots. I try to keep snapshots active as short as possible. A week old snapshot is really long in my opinion. Of course, there can be reasons that you need it, but normal operations would not require running a snapshot for such a long time. So how big should the room for snapshot be? Let’s put it at 15%.

So to summarize and build the formula:

30 x (your average disk size) + 30Gb VM swap + 15% of (30 x your average disk size) = calculated LUN size.

And to put the cherry on top, take your calculated LUN size and round it up to the next “handy” number. For example, 444Gb I would round up to 500Gb, 689Gb I would round up to 750Gb and so.

With this formula I think you can make a basic calculation and get an idea of what COULD suite your environment. Its no hard rule, it something you have to feel comfortable with and maybe after some time you would tweak this to your own experiences.

Would love to hear your thoughts and comments on this!!!

See full post at: Storage: How to size your LUNs?