At the Dutch VMUG event 2011 I gave a presentation on how to check your VMware environment to make sure it is healthy. When creating the presentation I had a lot of doubts because I was afraid everyone would think these points were very obvious. But on the other hand, when visiting customers and doing these health checks for them, I found a lot of those “obvious” issues in their environment. I decided to stick to my plan and test the audience and it turned out they were very happy with my presentation and I saw a lot of people in the audience taking notes. The replies afterwards also showed that for many people there were a lot of eye-openers in this presentation. I therefore decided to convert the power point presentation into this blogpost and hope my readers find it a valuable health check.

In the top menu bar of this blog you’ll find the “Health Check” section. There you can find the various pages that together give you a complete health check.

See full post at: VMware vSphere Health Check

Last evening I had a WebEx session with Tintri in which they told me about their “VM only” storage appliance VMstore and I must admit that I’m impressed with what they have to offer. I have not yet had the opportunity to test this appliance, all info in this blog post is from the WebEx session and documentation provided by Tintri.

What is Tintri VMstore?

It’s an easy to install storage box that comes in only one configuration: 8.5TB of usable data. In the box is a mix of SATA disks of flash disks. The storage is offered to your VMware environment as one big NFS datastore. By moving data back and forth from SATA to flash, VMstore will eliminate storage performance bottlenecks.

What’s under the hood?

The idea of the VMstore is that you no longer carve your storage into different volumes, LUNs, raid-configs, etc. You have just one big volume that is presented as one single datastore to your VMware infrastructure. Having just one single datastore and no LUNs with different performance characteristics, eliminates a lot of storage configuration and management.

What VMstore actually is doing is moving your data from slow rotating disks into super fast flash storage. Moving ALL of your data to flash would be very costly, so they use the flash storage as cache, but a rather big cache. Contrary to other vendors, VMstore uses the flash for read and write, not just read.

To make optimal use of the flash cache, all data that is moved into cache is compressed and deduped. Where other storage vendors use 64K blocks of data to move into cache, VMstore uses only 8K blocks, making it possible to more precisely address the data that should be moved to cache. Tintri says their hitting cache for 97% of all IOPS in production environment.

Of course the flash and 16 SATA disks are protected by RAID, which is a RAID6 level, but for your storage workload, you don’t need different RAID levels.

Auto-alignment

Another technique they are using, which will be announced soon, is auto-alignment. Yes, that is correct; VMstore will automatically align all those VMDK’s that you place on the VMstore. This is a feature I would welcome very much, not even for all the performance gains it would bring to VMstore, but for all those VMs that are still on my to-do list that need re-alignment. Maybe I can ‘test’ a VMstore appliance for a week and storage VMotion all my VMs back and forth between my current storage and the VMstore.

Silver, Gold, Platinum

Since there is just one big volume there is no option to differentiate between Silver, Gold or Platinum performance levels. The only influence you have on the performance of a VM (or single VMDK of a VM) is to pin it to the flash cache. Say a VM with a database running inside, is running for a few days and the most used parts of that VMDK have been moved into flash, you can now pin the VMDK into the flash storage. From now on the data blocks of this VMDK that were in flash, will remain in flash even if in normal use VMstore would start moving those blocks back to the SATA disks. Any extra blocks of this VMDK that are moved from SATA to flash, will also be kept in flash for as long as the VMDK is pinned.

Managing VMstore

The goal was to create storage that would need hardly any management and indeed, all the management you have on the VMstore is decisions on whether to pin or not pin a VM into your flash cache and maybe some day replace a disk.

VMstore has a very intuitive web interface in which you can quickly see how your storage is performing. Again, performance is key here, so the view that shows you how much capacity is left, is telling you about “Performance reserves”.

Seeing latency at VM level

A very powerful tool is seeing the latency at VM or VMDK level. In just a few clicks you can see how your VM is performing. Normally you had to first check at storage level what LUN was having high latency, then find out which VMs are running on it and try to figure out which one is the one with the high latency. No more need for that, just open the VMstore web interface.

Competition

VMstore is aiming at enterprise customers, since you need to have a certain workload on your storage before you’re running into performance bottlenecks caused by storage configurations. A small environment with just a few IOPS and looking for a lot of room to store data is not the customer that will benefit from a VMstore.

To give you an idea what Tintri is aiming for: They claim a VMstore can outperform an EMC Clariion with 250 spindles. Right now Tintri is testing the VMstore with 65/35 R/W workloads and claims to be able to hit a 50.000 IOPS.

A VMstore with 8.5TB storage should sell for around $65,000 – $68,000 list price.

Any drawbacks?

After listening to the presentation and discussing some topics, there remain some points that should be improved I think.

• There is just one controller (dual nic though) for the current box. You can choose for a RJ45 connection or 10Gbit connection, but it is still just one controller that connects the VMstore to your VMware infrastructure. This seems a big point for Enterprise ready storage. The 2^nd generation Vmstore, which will be presented at Vmworld, will contain two controllers.

• Another Enterprise feature that is missing right now and will probably available in the next release is replication. Right now there is no replication at all. Plans for Tintri are to add a-sync replication in the next release.

• In the current release there is no support for VMware VAAI yet, which means especially when offloading storage workloads from the hypervisor to the storage, you would gain some extra performance. However you won’t use VAAI that often during normal operation and the performance bennefit isn’t that big. In vSphere 5 VAAI for NFS will be introduced and Trinti is planning to include this in their next release.

• I’m not sure yet on the concept of just one model: 8.5TB. If you run out of space, you need to buy a second 8.5TB box. Think data growth within the company has to be really huge to justify buying 8.5TB at once.

• And then there of course is the point of real world performance. How will the VMstore handle a lot a random reads and writes? When will workloads be generating cache misses and how will the SATA disks perform in this scenario. We’ll have to wait till we get more real life data from customers.

Overall I very much liked what I saw. Of course I can’t comment on performance at all, but the presentation convinced me that VMstore will lower the cost of implementing and managing your storage, if VM storage is the only storage you need.

The view on latency at VM and VMDK level and the auto-alignment are fantastic. The complete absence of difficult storage management is a big big plus for the VMstore.  I think with the coming new version of the VMstore, it will be a real Enterprise ready device.

See full post at: Tintri VMstore – VM only storage appliance

When you build your virtual infrastructure with only ESXi hosts that you also lock down for security reasons, you might be in for a little surprise when you want to get your VI up and running again after major maintenance or a failure. First thing to do after the virtual infrastructure has been down, is to get vCenter up and running again. In a previous post “How to quickly recover from disaster” I already explained the idea of running vCenter always on the first host in your cluster. In case of failure you don’t have to search where DRS left your vCenter VM, you just connect the VI Client to the first host and start the vCenter VM. Don’t forget you need your Active Directory and SQL database before starting vCenter.

With an all ESXi environment and locked down hosts, you cannot use the VI Client to connect and start the necessary VMs, at least that is what many think, but this isn’t completely true. The Locked down mode does NOT prevent direct VI Client connections as many think, it does however prevent direct VI Client connetions made with the root-user account. The same goes for PowerCLI, vCLI, vMA, or any of the other public APIs. In locked down mode the root user has no direct access. You can however create an extra user on your ESXi install and assign this user administrator rights. Then, after enabling locked down mode, you can still make a direct VI Client connection to your ESXi box and perform some admin tasks like starting VMs.

Another option would be to just get access to the console of the ESXi host using ILO, KVM, DRAC or similar techniques and disable lockdown mode. After disabling lockdown mode, you can then again make root access using the VI Client.

To summarize:
- Lockdown mode for ESXi does prevent root access using VI Client, PowerCLI, vMA, API’s etc.
- Lockdown mode for ESXi does NOT prevent other users accessing the ESXi host using above mentioned tools. Just be sure to first create this user.
- Procedure in an enterprise to create that local user on all ESXi hosts, would be to use (for example) PowerShell to create that admin user and then enable the lockdown mode.

See full post at: About ESXi lockdown mode

Every now and then, you get that same talk over and over again about ESX security. Bottomline always is that they think ESX (or virtualization at a whole) is not secure enough, but they can never explain why it isn’t or are just biased because of old security rules and thoughts.The only way to try to convince them is to have a set of whitepapers and atricles at hand that give some good insight on ESX security.

Below is a number of links that can help you in this situation. Should you have some good links to share, please add them in the comments.

VMware Infrastructure Earns Common Criteria EAL4+ Certification http://blogs.vmware.com/security/2008/06/vmware-infrastr.html This post talks about the very high Certification VMware has earned. Although it is only for ESX 3.0.2, one can be assured that ESX is designed with security high on the priority list.

Good starting point on your search for security related docs: http://viops.vmware.com/home/community/security.

Two posts about the “Blue Pill” infection that had everybody scared, but turned out to be not so scarry: http://www.vmware.com/vmtn/blog/2006/08/#bluepillpoppers and http://www.vmware.com/vmtn/blog/2006/08/#slashdot_followup.

Also read this post by Christofer Hoff, an open letter to Joanna Rutkowska who researched the Blue Pill exploit. Joanna’s response can be found here http://theinvisiblethings.blogspot.com/2008/04/research-obfuscated.html. And a last response by Chris: http://rationalsecurity.typepad.com/blog/2008/04/perception-vs-v.html

Great doc: 20 Questions from security professionals. Steve Chambers (VMware) has put the 20 most asked questions from security professionals in writing and answers them.

This document discusses the architecture of VMware Infrastructure 3, focusing on the security aspects of the design: Security Design of the VMware Infrastructure 3 Architecture.

Do have a look at VMware’s Security Center.

Some articles about ESX in the DMZ: DMZ virtualization with VI3 and All Your Virtualized Storage Are Belong To Us.

See full post at: So you have that talk with your security officer again…

Today I stumbled upon a new product from Montego Networks, called HyperSwitchTM. With HyperSwitchTM it will be possible to integrate virtual network policies and access control with a high-availability virtual security switch. Some of the features:

• Virtual machine partitioning, 802.1Q VLANs and port-based security
• Secure VM-to-VM communications
• High-availability, policy-based switching
• Virtual network discovery, visibility and rogue detection
• Policy-based access control and auditing (L2-L4, Identity and Content Firewalls)
• Load balancing, 802.1D Spanning Tree, traffic mirroring and QoS
• Interoperable delivery of third-party security applications

HyperSwitchTM will be available in two versions, a starter edition and an enterprise edition. The starter edition is FREE, where as the enterprise edition will cost USD $495. The software will be released in April 2008 and will support VMware. Support for Citrix, Virtual Iron and Microsoft virtual environments will follow in Q3-2008.

Be sure to check out the website: http://www.montegonetworks.com/

See full post at: Montego Networks – Virtual Security Switch

Everybody must have already heard about VMware’s new feature called VMsafe. On day 2 of VMworld Europe, in the keynote speech, VMware founder and chief scientist Mendel Rosenblum announced VMsafe and gave an explanation of what VMsafe can do. To me it was a bit general and I tried to find some more background info on it and strip the marketing talk. I’ve merged information from a number of sources and added my own thoughts to it.

How do I install/activate VMsafe protection?
“VMsafe is a capability inherent within VMware Infrastructure and specifically within ESX Server. Once utilized and integrated with security partner solutions, customers need only purchase integrated solutions that will be available from partners. Solutions that integrate with VMsafe will be installed as virtual machines.”

So the VMsafe API will always be included in the default ESX installation and security products can talk to the API that is available on the ESX host. This API can only be used by a VM, running a third-party security product. This seems to me as a strong point, because malicious software first has to get into a VM before it could even abuse the API to get access to other VMs. And by having to go this way, malicious software first has to bypass that same API (or security software using the API) before it could get control.

What does VMsafe protect?
“VSAFE enables partners to build a virtualization-aware security solution in the form of a security virtual machine that can access, correlate and modify information based on the following virtual hardware:
1. Memory and CPU: VMsafe provides introspection of guest VM memory pages and cpu states.
2. Networking: Network packet-filtering for both in-hypervisor and within a Security VM.
3. Process execution (guest handling): in-guest, in-process APIs that enable complete monitoring and control of process execution.
4. Storage: Virtual machine disk files (VMDK) can be mounted, manipulated and modified as they persist on storage devices.”

In the physical world, malware first had to enter memory, disk or I/O before it would be detected and (hopefully) stopped. By using VMsafe, the malware can now be stopped before it enters the OS. Its like watching the whole block around a bank building for robbers instead of posting a security guard at every door and hoping you’re not missing a door.

By stopping the malware before it enters the guest, it can’t effect the guest in anyway, it is unable to run at the same privilege level as the guest security software, which sometimes enables the malware to kill the security software and take complete control. Having no OS the malware can run on makes it completely isolated.

Still, I’m wondering what the Security VM guest OS will be and if it will be impossible for malware to abuse the API and slip into the security VM. According to George Heron, chief science officer McAfee, this will not pose to be an issue. Quoting him from the VMworld News paper that was handed out to all attendees at VMworld Europe:

“Probably the most radical aspect of the VMsafe initiative is not the technology specifications, but the fact that VMware has made the bold decision to provide secure, third-party access to the information seen by the core of its technology – the hypervisor. Security purists and VMware’s competitors will undoubtedly argue that providing access to the hypervisor, albeit in a highly controlled manner, increases the risk of the hypervisor’s own integrity being compromised, and with it the security of every virtual machine that runs on top of it. VMsafe is architected in a manner that eliminates this threat by having the security product run in an isolated space outside of the context of the hypervisor.”

Ok, but still…. there is some talk between Security VM <-> ESX API <-> Guest VM. So how does this isolation work? I haven’t been able to figure this out yet, but I guess within short time there will be more articles and white papers available explaining things in detail.

Reading a lot of articles and press releases about VMsafe, I’m convinced that VMsafe is a great step into a safer enterprise environment. Being able to stop malware before it ever reaches the VM is really great. VMsafe will not just work outside the guest, it can also work at a deeper level then other security software could do before in a virtualized environment. And let’s not just look at anti-virus products, but on firewall level / network level there are great opportunities to. A lot of security products could not Already over 20 major leading security vendors have been talking with VMware to join their VMsafe program. With this step, VMware is painting a new vision on security in the data center.

Sources:

VMware VMsafe Security Technology
VMware’s VMsafe: Security Industry Defibrillator….Making Dying Muscle Twitch Again.
Archimedius

edit:
Duncan from http://www.yellow-bricks.com/ pointed me to a possible explanation of the technique used by VMsafe, Virtual Machine Communication Interface (VMCI). Read more about it here: http://pubs.vmware.com/vmci-sdk/VMCI_intro.html

See full post at: VMSafe, what is it exactly?