I have been using 1 lun per 1 datastore and everything seems to be running great. Do you see any real problems with doing this. Also, what have you seen in array size on the SAN. Do you see anything wrong with creating very large arrays and storing very small LUNs on each array. Are there any best practices you care to mention?

Thank you

As a worst case if you did a 3 x 1TB disk RAID-5 RAID group and ran 30 VMs on it you’ve got 30 servers contending for about 400 IOPS. Of course SAN cache performance etc alleviate this but surely there’d still be a major bottleneck.

I keep the lock contention at bay by limiting the number of VMs per VMFS to 25-30. With 30GB VMs, this works out to 750 – 900 GB per LUN.

Our onsite VMware TAM and our EMC storage reps actually recommend less, for busy VMs: 450 – 500 GB per LUN, with one VMFS per target.

Our VMs are never all active at the the same time, so we can get away with a higher density.

http://blogs.vmware.com/performance/2008/02/scalable-storag.html

Together with virtualization, we want to consolidate as much as possible without losing performance. In case of consolidation on LUN’s one might consider the number of VM’s that stop functioning when LUN data corruption occurs.
Last time we had a client with VM’s on 1TB LUN’s. One of those LUN’s appeared to be corrupted. There we were lucky we could still migrate the lot to another LUN. Therefore I would recommend to not consolidate TOO much VM’s onto a single datastore.

Should we therefore create lun’s per VM? No absolutely not. But we should consider downtime, data loss and recovery in case of LUN corruption.
I do my sizing for my implementations around 300-500GB per LUN. But that’s my experience

…you have “up to” 75GB of useless data to replicate…

I understand most of these blocks will be empty and therefore ignored, but on long running VMs the pagefile will still build up substantially – I would hope the VSWP stays empty (or you have bigger immediate problems) so lets say an average of 20-30GB of data for our 30 VMS… still not insignificant – especially when you think about a company with a couple thousand VMs trying to replicate overseas.

Gabe, you forgot to leave any room on the LUN for free space. 20% is the recommended amount with 10% a bare minimum.

So your forumla should read more like this:

30 x (average disk size) + 30 x (average memory allocation) + 15% of (average disk size total) + 20% (as free space) = calculated LUN size.

Keith, my understanding was that the default queue depth for most FC HBAs (QLogic/Emulex) drivers in ESX is 64 (although some are 128) and VMware support usually doesn’t want you playing with this
without good reason.

As for what VMware has to say on the subject, 50 active VMs per LUN is the max suggested and 12 VMs is what VMware consuling used to say – but I think a lot of that dated back to ESX 2.x.

20-25 VMs per LUN is what I usually do.

Also as remote SAN replication is becoming more and more common in VMware environments I have a few other thoughts. Assuming we are talking about Windows VMs only here for a sec (since most everyone understands those) and that they only require 1 VMDK (C:). I would create 2 “sister” LUNs.

LUNA = VMDK “C:” drives for each of the virtual machines
LUNB = VSWP & VMDK “D:” drives for each virtual machine (D: contains the Windows pagefile and anything else we don’t need to replicate)

Then I would only have to do SAN replication on LUNA and could completely ignore LUNB for daily replication (although you would need to get the D: VMDKs created on the other end initially – this could be manual).

If you think about it, if you have 30 active VMs on those two LUNs (using your numbers from before) with an average of 12GB VMDKs and 1GB RAM you have 75GB of useless data to replicate (1GB vswp and 1.5GB pagefile VMDK – assuming 1.5x RAM times 30 VMs) over the WAN. This is also probably the data that will be changing most often (at least for the pagefiles). Why replicate it if you don’t have to…

Thoughts?