To USB or not to USB, how do you boot your ESXi host?

To USB or not to USB?

On Sunday evening there was a good discussion on twitter about booting ESXi from USB stick or not. A number of arguments pro and con were made and like many discussions there is no real right or wrong. I decided to write this post to give my arguments on booting ESXi from USB stick.

Booting from USB stick

I’m very much in favor of booting ESXi hosts from USB stick. The USB stick is a very cheap medium and its power consumption is next to nothing. Compared to having an internal hard disk the costs are much lower. Since ESXi only needs the USB stick while booting it would be a waste of money and power to have a hard disk running in these hosts instead of an USB stick. Since ESXi only needs about 2GB, which is a fraction of the size of the smallest hard disk you can buy today, you would be wasting a lot of disk space when running from hard disk. So in hardware cost and power consumption, USB sticks are much cheaper than a hard disk.

Installation of ESXi on USB stick

The install procedure between USB stick or hard disk doesn’t differ that much when timed for one server, but are in favor of USB stick when having to quickly restore a failing disk or USB stick. Normally when I install USB sticks for my customers environment, I make sure I have a few extra USB sticks as spare. I always pre-install them with ESXi without running the configuration. In other words, when a host is booted from this stick, it picks up a DHCP address and then I add it to vCenter, apply a host profile and I’m done. Big advantage of USB install to me is that I can quickly pre-install a number of USB sticks using VMware Workstation on my desktop PC. When replacing a failing hard disk, I would have to do a full reinstall on a new hard disk and then apply a host profile to it. Now, I do admit that the difference in installation time is so small; it probably won’t be the deciding factor between USB and hard disk.

Reliability of USB sticks

Often when talking about booting from USB, I hear people talking about the low reliability of USB sticks. Unfortunately I cannot find any good reports that proof the reliability (or unreliability) of USB sticks and I haven’t encountered such problems myself yet, not saying nobody has had them. I do know that some HP DLxxx series server in the past (long time ago) kept on destroying every USB stick you fed them, so I won’t say there never were problems.

But even when a USB stick fails and has to be replaced, the impact is rather low. Like I already wrote, replacing the failed USB stick with a new pre-installed USB stick is very simple and takes little time. Replacing a failed hard disk would take much more time.

Since ESXi doesn’t use the USB stick after boot, a failure would only pose a problem during boot time, which is when the host already was in maintenance mode and your VMs had already been moved to other hosts. So the impact is very low.

Reliability in large environments

When operating in large enterprise environments, I do see a challenge when having to replace a failing USB stick on a regular basis. For small scale environments where access to the datacenter is easy, it doesn’t hurt when the admin has to get out of his chair once in a while and replace an USB stick. But for larger datacenters, where you have to report one day in advance that you want physical access, this can be a real problem. And of course, running a 1000 hosts with a 1% USB stick failure rate per year, will make you travel to the datacenter 10 times each year. But then again, how high is the hard disk failure rate?

Conclusion

Well, not a real hard conclusion because the failure rate of USB sticks compared to hard disk will be very decisive. My experiences with USB sticks until now are very good and I would always go for USB stick, unless someone can proof me otherwise.

So, please respond in the comments with your experiences with USB stick and hard disk failures, or any other remarks you have.

Tip: also read Setting logfile location, swap file, SNMP and vmkcore partition in ESXi