We all know how we can set the correct time in ESX and in ESXi. It’s quite simple, just right click the host, go to the configuration tab. There you select “Time configuration” and “properties”.  The configuration screen that appears is self-explanatory and after entering the NTP server addresses you are up and running.  Now on every subsequent connection you make, you will see that the correct local time is displayed.
The strange thing I found when testing a nice appliance is that internally ESXi still uses UTC time.  This appliance is called PhpLogCon, it receives syslogs from a number of systems and merges them into one log. You can then filter on events, hosts, etc. I have my ESXi host and my router logging to it.  For now it works great.
However, what I noticed was that the log entries from ESXi were one hour off, after a little investigation and some help from @svenh, I found that ESXi always runs in UTC time internally and therefore the logs have the UTC time stamp to. for a little more information on this visit this post in the VMware Community.
So, ESXi will always run in UTC time.  Now, is this a problem?  The first issue to think of is, VM time sync, will Guests that are set to sync time with their host be one hour off? Hmmm… nope, because under the covers ESX also does it’s time keeping in UTC time, it just displays it to you in your preferred time zone.  So if a VM running on ESX / ESXi asks the host about the correct time, the host will always return UTC time and the VM will calculate the correct time based on it’s own time zone, ie. how many hours to add or subtract.  So for guest OS’s there shouldn’t be a problem. That also includes Active Directory Domain Controllers.
There is however, an issue when you start using the internal logs of ESXi. As already stated, I’m allowing ESXi to output all syslogs to the a fore mentioned appliance I am testing.  In this appliance the logs show up with the “wrong” time stamp ie. UTC not my timezone. Therefore if using these logs to troubleshoot a problem, one would have to be careful to first align the ESXi logs with the logs from other systems, like SAN, routers, Windows, etc.  I often have to wade through the vmkernel logs of ESX hosts at work (sad I know, but there is not much else to do), to find related entries whenever there was an issue and if their time would be off for one or more hours, it would make things a little more difficult.
I would love to hear remarks from other professionals in the field on this, are there situations in which you foresee problems? Would you have simple solutions to my “issue”? Would anyone know what time stamp is on SNMP traps from ESXi and how this is interpreted by the management systems?
Remarks:
- ESXi doesn’t have a /var/log/vmkernel anymore, they have merged in to /var/log/messages
- To redirect output of the syslogs to an external system: Connect with the VI Client to the host, go to configuration tab, advanced settings, Syslog -> remote and enter the IP of the remote host in Syslog.Remote.Hostname
- Thanks to Tom Howarth for reviewing this post.
The most important thing is to be aware of timezone differences when comparing logs. In previous projects I had the same “issue” comparing IIS, Exchange smtp logfiles and .NET logfiles. The key in debugging is that you know your sources so that you know the timezone they operate. Daylight saving time is another nice complication to add to it by the way.
I used a (windows) program called baregrep (it has the possibility to use regular expressions) to filter the logs.
I always set it to UTC anyway, it makes it easier to correlate events from devices in different timezones. I can appreciate how it would be annoying for you where everything is in the same one.
Well, first to your issue. You might be able to change the timezone, by updating /etc/localtime (not tested), but that’s how you would change the timezone on a “normal” linux system.
Second, on the implications off this:
Well, we recently had a call with IBM/VMware where we some issues with VM’s on a certain storage (IBM DS4000) were gonna go nuts and kick their whole filesystem (either xfs, reiserfs or ext3/4); so we went ahead and sent the storage/ESX logs to VMware. Apparently they didn’t find any issues at the specified time (the time, where we created a new array on the storage resulting in the read-only filesystem(s)). So I think, this might be an implication of ESX/ESXi using UTC/vs. our local time.
Did anyone find a solution to set the timezone in ESXi. It´s a horrible bug!
The way it affects me is in reporting. For instance, we run a status report that checks all of the servers and VMs for critical processes such as backups, updates to anti-spam and anti-virus, software patch availability etc. Order is important. Timestamped output from ESXi will be GMT while other processes will be localized. To keep it all straight, we schedule all of the processes from one VM. However, when you read the report, mental gymnastics are required based on GMT offset and daylight savings time to determine the date and time when the ESXi-run processes started and finished. All operating systems work in GMT but VMware doesn't output localized time.
What do you use for the reporting?
We are at the UTC+12 timezone. We are using ESXi 4.0, and are also using VDR. At times VDR hangs whilst backing up a server, and the only way to rectify is to poweroff the VDR appliance, and power back on. I have set scheduled tasks to perform this around 6am. I have found that the scheduled tasks use the UTC date to determine which day to run the task, but it uses the NTP time of day. This means that if I schedule the task for a Monday (8th), it runs on the 8th according to UTC time, which is my Tuesday. However, it still runs at the NTP time of 6am……Odd…
I have managed to get around this by scheduling the job for the day before (i.e. Sunday 7th).
It would be good if ESXi could be changed to allow the use of the local time zone.
If you would like to use Loganalyzer (previously php-logcon) with your ESXi environment and need to convert time to the current time in your area or the same as your syslogger I would suggest installing syslog-ng and configuring syslog-ng to log to a DB of your choice connecting loganalyzer and then telling syslog-ng to keep the syslog timestamps the same as the syslog server, that is a workaround to your problem.
Thanks for the nice info u helped me from searching the whole ocean for a command to change the ESXi server from UTC time zone.