vCenter SSO changes when demoting domain controller

I’m still getting used to the important part vCenter SSO (Single Sign-On) is playing in vSphere 5.1. In my home lab I was switching domain controllers from W2k8 to Windows 2012. Transferred FSMO roles, integrated DNS, changed IP addresses for DNS on all servers and all seemed fine. My w2k8-dom01 server was demoted and removed. Few days later when trying to make a vCenter connection, I couldn’t logon anymore. As a good Windows Admin I of course first rebooted the vCenter VM but (as all real admins know) that seldom fixes the issue. Diving in the vCenter log files at “C:\ProgramData\VMware\VMware VirtualCenter\Logs” I found the following error:

2013-01-27T10:46:20.302+01:00 [04304 info ‘[SSO][SsoAdminFacadeImpl]’] [FindPersonUser] 2013-01-27T10:46:20.427+01:00 [04304 warning ‘Default’] Warning, existence of user “VANZANTEN\Administrator” unknown, permission may not be effective until it is resolved. 2013-01-27T10:46:20.427+01:00 [04304 error ‘Default’] The user account “VANZANTEN\Administrator” could not be successfully resolved. Check network connectivity to domain controllers and domain membership. Users may not be able to log in until connectivity is restored. 2013-01-27T10:46:20.427+01:00 [04304 error ‘Default’] [ACL] Adding unresolved permission for user “VANZANTEN\Administrator” 2013-01-27T10:46:20.427+01:00 [04304 info ‘[SSO]’] [UserDirectorySso] GetUserInfo(VANZANTEN\vSphereAdminGroup, true) 2013-01-27T10:46:20.427+01:00 [04304 info ‘[SSO][SsoAdminFacadeImpl]’] [FindGroup] 2013-01-27T10:46:20.536+01:00 [04304 warning ‘Default’] Warning, existence of group “VANZANTEN\vSphereAdminGroup” unknown, permission may not be effective until it is resolved. 2013-01-27T10:46:20.536+01:00 [04304 error ‘Default’] The group account “VANZANTEN\vSphereAdminGroup” could not be successfully resolved. Check network connectivity to domain controllers and domain membership. Users may not be able to log in until connectivity is restored.

That is when the lightbulb in my head went on (hey it was Sunday morning and I didn’t have my coffee yet).  I figured the vCenter SSO (Single Sign-On) service was still trying to talk to the old domain controller for authentication, so I opened the administration page of SSO:  https://<your SSO server>:9443/vsphere-client/ Login with:  admin@system-domain  (or root@system-domain for the vCenter Server Appliance). You did write down that password, did you? Go to: “Sign-On and Discovery” -> Configuration and look at the identity sources. There it is, the reference to the old domain controller. vCenter SSO configuration

Change vCenter SSO domain controller

Changing it is easy, click “Edit identity source” and change the name of the new domain controller. Test your settings and you would expect to be done then. vCenter SSO edit identity source

Unfortunately I received the following error: “The edit identity source operation failed for the entity with the following error message. Cannot connect to one or more of the provided external server URLs: w2k8-dom01.vanzanten.local:389″.

vCenter SSO identity source failed

Strangely enough it is still pointing at the old domain controller. The easiest way to solve this was to just delete the entry and create a new one with the new domain controller in. Just to be sure I restarted the vCenter SSO (Single Sign-On) service first and then vCenter Server would start without any issues.

Update:

In the comments on this post, Jad (JM) wrote that you can also use the domain name when entering the primary server URL. In my case I would NOT enter: “ldap://w12-dc01.vanzanten.local” as my Primary URL but just “ldap://vanzanten.local”. vCenter SSO will then query the domain for the special domain controller DNS record and use this to find the domain controller to talk to. Btw, this doesn’t work if you already have entered two URLs and want to change them and make the second one empty. Think it is purely a GUI issue. When I empty the second URL, save it and then edit to check again, it keeps the old entry. When you delete the whole entry and create a new one that has an empty secondary URL, you’re fine. Thanks Jad!

  • http://twitter.com/BjornHouben Bjorn Houben

    It’s always hard to determine what static links/references there are when replacing Domain Controllers and it is easy to overlook some. Thanks for the info.

  • amusica

    If you have web installed can you still with with that? I am having that orob after the change.

  • @millardjk

    This is an avoidable error on VMware’s part. Instead of statically retaining the DC, they should be using DNS lookups to find the DCs in the environment.
    Sheesh!

  • http://www.GabesVirtualWorld.com Gabrie van Zanten

    Well, that is what I thought too :-)

  • http://twitter.com/clbarnett clbarnett

    Wow, if it’s pointing directly to a DC instead of AD, then that would imply just downtime on the DC would break SSO. That’s an *awful* design on VMware’s part, adding in a dependency to a single server. Can anyone confirm that downtime on the configured DC will break SSO?

  • http://www.GabesVirtualWorld.com Gabrie van Zanten

    You can enter TWO domain controllers !!! But in my case I had shutdown both the w2008 domain controllers and was running on the two new W2012 domain controllers. Btw, I do think it would be better if you would just enter a domain name and let SSO figure out through DNS what the domain controllers are.

  • amusica

    It helps when not driving and posting…sorry.. Can you still login with AD creds via the Web client? Mine stopped working after adding new DC’s and upgrading identity source (unless using admin@System-Domain

  • http://www.GabesVirtualWorld.com Gabrie van Zanten

    No, you have to login with admin@system-domain or root@system-domain when using the vCenter Appliance

  • JM

    Not a Vmware issue per se :) An easy way to avoid this issue is to point to the domain instead of a specific host. In your case, you can simply specify vanzanten.local:389.

    If you dig in the DNS setup of an AD domain, you’ll notice an empty (that is, ) A record for each DC.

    Should you remove or add a DC, SSO will pick it up quickly enough, depending on the TTL in your DNS.

    If you prefer to keep some control over the process, you can simply create multiple A records with the same name (something like ldap.vanzanten.local) pointing to whichever DCs you want to use.

  • JM

    Gah. Sorry about the errant tags, didn’t think Disqus would do this.

  • http://www.GabesVirtualWorld.com Gabrie van Zanten

    Thank you for the comment. I tested it and worked :-) I updated my post to include your tip.

  • Box293

    This works fine with a single AD site, but when you have sites separated by WAN links, using domain.local will resolve to any DNS server at any site. Using domain.local just uses DNS round robin and in no way is related to how sites are separated in active directory sites and services.

  • Jason

    I believe I have this error, but the vsphere service wont even start anymore so I can’t get in to make this change. Any ideas?

  • sysxperts

    I’ve also gotten into habit of connecting to 3389 (global catalog) instead of 389 for multi-site configurations since discovering issues in the way referrals are handled causing authentication timeouts

  • Michael

    Life safer! Thank you! Vsphere did have a backup DC to point to, but that guy ate it sometime last year and I demoted the main DC this weekend and woke up to find none of my NetApp SMVI snapshots took place :-O

    Again thank you!

  • http://www.GabesVirtualWorld.com Gabrie van Zanten

    Glad it helped :-)

  • TJappleby

    Hi Gabrie, Great article thank you. I encountered a similar problem during a DR site, primary site outage and therefore primary AD servers down – unable to login to vCenter from DR site. So to me looks like same issue as you. However, in vCenter 5.5 no longer specify ‘hardcoded’ AD server in SSO configuration, but domain name with ‘active directory (integrated windows authentication) option. Has anyone else come across this before? How to get SSO to pickup DR AD servers if it losses access to primary site DCs? Does the placement of the FSMO roles have any bearing on this?