Vmware vCenter Server 6.7 -- ldap will not bind

[davehouser1]

vCenter version: VMware-VCSA-all-6.7.0-14070457
UCS version: 4.4-1 errata196 (Blumenthal)

Our vCenter Server is unable to bind to UCS through ldap.
My Process:

Configure SSO through “Active Directory Domain”:
- In vCenter: Administration > SSO > Configuration > Active Directory Domain
- Selected “Join AD”
- Join to the domain, reboot vCenter (Note: I can also replicate this through vCenter CLI commands)
- after reboot, I go to Identity Source > Add Identity source
- PROBLEM: When going to Single Sign On > Users and Groups, and select the newly added domain, I receive “A vCenter Single Sign-On service error occured”

Upon further inspection of the the /var/log/vmware/sso/ssoAdminServer.log. I see the following every time I try to search UCS:

[WARN ][2019-07-30T16:55:40.650-04:00][jyqan3yc-174-auto-4x-h5:70000052] ServerUtils - cannot bind connection: [ldap://taxmducc01-v.cybertax.cso.com, null]
[ERROR][2019-07-30T16:55:40.650-04:00][jyqan3yc-174-auto-4x-h5:70000052] ServerUtils - cannot establish connection with uri: ldap://taxmducc01-v.cybertax.cso.com
[INFO ][2019-07-30T16:55:40.650-04:00][jyqan3yc-174-auto-4x-h5:70000052] ActiveDirectoryProvider - removeDcInfo - domain [CYBERTAX.CSO. COM], domainFQDN [taxmducc01-v.cybertax.cso. com], domainIpAddress [10.104.8.110]
[ERROR][2019-07-30T16:55:40.650-04:00][jyqan3yc-174-auto-4x-h5:70000052] ActiveDirectoryProvider - Failed to get non-GC connection to domain CYBERTAX.CSO. COM - domain controller might be offline
com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 40022][LW_ERROR_PASSWORD_MISMATCH][The password is incorrect for the given username]

I also tried to integrate directly with the “Active Directory over LDAP” function. However every time I try to connect to UCS I keep getting “Check the network settings and make sure you have network access to the identity source”, not sure why as the administrator username and password worked through AD join through the web and cli,

• Note: I tried changing the user: to cn=Administrator,cn=users,dc=cybertax,dc=cso,dc=com. As well as using the NETBIOS name CYBERTAX\Administrator. Nothing worked.

• Trouble shooting: I tried deploying a new vCenter (for testing purposes) and integrating to a Windows Domain. it worked with no problems.

Not sure what I am missing here. Any ideas?

[Moritz_Bunkus]

I’ve run into the same problem and don’t have a solution, unfortunately. I opted not to use LDAP/“Active Directory over LDAP” but to join the vCenter into the UCS AD (type “Active Directory (Windows integrated authentication)”). That worked instantly. And yes, you can still log in from non-domain joined machines, e.g. from a Linux machine without Kerberos authentication.

[davehouser1]

Thanks for the reply.
Yes, I agree, Using “Active Directory (Windows integrated authentication)” is able to join the UCS domain with no problem.
However when I try to search the domain, it fails with the “A vCenter Single Sign-On service error occurred”, and the log continually shows “ServerUtils - cannot bind connection”.

I tried performing an LDAP query from the vCenter server with the following:

ldapsearch -x -H ldaps://taxmducc01-v:636 -b dc=cybertax,dc=cso,dc=com -D cn=administrator,cn=users,dc=cybertax,dc=cso,dc=com -W
Enter LDAP Password:
ldap_sasl_bind(SIMPLE): Can’t contact LDAP server (-1)

I also tried without SSL

ldapsearch -x -H ldap://taxmducc01-v:389 -b dc=cybertax,dc=cso,dc=com -D cn=administrator,cn=users,dc=cybertax,dc=cso,dc=com -W
Enter LDAP Password:
ldap_bind: Strong(er) authentication required (8)
** additional info: BindSimple: Transport encryption required.**

The following link, Moritz_Bunkus clearly explains that AD connections via 389 just wont work, and that my 636 is failing due to bad cert trust.

LDAP Connection Issue

Hey, on a UCS server there are actually two LDAP servers listening: [ul][li]Samba on ports 389 (unencrypted or with StartTLS) and 636 (SSL encrypted)[/li] [li]OpenLDAP on ports 7389 (unencrypted or with StartTLS) and 7636 (SSL encrypted)[/li][/ul] Both contain roughly the same data; the Univention S4 Connector service is used for synchronizing that. However, the schemas used are similar but slightly different. For example, the DN for the user “administrator” is “uid=Administrator,cn=users,$b…

I did a TCP dump, and of course Univention is giving an “unknown CA” error. I have Univention’s rootCA loaded to vCenter, however I dont know how to load vCenter’s rootCA to Univention.

What is the process to load vCenter’s cert to Univention AD SSL store so it will trust the server? Is it done through openssl, or is there some special ucr command to load rootCA’s to Univention’s AD trust store?

I maybe going in the wrong direction, let me know if I am. To be honest if I can just get 389 to work that would be a start, is that even possible?

[kkorte]

To get around the Can’t contact LDAP Server, you can either deactivate the certificate check temporarily or import the certificate into the local certificate store. To deactivate it temporarily start your LDAP Search with
LDAPTLS_REQCERT=never
and use the full FQDN. So it should be something like:
LDAPTLS_REQCERT=never ldapsearch -x -H ldaps://taxmducc01-v.cybertax.cso.com:636 -b dc=cybertax,dc=cso,dc=com -D cn=administrator,cn=users,dc=cybertax,dc=cso,dc=com -W

Alternatively, you can import the certificate into the certificate store. See the VMware Documentation https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-B635BDD9-4F8A-4FD8-A4FE-7526272FC87D.html

You can find the root CA in the Menu in the upper right corner of the UCS Master under certificate.

Possibly, the certificate is what is causing the bind error. You can authenticate successful and join the Domain, which are done via Kerberos, and then get an error reading the LDAP.

[davehouser1]

kkorte,

I tried LDAPTLS_REQCERT=never.
It worked.

Unfortunately I don’t believe there is a way to apply this setting for vCenter to do this through the web interface.
Believe I am forced to use the SSL via 636.

I reviewed the link you provided. This appears to be directions to access VMware Platform Services Controller, as this vCenter Server Appliance instance is an all in one, access to the certification management tools must be done via vCenter web interface. I already loaded the UCS cert to this location, and rebooted vCenter. This makes no difference, same errors happen.

I was able to find a way to set up AD through LDAP via the CLI, see this link
https://kb.vmware.com/s/article/67304

Here are the results. I also did a tcpdump to confirm TLS is working (it is), this fails at an application level.

Any ideas?

[kkorte]

Can you try to use the global catalog server on the UCS Server by switching to port 3269

[davehouser1]

I tried changing the port to 3269.
Same error.

ERROR: Failed to probe provider connectivity [URI: ldaps://taxmducc01-v.cybertax.cso.com:3269 ]; tenantName [cybertax.com], userName [CN=Administrator,CN=users,DC=cybertax,DC=cso,DC=com]
com.vmware.identity.idm.IDMLoginException: Failed to probe provider connectivity [URI: ldaps://taxmducc01-v.cybertax.cso.com:3269 ]; tenantName [cybertax.com], userName [CN=Administrator,CN=users,DC=cybertax,DC=cso,DC=com]

Is there an ldapsearch equivalent to this command?

[kkorte]

Where does the cybertax.com come from? Is that something configure on the VMWare Server?

[davehouser1]

Yes, it is the local SSO on the server. That is how vCenter server manages it’s internal local user database.

[kkorte]

Can you check in the log files, whether there is any recent SSL error?

Reading through the article you mentioned, they explicitly require the certificat in a particular format. Can you use the following SSL command to convert the certificate

openssl x509 -outform der -in /tmp/CAcert.pem -out /tmp/CAcert.der

and then use

/tmp/CAcert.der

in the sso-config command?

[davehouser1]

I adjusted to .der format, same error. on 636 and 3269.

ERROR: Failed to probe provider connectivity

Not sure this is an SSL problem , The .pcap shows a handshake and appdata transfer, this was with the .pem, and the .der file.

Do you think the tenantName is the problem? I believe I can adjust that value, what should the tenantName be?

[kkorte]

My guess would be, that it should be equal to the Domain Name

cybertax.cso.com

Is there anything visible in the logs on the UCS
/var/log/samba/log.samba
/var/log/samba/log.smb
/var/log/auth.log

[davehouser1]

Unable to change the tenantName to anything but cybertax.com
I tailed logs while trying to connect with active directory over LDAP, nothing comes in to those three when doing so.

I tried going back to using the vmware “Windows Authentication” based method (Using this method will connect, after a reboot I am able to add it as an identity source, however I am unable to search the source.)
I get the following error:

The logs you mentioned show nothing when using this method, I did a tcpdump and I see traffic. Looks like the system is trying to use Kerberos to connect.

The vCenter log shows the same error. I looked further to see if I missed something, I found this error:

Looking online I found this link:

https://kb.vmware.com/s/article/2147174

This KB details this maybe an issue with time skew or processing. I checked htop on vcenter, there is plenty of CPU and memory. I also checked time / NTP, I have vcenter pointing to UCS, and the look synced. (.25 is vCenter (left red), .110 is UCS (right green))

Is there a Kerberos log I can view? I know Kerberos is based timed ticket exchanged,
When looking closely at the pcap trace it looks like VCSA is trying to authenticate via Kerberos but never receives an acknowledgement from UCS, and the connection closes.

[kkorte]

Kerberos is time sensitive. However, the default settings is to allow a time difference of 5 Minutes.

You might need to prefix the Domain to the username e.g. cybertax\Administrator

[Moritz_Bunkus]

During installation of your vCenter instance you’re required to create a new vSphere SSO infrastructure. You can chose the name to use there. Did you perchance use the same name for your vSphere SSO infrastructure that you use for your AD infrastructure?

[davehouser1]

kkorte: I tried using “CYBERTAX\Administrator” for the user name trying with Windows Authentication as well as Active Directory over LDAP. I also tried via CLI. All failed with the same error.

Moritz_Bunkus: Good point. Currently the vCenter server is using the domain “cybertax.com”. The UCS server is using “cybertax.cso.com”. Its important to note that I have a Windows DC with the same domain “cybertax.cso.com” on the same subnet for testing purposes. I dont believe this would be causing the problem as vCenter’s DNS entry is only pointing to the UCS server, also whenever I specify an address for Active Directory over LDAP I use the UCS server’s address.
Also note, I did deploy a new vCenter Server for testing purposes with the domain “vsphere.local” and this server runs into the same issues as the “cybertax.com” production vCenter server.
Also note, when I configure Active Directory over LDAP to point to the Window’s DC on the production vCenter, or the test vCenter, I am able to authenticate and it works. Settings I used below when authenticating to the Window DC that worked:

[davehouser1]

Update: I deployed a version of FreeIPA server, and was able to successfully connect and search via OpenLDAP with no problems.

I even tried redeploying UCS, same problem, vCenter server is unable to connect to UCS in anyway. Is the version of vCenter I am using supported / been tested?

VMware-VCSA-all-6.7.0-14070457

[kkorte]

Can you test the reaction, when trying the openLDAP Bind on UCS. The command should be the following (appart from the password)

sso-config.sh -add_identity_source -type openldap -baseUserDN "cn=users,dc=cybertax,dc=cso,dc=com " -baseGroupDN "cn=groups,dc=cybertax,dc=cso,dc=com " -domain "cybertax.cso.com" -alias "CYBERTAX" -username "uid=Administrator,cn=users,dc=cybertax,dc=cso,dc=com" -password 'password' -primaryURL "ldap://taxmducc01-v.cybertax.cso.com:7389"

[davehouser1]

kkorte,

Good news! I was finally able to get Active Directory to work! I am not 100% sure what I did to fix it. Here are the following things I did last:

1. fixed vCenter’s hostname, and made sure only UCS’s IP was the only DNS server used by vCenter Server with /opt/vmware/share/vami/vami_config_net

2. I performed the following scripts form this link (not sure if this is related to openLDAP only or Samba)
   memberOf attribute: Group memberships of user and computer objects

3. Using a domain admin I created (which inherited all the same groups as Administrator) called ssoadmin.

4. A series of reboots on both systems.

Note: I am also able to authenticate and dig via openLDAP 7389 and 7636, however I am going to use Active Directory integration as it allows me to use groups from UCS.

Thank you all for your help, and Patience.

[davehouser1]

UPDATE:
So yeah the solution I provided 9 months ago, was not a solution at all. There is nothing wrong with UCS, you can deploy right out of the box and it should work. The problem was with vCenter configuration and certs. I don’t know why but I could never get the “integrated active directory” identity service to work. It worked 9 months ago, then a month later it broke. What was odd was vCenter still allowed authentication with UCS, I could couldn’t add new groups or users. I finally got around to trying to fix it today as part of integrating a backup UCS domain server.

For those who come across this from Google do the following:

*** In UCS, MAKE sure you have a DNS record build for vcsa, and a PTR reverse lookup for vcsa.
*** In UCS, Make sure to export the certificate from /etc/univention/ssl/ucsCA/CAcert.pem. Note: It is very important to pull the latest cert, if you just added a backup domain UCS server, this cert will change, grab the latest.

Now go to vcsa Management interface (https://vcsaip:5480)

1. Make sure DNS and NTP is configured for both master and backup servers. If you cannot configure this value due to errors you can do it from cli on vCenter, you will need to enable SSH and get access to the cli for root.
   /opt/vmware/share/vami/vami_config_net
   Use option 4 to change DNS from the menu, and reboot.

Now head to vcsa web interface (https://vcsaip)

1. Log into vcsa HTML5 web interface with the administrator login (has to be this)
2. Go to vCenter Client > Administrator > Single Sign On > Configuration > Identity Services
3. Select “Add Identity Source”
4. Select “Active Directory over LDAP”, For whatever reason I could not get Active Directory Integrated services to work.

5. If it adds successfully test the integration by going to vCenter Client > Administrator > Single Sign On Users and Groups, and select “Domain” to the domain you added. If you see users and groups, integration worked.

vcsa Note: If you shutdown the UCS master and try to login with AD creds vcsa will try to use the UCS backup you configured, however timeout will take a while for the secondary Server URL to be used, around 5 min. (You can keep an eye on /var/log/vmware/sso/ssoAdminServer.log too see progress). My guess there is some setting in vCenter that is setting the timeout, just be patient and it will login eventually. If I find anything else about this I will share.