Problem: Keycloak LOGIN_ERROR: username_in_use and invalid_user_credentials in Distributed LDAP Deployments

Knowledge Base Community
, , , ,
1

Problem

Users were unable to authenticate against an OpenDesk environment integrated with Keycloak.

The issue manifested as login failures in the authentication flow and inconsistent user resolution between LDAP replicas.

The following errors were visible in the Keycloak logs:

2026-02-11 11:53:00,271 WARN [org.keycloak.events] (executor-thread-1940) type="LOGIN_ERROR", realmId="opendesk", realmName="opendesk", clientId="https:///univention/oidc/", userId="f::name.surname", ipAddress="", error="invalid_user_credentials", auth_method="openid-connect", auth_type="code", redirect_uri="https:///univention/oidc/", code_id="", username="name.surname"

2026-02-11 11:51:59,458 WARN [org.keycloak.events] (executor-thread-8378) type="LOGIN_ERROR", realmId="opendesk", realmName="opendesk", clientId="https:///univention/oidc/", userId="null", ipAddress="", error="username_in_use", auth_method="openid-connect", auth_type="code", redirect_uri="https:///univention/oidc/", code_id="", username="name.surname"

Further investigation revealed inconsistent LDAP data between primary and secondary LDAP nodes. Multiple LDAP objects with the same uid but different entryUUID values existed on some LDAP secondaries.

The issue affected significantly more than a single account. Approximately 150 user objects showed inconsistent replication states across LDAP replicas.

Root Cause

The root cause was an inconsistent LDAP replication state between LDAP primary and secondary nodes.

Two major inconsistency patterns were identified:

  1. Duplicate LDAP objects with identical uid values but different entryUUID values existed on some secondary LDAP nodes.
  2. Secondary LDAP nodes contained newer password-related attributes and timestamps than the LDAP primaries.

This resulted in different LDAP replicas returning different user objects during authentication requests.

Because Keycloak queried inconsistent LDAP backends, the authentication flow produced:

  • username_in_use
  • invalid_user_credentials

The inconsistent replication state likely originated from replication failures or synchronization corruption within the LDAP syncrepl process.

Solution

The recommended remediation approach was to rebuild the affected LDAP secondary replicas sequentially.

Important Notes Before Proceeding

Before recreating LDAP secondaries:

  • verify whether secondary nodes contain newer or unique data
  • compare password-related attributes
  • verify whether user objects exist exclusively on secondaries
  • review modifyTimestamp
  • review entryCSN
  • review pwdChangedTime

Because secondaries contained newer password-related information in some cases, the environment required careful validation before discarding replica data.

Recreate LDAP Secondary Pods Sequentially

Delete the PersistentVolumeClaim and then the affected LDAP secondary pod.

Example:

kubectl delete pvc -n ${NAMESPACE} shared-data-ums-ldap-server-secondary-0
kubectl delete pod -n ${NAMESPACE} ums-ldap-server-secondary-0

Repeat the procedure for the remaining LDAP secondary pods.

Important:

  • only recreate one secondary at a time
  • wait for complete synchronization before continuing with the next secondary
  • monitor synchronization progress carefully

Verify Synchronization Status

Synchronization status can be verified through:

  • syncrepl logs
  • contextCSN convergence
  • LDAP service availability

The LDAP secondary service can be monitored to verify whether rebuilt replicas become operational again.

Example:

kubectl -n $NAMESPACE get services | grep ldap

Verify LDAP Object Consistency

Compare LDAP objects across all LDAP nodes before and after synchronization.

Example command:

kubectl exec -n $NAMESPACE ums-ldap-server-primary-0 -- ldapsearch -LLL -x -D "$(kubectl get -n $NAMESPACE configmaps ums-ldap-server-primary -o json | jq -r '.data.ADMIN_DN')" -w "$(kubectl get -n $NAMESPACE secrets ums-ldap-server-admin -o json | jq -r '.data.password' | base64 -d)" -b "$(kubectl get -n $NAMESPACE configmaps ums-ldap-server-primary -o json | jq -r '.data.LDAP_BASEDN')" uid=<USER> '+' '*' | less

Run the command against:

  • all LDAP primaries
  • all LDAP secondaries

Compare:

  • entryUUID
  • modifyTimestamp
  • entryCSN
  • password-related attributes
  • object existence
  • membership attributes

Investigation

Initial LDAP Inconsistency Findings

Two LDAP objects with the same uid but different entryUUID values existed on some secondary nodes.

LDAP data from affected secondaries:

dn: uid=name.surname,ou=users,ou=00,ou=,ou=tenants,dc=
structuralObjectClass: inetOrgPerson
entryUUID: 1ed06760-54c8-1040-9e86-49b22db34325
creatorsName: uid=,cn=users,dc=
createTimestamp: 20251113103447Z
memberOf: cn=Default group,cn=groups,dc=
memberOf: cn=managed-by-attribute-Fileshare,cn=groups,dc=
memberOf: cn=managed-by-attribute-Knowledgemanagement,cn=groups,dc=
memberOf: cn=managed-by-attribute-Livecollaboration,cn=groups,dc=
memberOf: cn=managed-by-attribute-Videoconference,cn=groups,dc=
memberOf: cn=-users,ou=groups,ou=00,ou=,ou=tenants,dc=
memberOf: cn=Domain Users,cn=groups,dc=
pwdChangedTime: 20251128112219Z
entryCSN: 20251128112219.385099Z#000000#003#000000
modifyTimestamp: 20251128112219Z
modifiersName: cn=admin,dc=
entryDN: uid=name.surname,ou=users,ou=00,ou=,ou=tenants,dc=
subschemaSubentry: cn=Subschema
hasSubordinates: FALSE

dn: uid=name.surname,cn=users,dc=
structuralObjectClass: inetOrgPerson
entryUUID: 4213527a-55ac-1040-8892-079a62eba42b
creatorsName: uid=,cn=users,dc=
createTimestamp: 20251114134751Z
entryCSN: 20251114134751.578035Z#000000#002#000000
modifyTimestamp: 20251114134751Z
modifiersName: cn=admin,dc=
entryDN: uid=name.surname,cn=users,dc=
subschemaSubentry: cn=Subschema
hasSubordinates: FALSE

LDAP data from primaries and one secondary:

dn: uid=name.surname,ou=users,ou=00,ou=,ou=tenants,dc=
structuralObjectClass: inetOrgPerson
entryUUID: 4213527a-55ac-1040-8892-079a62eba42b
creatorsName: uid=,cn=users,dc=
createTimestamp: 20251114134751Z
memberOf: cn=Default group,cn=groups,dc=
memberOf: cn=managed-by-attribute-Fileshare,cn=groups,dc=
memberOf: cn=managed-by-attribute-Knowledgemanagement,cn=groups,dc=
memberOf: cn=managed-by-attribute-Livecollaboration,cn=groups,dc=
memberOf: cn=managed-by-attribute-Videoconference,cn=groups,dc=
memberOf: cn=-users,ou=groups,ou=00,ou=,ou=tenants,dc=
memberOf: cn=Domain Users,cn=groups,dc=
pwdChangedTime: 20260211115046Z
entryCSN: 20260211115046.846997Z#000000#002#000000
modifiersName: cn=admin,dc=
modifyTimestamp: 20260211115046Z
entryDN: uid=name.surname,ou=users,ou=00,ou=,ou=tenants,dc=
subschemaSubentry: cn=Subschema
hasSubordinates: FALSE

Extended Replication Analysis

Additional investigation showed:

  • approximately 150 affected accounts
  • password-related attribute divergence
  • objects existing only on selected replicas
  • newer timestamps on secondaries
  • modified telephone numbers
  • additional jpegPhoto attributes

In many cases, only password-related attributes differed.

Example differences found only on affected secondaries:

userPassword:: [different value]
krb5Key:: [different value]
krb5Key:: [different value]
krb5Key:: [different value]
krb5Key:: [different value]
krb5Key:: [different value]
krb5Key:: [different value]
krb5KeyVersionNumber: 3
pwhistory: {BCRYPT}[different value] {BCRYPT}[different value]
sambaNTPassword: [different value]
shadowLastChange: 20426
sambaPwdLastSet: 1764832021
pwdChangedTime: 20251204070701Z
entryCSN: 20251204070701.070870Z#000000#003#000000
modifyTimestamp: 20251204070701Z

Additional attributes present only on some secondaries:

objectClass: univentionPasswordSelfService
univentionPasswordSelfServiceEmail: 
univentionPasswordRecoveryEmailVerified: TRUE

Service Verification

LDAP services were verified using:

kubectl -n agora-oegd get services | grep ldap

Output:

ums-ldap-notifier                           ClusterIP          <none>            6669/TCP                                       38d
ums-ldap-server                             ClusterIP          <none>            389/TCP,636/TCP                                38d
ums-ldap-server-primary                     ClusterIP          <none>            389/TCP,636/TCP                                38d
ums-ldap-server-primary-0                   ClusterIP          <none>            389/TCP,636/TCP                                38d
ums-ldap-server-primary-1                   ClusterIP          <none>            389/TCP,636/TCP                                38d
ums-ldap-server-primary-notifier            ClusterIP          <none>            389/TCP,636/TCP                                38d
ums-ldap-server-secondary                   ClusterIP      None         <none>            389/TCP,636/TCP                                38d

Connectivity testing confirmed:

  • secondaries could connect to primaries
  • primaries could connect to secondaries
  • LDAP TCP connectivity was functional

Additional Operational Considerations

The readiness probes for LDAP secondaries only verified TCP socket availability.

As a result:

  • LDAP secondaries could become reachable before synchronization was fully complete
  • read requests routed to rebuilding secondaries could temporarily fail
  • round-robin traffic distribution increased the risk of intermittent authentication failures during replica rebuilds

Additional Notes

  • Password changes were suspected to originate from the user self-service portal or password recovery workflows.
  • Password-related inconsistencies strongly indicated replication divergence after user password updates.
  • The issue demonstrated that LDAP replicas can contain newer data than primaries in broken replication scenarios.
  • Careful validation is required before deleting replicas if secondaries contain potentially newer authentication data.
  • Sequential secondary rebuilds remain the safest remediation strategy in heavily diverged replication environments.