Problem: UCS join script 96univention-samba4.inst fails with LDAP ENTRY ALREADY EXISTS due to stale Samba objects

Problem

After upgrading a UCS system from version 5.0 to 5.2, the Backup node fails to join the domain. This affects both the automatic execution via run join scripts during upgrade and manual execution using univention join.

The failure occurs in joinscript 96univention.samba4.inst.

The following errors appear in the join log at /var/log/univention/join.log:

ldb: Unable to open tdb '/var/lib/samba/private/secrets.ldb': No such file or directory
ldb: Failed to connect to '/var/lib/samba/private/secrets.ldb' with backend 'tdb': Unable to open tdb '/var/lib/samba/private/secrets.ldb': No such file or directory
Could not find machine account in secrets database: Failed to fetch machine account password from secrets.ldb: Could not open secrets.ldb and failed to open
/var/lib/samba/private/secrets.tdb: NT_STATUS_CANT_ACCESS_DOMAIN_INFO

Additionally, the following critical error is present:

ERROR(ldb): uncaught exception - LDAP error 68 LDAP ENTRY ALREADY EXISTS
<Entry CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de already exists>

Investigation

In the environment, the node UCS5 was completely removed from LDAP as well as from the Samba database. However, residual objects were still present in the Samba database under:
CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de

The following join log confirms the issue:

/var/log/univention/join.log

RUNNING 96univention-samba4.inst
2026-03-27 13:26:58.231400284+01:00 (in joinscript_init)
2026-03-27T13:26:59.123470+01:00     INIT 
2026-03-27T13:26:59.131987+01:00     EXIT 
Not updating samba4/role
Multifile: /etc/samba/smb.conf
Object exists: cn=Builtin,dc=univention,dc=de
No modification: cn=Enterprise Domain Controllers,cn=groups,dc=univention,dc=de
WARNING: cannot append cn=DC Backup Hosts,cn=groups,dc=univention,dc=de to nestedGroup, value exists
No modification: cn=Enterprise Domain Controllers,cn=groups,dc=univention,dc=de
WARNING: cannot append cn=ucs5,cn=dc,cn=computers,dc=univention,dc=de to hosts, value exists
Stopping samba-ad-dc (via systemctl): samba-ad-dc.service.
Stopping nmbd (via systemctl): nmbd.service.
Setting kerberos/kdc
Setting kerberos/kpasswdserver
File: /etc/krb5.conf
Setting slapd/port
Multifile: /etc/ldap/slapd.conf
File: /etc/init.d/slapd
Setting slapd/port/ldaps
Multifile: /etc/ldap/slapd.conf
File: /etc/init.d/slapd
Restarting slapd (via systemctl): slapd.serviceWarning: The unit file, source configuration file or drop-ins of slapd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
.
Create windows/wins-support
Multifile: /etc/samba/smb.conf
Join against S4 Connector server: ucs1
Forest           : univention.de
Domain           : univention.de
Netbios domain   : UNIVENTION
DC name          : ucs1.univention.de
DC netbios name  : UCS1
Server site      : Default-First-Site-Name
Client site      : Default-First-Site-Name
INFO 2026-03-27 13:27:14,959 pid:541347 /usr/lib/python3/dist-packages/samba/join.py #1605: workgroup is UNIVENTION
INFO 2026-03-27 13:27:14,959 pid:541347 /usr/lib/python3/dist-packages/samba/join.py #1608: realm is univention.de
ldb: Unable to open tdb '/var/lib/samba/private/secrets.ldb': No such file or directory
ldb: Failed to connect to '/var/lib/samba/private/secrets.ldb' with backend 'tdb': Unable to open tdb '/var/lib/samba/private/secrets.ldb': No such file or directory
Could not find machine account in secrets database: Failed to fetch machine account password from secrets.ldb: Could not open secrets.ldb and failed to open /var/lib/samba/private/secrets.tdb: NT_STATUS_CANT_ACCESS_DOMAIN_INFO
ERROR(ldb): uncaught exception - LDAP error 68 LDAP ENTRY ALREADY EXISTS -  <Entry CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de already exists> <>
  File "/usr/lib/python3/dist-packages/samba/netcmd/__init__.py", line 353, in _run
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/samba/netcmd/domain/join.py", line 129, in run
    join_DC(logger=logger, server=server, creds=creds, lp=lp, domain=domain,
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1621, in join_DC
    ctx.do_join()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1509, in do_join
    ctx.join_add_objects()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 657, in join_add_objects
    ctx.samdb.add(rec)
WARNING: The option -k|--kerberos is deprecated!
Adding CN=UCS5,OU=Domain Controllers,DC=univention,DC=de
Adding CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de
Join failed - cleaning up
Deleted CN=UCS5,OU=Domain Controllers,DC=univention,DC=de
Failed to join against the S4 Connector server ucs1.
Make sure the server is online or if this server is no longer in use,
please completely remove the server object from the domain.
Forest           : univention.de
Domain           : univention.de
Netbios domain   : UNIVENTION
DC name          : ucs1.univention.de
DC netbios name  : UCS1
Server site      : Default-First-Site-Name
Client site      : Default-First-Site-Name
INFO 2026-03-27 13:27:15,375 pid:541352 /usr/lib/python3/dist-packages/samba/join.py #104: Finding a writeable DC for domain 'univention.de'
INFO 2026-03-27 13:27:15,381 pid:541352 /usr/lib/python3/dist-packages/samba/join.py #106: Found DC ucs1.univention.de
INFO 2026-03-27 13:27:15,426 pid:541352 /usr/lib/python3/dist-packages/samba/join.py #1605: workgroup is UNIVENTION
INFO 2026-03-27 13:27:15,426 pid:541352 /usr/lib/python3/dist-packages/samba/join.py #1608: realm is univention.de
ldb: Unable to open tdb '/var/lib/samba/private/secrets.ldb': No such file or directory
ldb: Failed to connect to '/var/lib/samba/private/secrets.ldb' with backend 'tdb': Unable to open tdb '/var/lib/samba/private/secrets.ldb': No such file or directory
Could not find machine account in secrets database: Failed to fetch machine account password from secrets.ldb: Could not open secrets.ldb and failed to open /var/lib/samba/private/secrets.tdb: NT_STATUS_CANT_ACCESS_DOMAIN_INFO
ERROR(ldb): uncaught exception - LDAP error 68 LDAP ENTRY ALREADY EXISTS -  <Entry CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de already exists> <>
  File "/usr/lib/python3/dist-packages/samba/netcmd/__init__.py", line 353, in _run
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/samba/netcmd/domain/join.py", line 129, in run
    join_DC(logger=logger, server=server, creds=creds, lp=lp, domain=domain,
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1621, in join_DC
    ctx.do_join()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 1509, in do_join
    ctx.join_add_objects()
  File "/usr/lib/python3/dist-packages/samba/join.py", line 657, in join_add_objects
    ctx.samdb.add(rec)
WARNING: The option -k|--kerberos is deprecated!
Adding CN=UCS5,OU=Domain Controllers,DC=univention,DC=de
Adding CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de
Join failed - cleaning up
Deleted CN=UCS5,OU=Domain Controllers,DC=univention,DC=de
Failed to join the domain univention.de.
Make sure the server is online or if this server is no longer in use,
please completely remove the server object from the domain.
EXITCODE=1

The following ldbsearch output shows the remaining objects in full detail:

ldbsearch -H /var/lib/samba/private/sam.ldb -b "CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de"

# record 1
dn: CN=NTDS Settings,CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de
objectClass: top
objectClass: applicationSettings
objectClass: nTDSDSA
cn: NTDS Settings
instanceType: 4
whenCreated: 20240626103822.0Z
hasMasterNCs: DC=univention,DC=de
hasMasterNCs: CN=Configuration,DC=univention,DC=de
hasMasterNCs: CN=Schema,CN=Configuration,DC=univention,DC=de
uSNCreated: 31123
dMDLocation: CN=Schema,CN=Configuration,DC=univention,DC=de
invocationId: daadaca3-1246-4071-81a5-66c467900000
showInAdvancedViewOnly: TRUE
name: NTDS Settings
objectGUID: b524b6bd-6b01-48c7-948a-9f167fe000000
options: 1
systemFlags: 33554432
objectCategory: CN=NTDS-DSA,CN=Schema,CN=Configuration,DC=univention,DC=de
msDS-Behavior-Version: 4
msDS-HasDomainNCs: DC=univention,DC=de
msDS-hasMasterNCs: DC=DomainDnsZones,DC=univention,DC=de
msDS-hasMasterNCs: DC=univention,DC=de
msDS-hasMasterNCs: CN=Configuration,DC=univention,DC=de
msDS-hasMasterNCs: CN=Schema,CN=Configuration,DC=univention,DC=de
msDS-hasMasterNCs: DC=ForestDnsZones,DC=univention,DC=de
whenChanged: 20250207131210.0Z
uSNChanged: 45452
distinguishedName: CN=NTDS Settings,CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de
# record 2
dn: CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de
objectClass: top
objectClass: server
cn: UCS5
instanceType: 4
whenCreated: 20240626103821.0Z
uSNCreated: 31122
showInAdvancedViewOnly: TRUE
name: UCS5
objectGUID: 0a4cf0d7-58d1-4df3-876b-61673920000
systemFlags: 1375731712
dNSHostName: ucs5.univention.de
objectCategory: CN=Server,CN=Schema,CN=Configuration,DC=univention,DC=de
whenChanged: 20250207131610.0Z
uSNChanged: 45456
distinguishedName: CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de

# returned 2 records
# 2 entries
# 0 referrals

Solution

Remove the stale Samba objects manually from the Samba database on the Primary node.

Step 1 Remove NTDS Settings object

ldbdel -H /var/lib/samba/private/sam.ldb "CN=NTDS Settings,CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de"

Step 2 Remove server object

ldbdel -H /var/lib/samba/private/sam.ldb "CN=UCS5,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=univention,DC=de"

Step 3 Run the join script on the affected node (UCS5)

univention-join

Result

After removing the stale objects from the Samba database, the join process completes successfully.

The required objects are recreated correctly during the join, resolving the LDAP ENTRY ALREADY EXISTS error.