Problem:
samba-tool drs showrepl shows WERR_FILE_NOT_FOUND.
==== INBOUND NEIGHBORS ====
DC=DomainDnsZones,DC=schein,DC=ig
Default-First-Site-Name\UCS-BACKUP via RPC
DSA object GUID: 304ed834-2c38-4d75-8ec5-25fd358cd397
Last attempt @ Mon Jul 1 12:34:13 2024 CEST failed, result 2 (WERR_FILE_NOT_FOUND)
874258 consecutive failure(s).
Last success @ Mon Jan 22 20:39:04 2024 CET==== OUTBOUND NEIGHBORS ====
DC=DomainDnsZones,DC=schein,DC=ig
Default-First-Site-Name\UCS-BACKUP via RPC
DSA object GUID: 304ed834-2c38-4d75-8ec5-25fd358cd397
Last attempt @ Mon Jul 1 12:34:21 2024 CEST failed, result 2 (WERR_FILE_NOT_FOUND)
30 consecutive failure(s).
Last success @ NTTIME(0)
The output of samba-tool drs showrepl is essentially divided into three sections after the header essentially into three sections: INBOUND, OUTBOUND, and KCC connections. The perspective of INBOUND/OUTBOUND refers to the flow of replication data. An error in a connection listed under INBOUND means either that system A could not authenticate itself to system B, or that the DRSUAPI interface on system B cannot be reached, or interface is not accessible on system B, or that there is a problem with the replication of replication of individual objects.
Investigation:
Process to find the root cause:
Step1: Check the samba Database and fix errors/warnings
samba-tool dbcheck --cross-ncs
showed this:
ERROR: Link (forward) mismatch for ‘serverReference’ (1) on ‘CN=UCS-BACKUP,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig’ to ‘serverReferenceBL’ (0) on ‘CN=UCS-BACKUP,OU=Domain Controllers,DC=schein,DC=ig’
ERROR: missing backlink attribute ‘serverReferenceBL’ in CN=UCS-BACKUP,OU=Domain Controllers,DC=schein,DC=ig for link serverReference in CN=UCS-BACKUP,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig
Not fixing missing backlink serverReferenceBL
NOTE: old (due to rename or delete) DN string component for lastKnownParent in object CN=NTDS Settings\0ADEL:82d1b394-28dc-4bb7-b58d-b5f0454bfef0,CN=UCS-BACKUP\0ADEL:5a
5177af-ce94-4298-83d6-7b8b90438b83,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig - CN=UCS-BACKUP,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig
samba-tool dbcheck --cross-ncs --fix --yes
restart the service, when finished:
/etc/init.d/samba restart
You should do that on all samba DCs!
Step2: Check logfile for any hints
less /var/log/samba/log.samba
In this case, we did not have any hint of the problem neither on system A (primary) nor on system B (backup)
Step3: Increase the loglevel for the corresponding process
samba-tool processes
Service: PID
--------------------------------------
cldap_server 22107
dnsupdate 22126
dreplsrv 22111
kccsrv 22123
kdc_server 22109
kdc_server(worker 3) 22136
kdc_server(worker 0) 22117
kdc_server(worker 1) 22127
kdc_server(worker 2) 22133
ldap_server 22104
ldap_server(worker 1) 22157
ldap_server(worker 2) 22159
ldap_server(worker 0) 22155
ldap_server(worker 3) 22162
notify-daemon 22188
rpc_server 22099
rpc_server(worker 2) 22137
rpc_server(worker 0) 22122
rpc_server(worker 1) 22131
rpc_server(worker 3) 22139
samba 22085
winbind_server 22129
wrepl_server 22102
shows lots of services, we need for the drs replication the dreplsrv 22111 service with its pid.
smbcontrol 22111 debuglevel
shows the curren debug level. We need this to reset to this later
smbcontrol 22111 debug 5
Debug 5 should be sufficient you can go up to 10, but this is really much stuff you will get.
And here it comes:
[2024/07/03 11:19:18.038719, 3, pid=3387] …/…/source4/libcli/resolve/dns_ex.c:502(pipe_handler)
dns child failed to find name ‘82d1b394-28dc-4bb7-b58d-b5f0454bfef0._msdcs.schein.ig’ of type A
[2024/07/03 11:19:18.038929, 5, pid=3387] …/…/source4/dsdb/repl/drepl_notify.c:199(dreplsrv_notify_op_callback)
dreplsrv_notify_op_callback: dreplsrv_notify: Failed to send DsReplicaSync to 304ed834-2c38-4d75-8ec5-25fd358cd397._msdcs.schein.ig for DC=DomainDnsZones,DC=schein,DC=ig - NT_STATUS_OBJECT_NAME_NOT_FOUND : WERR_FILE_NOT_FOUND
Step4:
Search for the mentioned “object” 82d1b394-28dc-4bb7-b58d-b5f0454bfef0
Use your base here to get all entries and search for the string:
univention-s4search -b DC=schein,DC=ig--cross-ncs --show-binary --show-deleted|less
We got two deleted objects
Solution:
Simple as always:
root@primary:~ # ldbdel -H /var/lib/samba/private/sam.ldb 'CN=NTDS Settings\0ADEL:82d1b394-28dc-4bb7-b58d-b5f0454bfef0,CN=UCS-BACKUP\0ADEL:5a5177af-ce94-4298-83d6-7b8b90438b83,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig' --relax
root@primary:~ # ldbdel -H /var/lib/samba/private/sam.ldb 'dc=82d1b394-28dc-4bb7-b58d-b5f0454bfef0\0ADEL:132603a0-d7bf-4f67-a700-65e3237adbed,CN=Deleted Objects,DC=ForestDnsZones,DC=schein,DC=ig' --relax
root@primary:~ # /etc/init.d/bind9 restart
root@primary:~ # /etc/init.d/samba restart
For other solutions to this problem see: