Problem:
samba-tool drs showrepl shows WERR_FILE_NOT_FOUND.
==== INBOUND NEIGHBORS ====
DC=DomainDnsZones,DC=schein,DC=ig
Default-First-Site-Name\UCS-BACKUP via RPC
DSA object GUID: 304ed834-2c38-4d75-8ec5-25fd358cd397
Last attempt @ Mon Jul 1 12:34:13 2024 CEST failed, result 2 (WERR_FILE_NOT_FOUND)
874258 consecutive failure(s).
Last success @ Mon Jan 22 20:39:04 2024 CET==== OUTBOUND NEIGHBORS ====
DC=DomainDnsZones,DC=schein,DC=ig
Default-First-Site-Name\UCS-BACKUP via RPC
DSA object GUID: 304ed834-2c38-4d75-8ec5-25fd358cd397
Last attempt @ Mon Jul 1 12:34:21 2024 CEST failed, result 2 (WERR_FILE_NOT_FOUND)
30 consecutive failure(s).
Last success @ NTTIME(0)
The output of samba-tool drs showrepl is essentially divided into three sections after the header essentially into three sections: INBOUND, OUTBOUND, and KCC connections. The perspective of INBOUND/OUTBOUND refers to the flow of replication data. An error in a connection listed under INBOUND means either that system A could not authenticate itself to system B, or that the DRSUAPI interface on system B cannot be reached, or interface is not accessible on system B, or that there is a problem with the replication of replication of individual objects.
WERR_FILE_NOT_FOUNDon, for example, the backup server → means: the backup DC thinks the partition does not exist on the source, even though it is correctly available for other DCs.- Typical for inconsistent replication metadata on the backup:
- Old connection objects or incorrect replNeighbour entries
- Defective or incomplete
NTDS settingsfor the partition - Local database of the backup DC is somewhat “twisted”
Investigation:
Process to find the root cause:
Step1: Check the samba Database and fix errors/warnings
samba-tool dbcheck --cross-ncs
showed this:
ERROR: Link (forward) mismatch for ‘serverReference’ (1) on ‘CN=UCS-BACKUP,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig’ to ‘serverReferenceBL’ (0) on ‘CN=UCS-BACKUP,OU=Domain Controllers,DC=schein,DC=ig’
ERROR: missing backlink attribute ‘serverReferenceBL’ in CN=UCS-BACKUP,OU=Domain Controllers,DC=schein,DC=ig for link serverReference in CN=UCS-BACKUP,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig
Not fixing missing backlink serverReferenceBL
NOTE: old (due to rename or delete) DN string component for lastKnownParent in object CN=NTDS Settings\0ADEL:82d1b394-28dc-4bb7-b58d-b5f0454bfef0,CN=UCS-BACKUP\0ADEL:5a
5177af-ce94-4298-83d6-7b8b90438b83,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig - CN=UCS-BACKUP,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig
samba-tool dbcheck --cross-ncs --fix --yes
restart the service, when finished:
/etc/init.d/samba restart
You should do that on all samba DCs!
Step2: Check logfile for any hints
less /var/log/samba/log.samba
In this case, we did not have any hint of the problem neither on system A (primary) nor on system B (backup)
Step3: Increase the loglevel for the corresponding process
samba-tool processes
Service: PID
--------------------------------------
cldap_server 22107
dnsupdate 22126
dreplsrv 22111
kccsrv 22123
kdc_server 22109
kdc_server(worker 3) 22136
kdc_server(worker 0) 22117
kdc_server(worker 1) 22127
kdc_server(worker 2) 22133
ldap_server 22104
ldap_server(worker 1) 22157
ldap_server(worker 2) 22159
ldap_server(worker 0) 22155
ldap_server(worker 3) 22162
notify-daemon 22188
rpc_server 22099
rpc_server(worker 2) 22137
rpc_server(worker 0) 22122
rpc_server(worker 1) 22131
rpc_server(worker 3) 22139
samba 22085
winbind_server 22129
wrepl_server 22102
shows lots of services, we need for the drs replication the dreplsrv 22111 service with its pid.
smbcontrol 22111 debuglevel
shows the curren debug level. We need this to reset to this later
smbcontrol 22111 debug 5
Debug 5 should be sufficient you can go up to 10, but this is really much stuff you will get.
And here it comes:
[2024/07/03 11:19:18.038719, 3, pid=3387] …/…/source4/libcli/resolve/dns_ex.c:502(pipe_handler)
dns child failed to find name ‘82d1b394-28dc-4bb7-b58d-b5f0454bfef0._msdcs.schein.ig’ of type A
[2024/07/03 11:19:18.038929, 5, pid=3387] …/…/source4/dsdb/repl/drepl_notify.c:199(dreplsrv_notify_op_callback)
dreplsrv_notify_op_callback: dreplsrv_notify: Failed to send DsReplicaSync to 304ed834-2c38-4d75-8ec5-25fd358cd397._msdcs.schein.ig for DC=DomainDnsZones,DC=schein,DC=ig - NT_STATUS_OBJECT_NAME_NOT_FOUND : WERR_FILE_NOT_FOUND
Step4:
Search for the mentioned “object” 82d1b394-28dc-4bb7-b58d-b5f0454bfef0
Use your base here to get all entries and search for the string:
univention-s4search -b DC=schein,DC=ig--cross-ncs --show-binary --show-deleted|less
We got two deleted objects
Solution:
Simple as always:
root@primary:~ # ldbdel -H /var/lib/samba/private/sam.ldb 'CN=NTDS Settings\0ADEL:82d1b394-28dc-4bb7-b58d-b5f0454bfef0,CN=UCS-BACKUP\0ADEL:5a5177af-ce94-4298-83d6-7b8b90438b83,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=schein,DC=ig' --relax
root@primary:~ # ldbdel -H /var/lib/samba/private/sam.ldb 'dc=82d1b394-28dc-4bb7-b58d-b5f0454bfef0\0ADEL:132603a0-d7bf-4f67-a700-65e3237adbed,CN=Deleted Objects,DC=ForestDnsZones,DC=schein,DC=ig' --relax
root@primary:~ # /etc/init.d/bind9 restart
root@primary:~ # /etc/init.d/samba restart
possible culprit:
It is also usefull, if this not solves the issue, to check your interfaces, configured on the system and the bindingsfor the interfaces in samba.
ucr get samba/interfaces
an other possible culprit:
Maybe this is not enough and the problem is a little bit deaper in the NDR: struct repsFromToBlob. This is the internal structure samba saves in LDB. In this case repsFrom which is an multivalue entry and lists all known replication sources for this partition, we might have an old server dns-entry there. → Backup DC believes that this source is responsible for replication. However, the source no longer exists → WERR_FILE_NOT_FOUND when attempting to retrieve changes.
So samba-tool drs kcc could renew these entries, but if
samba-tool drs kcc does not the job, you can replicate the specific object with the faulty entry via
samba-tool drs replicate --local "$(hostname)" <SLAVE> --single-object "DC=DomainDnsZones,$( ucr get ldap/base )" -d5 -P
There should be an debug output, that e.g repsFrom was updated.
If this is not possible or sucessfull you can (in case you have another replication partner, who does not the WERR_FILE_NOT_FOUND problem) use the entries to modify the entries manually.
It is not possible to just edit the DC Name, because the repsFrom entry is a NDR-serialisierter Binary-Blob (Network Data Representation).
So you get the RepsFrom from a healthy server and edit the entry on the problematic server.
# record 1
dn: DC=DomainDnsZones,DC=univention,DC=intranet
objectClass: top
objectClass: domain
objectClass: domainDNS
description: Microsoft DNS Directory
instanceType: 13
whenCreated: 20250924143133.0Z
whenChanged: 20250924143133.0Z
uSNCreated: 3967
name: DomainDnsZones
objectGUID: 7521b17a-4008-4e60-99ed-438413c61ebd
objectCategory: CN=Domain-DNS,CN=Schema,CN=Configuration,DC=univention,DC=intr
anet
msDS-NcType: 0
dc: DomainDnsZones
wellKnownObjects: B:32:6227F0AF1FC2410D8E3BB10615BB5B0F:CN=NTDS Quotas,DC=Doma
inDnsZones,DC=univention,DC=intranet
wellKnownObjects: B:32:18E2EA80684F11D2B9AA00C04F79F805:CN=Deleted Objects,DC=
DomainDnsZones,DC=univention,DC=intranet
wellKnownObjects: B:32:2FBAC1870ADE11D297C400C04FD8D5CD:CN=Infrastructure,DC=D
omainDnsZones,DC=univention,DC=intranet
wellKnownObjects: B:32:AB8153B7768811D1ADED00C04FD8D5CD:CN=LostAndFound,DC=Dom
ainDnsZones,DC=univention,DC=intranet
uSNChanged: 3974
msDs-masteredBy: CN=NTDS Settings,CN=UCS5PRIMARY,CN=Servers,CN=Default-First-S
ite-Name,CN=Sites,CN=Configuration,DC=univention,DC=intranet
msDs-masteredBy: CN=NTDS Settings,CN=UCS5BACKUP,CN=Servers,CN=Default-First-Si
te-Name,CN=Sites,CN=Configuration,DC=univention,DC=intranet
replUpToDateVector:: AgAAAAAAAAABAAAAAAAAAPm9AeGIGmFIsAMnD4QZo1lPEAAAAAAAAACAP
tXesZ0B
repsFrom:: AQAAAAAAAAAUAQAAAAAAACU9Ah8DAAAAJT0CHwMAAAAAAAAA0AAAAEQAAAB0AAAAERE
RERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERERER
ERERERERERERERERERERERERERERERERAAAAAE8QAAAAAAAAAAAAAAAAAABPEAAAAAAAAGA1JxD0t
mZMlJdtcZyGJaX5vQHhiBphSLADJw+EGaNZAAAAAAAAAAAAAAAAAAAAAEAAAAAxMDI3MzU2MC1iNm
Y0LTRjNjYtOTQ5Ny02ZDcxOWM4NjI1YTUuX21zZGNzLnVuaXZlbnRpb24uaW50cmFuZXQA
repsTo:: AQAAAAAAAAAUAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA0AAAAEQAAAAcAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGA1JxD0tmZ
MlJdtcZyGJaUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAxMDI3MzU2MC1iNmY0
LTRjNjYtOTQ5Ny02ZDcxOWM4NjI1YTUuX21zZGNzLnVuaXZlbnRpb24uaW50cmFuZXQA
distinguishedName: DC=DomainDnsZones,DC=univention,DC=intranet
If there is no other server to copy from, you can create an alias entry via UMC for the faulty entry, as a workaround. In DNS Module, there are already these entries, use the faulty one to create an alias, to an existing replication partner.
For other solutions to this problem see: