System Diagnostics - check Samba Replication errrors

UCS - Univention Corporate Server
#1

Hello everyone, after upgrading to 4.3 I noticed many errors in the diagnostics. Join scripts fails (i have another post for that) and users had to removed from one group and re-added to clear 20 other errors.

I am stuck on this one… I originally performed AD takeover from a Zentyal network about a year ago and everything seemed to be good but now I am seeing errors. the old servers SGFS1 and SGFS2 are no longer in use (haven’t been since moving away from zentyal. I thought I removed them all correctly before… I see this:

`samba-tool drs showrepl` returned a problem with the replication.
Inbound 'CN=Configuration,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS2 (WERR_FILE_NOT_FOUND)
Inbound 'CN=Configuration,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS3 (WERR_FILE_NOT_FOUND)
Inbound 'CN=Schema,CN=Configuration,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS2 (WERR_FILE_NOT_FOUND)
Inbound 'CN=Schema,CN=Configuration,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS3 (WERR_FILE_NOT_FOUND)
Inbound 'DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS2 (WERR_FILE_NOT_FOUND)
Inbound 'DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS3 (WERR_FILE_NOT_FOUND)
Inbound 'DC=DomainDnsZones,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS2 (WERR_FILE_NOT_FOUND)
Inbound 'DC=DomainDnsZones,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS3 (WERR_FILE_NOT_FOUND)
Inbound 'DC=ForestDnsZones,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS2 (WERR_FILE_NOT_FOUND)
Inbound 'DC=ForestDnsZones,DC=sgvfr,DC=lan': error during DRS replication from Default-First-Site-Name/SGFS3 (WERR_FILE_NOT_FOUND)

How can I remove these and clear the errors? I have tried with the snap-in from MMC in windows and i cannot remove the entries… I don’t see them in DNS or LDAP either…

any thoughts please?

#2

Hey,

first, verify whether the Samba4 LDAP still contains entries for those two servers, e.g. with something like unvention-s4search --cross-ncs | grep -Ei 'sgfs2|sgfs3'. If they do, you can try removing the references with a little helper script:

/usr/share/univention-samba4/scripts/purge_s4_computer.py --computername=<hostname>

You should probably run that for both server names anyway.

The script is mentioned in the following knowledge base article:

https://help.univention.com/t/samba-4-troubleshooting/32

Kind regards,
mosu

#3

Hi Mosu, thanks. I should have mentioned I did try that before with no luck. the hostnames are not found when trying to purge either system. Apologize for my questions, and the missing info from the original post… I do try to research and attempt to resolve before posting… I hope i’m missing something very simple.

Working DC’s removed to shorten the list.

root@ucs1:/home/dmadmin# samba-tool drs showrepl
Default-First-Site-Name\UCS1
DSA Options: 0x00000001
DSA object GUID: 08aab86f-8814-4a7a-86b2-87a25b68c6ed
DSA invocationId: 8a5c71b9-7d15-4d46-9264-612f44e9ccae

==== INBOUND NEIGHBORS ====

CN=Configuration,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS2 via RPC
                DSA object GUID: 84933a9b-bcac-4632-87aa-3db9129b9c03
                Last attempt @ Mon Mar 19 10:12:02 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52290 consecutive failure(s).
                Last success @ NTTIME(0)

CN=Configuration,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS3 via RPC
                DSA object GUID: 1949d833-917b-4ff0-ade0-ffd16f10e314
                Last attempt @ Mon Mar 19 10:12:02 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52290 consecutive failure(s).
                Last success @ NTTIME(0)

CN=Schema,CN=Configuration,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS2 via RPC
                DSA object GUID: 84933a9b-bcac-4632-87aa-3db9129b9c03
                Last attempt @ Mon Mar 19 10:12:02 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52290 consecutive failure(s).
                Last success @ NTTIME(0)

CN=Schema,CN=Configuration,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS3 via RPC
                DSA object GUID: 1949d833-917b-4ff0-ade0-ffd16f10e314
                Last attempt @ Mon Mar 19 10:12:02 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52290 consecutive failure(s).
                Last success @ NTTIME(0)

DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS2 via RPC
                DSA object GUID: 84933a9b-bcac-4632-87aa-3db9129b9c03
                Last attempt @ Mon Mar 19 10:12:02 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52290 consecutive failure(s).
                Last success @ NTTIME(0)

DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS3 via RPC
                DSA object GUID: 1949d833-917b-4ff0-ade0-ffd16f10e314
                Last attempt @ Mon Mar 19 10:12:02 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52290 consecutive failure(s).
                Last success @ NTTIME(0)

DC=DomainDnsZones,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS2 via RPC
                DSA object GUID: 84933a9b-bcac-4632-87aa-3db9129b9c03
                Last attempt @ Mon Mar 19 10:12:01 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52291 consecutive failure(s).
                Last success @ NTTIME(0)

DC=DomainDnsZones,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS3 via RPC
                DSA object GUID: 1949d833-917b-4ff0-ade0-ffd16f10e314
                Last attempt @ Mon Mar 19 10:12:01 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52291 consecutive failure(s).
                Last success @ NTTIME(0)

DC=ForestDnsZones,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS2 via RPC
                DSA object GUID: 84933a9b-bcac-4632-87aa-3db9129b9c03
                Last attempt @ Mon Mar 19 10:12:01 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52291 consecutive failure(s).
                Last success @ NTTIME(0)

DC=ForestDnsZones,DC=sgvfr,DC=lan
        Default-First-Site-Name\SGFS3 via RPC
                DSA object GUID: 1949d833-917b-4ff0-ade0-ffd16f10e314
                Last attempt @ Mon Mar 19 10:12:02 2018 PDT failed, result 2 (WERR_FILE_NOT_FOUND)
                52290 consecutive failure(s).
                Last success @ NTTIME(0)

==== OUTBOUND NEIGHBORS ====

No errors here.. correctly shows UCS2


==== KCC CONNECTION OBJECTS ====

Connection --
        Connection name: 7bd070c4-ec49-4c62-9b5b-507d8f39cfa4
        Enabled        : TRUE
        Server DNS name : ucs2.sgvfr.lan
        Server DN name  : CN=NTDS Settings,CN=UCS2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan
                TransportType: RPC
                options: 0x00000001
Warning: No NC replicated for Connection!
Connection --
        Connection name: b1f0c28b-d87a-4341-a67a-33815409d25e
        Enabled        : TRUE
        Server DNS name : SGFS2.sgvfr.lan
        Server DN name  : CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan
                TransportType: RPC
                options: 0x00000001
Warning: No NC replicated for Connection!
Connection --
        Connection name: bae98df6-4279-4f43-9095-da34ae5e0f28
        Enabled        : TRUE
        Server DNS name : SGFS3.sgvfr.lan
        Server DN name  : CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan
                TransportType: RPC
                options: 0x00000001
Warning: No NC replicated for Connection!
root@ucs1:/home/dmadmin# univention-s4search --cross-ncs | grep -Ei 'sgfs2|sgfs3'
masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN
masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN
msDs-masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Na
msDs-masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Na
dn: CN=Mailbox Store (SGFS2),CN=First Storage Group,CN=InformationStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=SGVFR,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=sgvfr,DC=lan
cn: Mailbox Store (SGFS2)
displayName: Mailbox Store (SGFS2)
adminDisplayName: Mailbox Store (SGFS2)
name: Mailbox Store (SGFS2)
 ervers/cn=SGFS2/cn=Microsoft Private MDB
msExchOwningServer: CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Admin
msExchHomePublicMDB: CN=Public Folder Store (SGFS2),CN=First Storage Group,CN=
 InformationStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Adminis
distinguishedName: CN=Mailbox Store (SGFS2),CN=First Storage Group,CN=Informat
 ionStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administrative
msExchServer1NetworkAddress: sgfs2
msExchServer1HighestUSNVector: SGFS2:53500
msExchAddressListServiceLink: CN=SGFS2,CN=Servers,CN=First Administrative Grou
dn: CN=Public Folder Store (SGFS2),CN=First Storage Group,CN=InformationStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=SGVFR,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=sgvfr,DC=lan
cn: Public Folder Store (SGFS2)
displayName: Public Folder Store (SGFS2)
adminDisplayName: Public Folder Store (SGFS2)
proxyAddresses: SMTP:SGFS2-IS@sgvfr.lan
mailNickname: SGFS2-IS
name: Public Folder Store (SGFS2)
 ervers/cn=SGFS2/cn=Microsoft Public MDB
textEncodedORAddress: c=US;a= ;p=First Organizati;o=Exchange;s=SGFS2-IS;
mail: SGFS2-IS@sgvfr.lan
msExchOwningServer: CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Admin
distinguishedName: CN=Public Folder Store (SGFS2),CN=First Storage Group,CN=In
 formationStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administr
dn: CN=InformationStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=SGVFR,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=sgvfr,DC=lan
distinguishedName: CN=InformationStore,CN=SGFS2,CN=Servers,CN=First Administra
dn: CN=First Storage Group,CN=InformationStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=SGVFR,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=sgvfr,DC=lan
distinguishedName: CN=First Storage Group,CN=InformationStore,CN=SGFS2,CN=Serv
fromServer: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN
dn: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan
distinguishedName: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-
dn: CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan
cn: SGFS2
name: SGFS2
dNSHostName: SGFS2.sgvfr.lan
distinguishedName: CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=
fromServer: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN
fromServer: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN
msDS-NC-Replica-Locations: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-Fir
msDS-NC-Replica-Locations: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-Fir
dn: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan
distinguishedName: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-
fromServer: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN
msDS-NC-Replica-Locations: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-Fir
msDS-NC-Replica-Locations: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-Fir
masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN
masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN
msDs-masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Na
msDs-masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Na
dn: CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan
cn: SGFS3
name: SGFS3
dNSHostName: SGFS3.sgvfr.lan
distinguishedName: CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=
siteFolderServer: CN=Public Folder Store (SGFS2),CN=First Storage Group,CN=Inf
 ormationStore,CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administra
dn: CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=SGVFR,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=sgvfr,DC=lan
cn: SGFS2
adminDisplayName: SGFS2
networkAddress: ncacn_vns_spp:SGFS2
networkAddress: netbios:SGFS2
networkAddress: ncacn_np:SGFS2
networkAddress: ncacn_spx:SGFS2
networkAddress: ncacn_ip_tcp:sgfs2.sgvfr.lan
networkAddress: ncalrpc:SGFS2
name: SGFS2
 ervers/cn=SGFS2
distinguishedName: CN=SGFS2,CN=Servers,CN=First Administrative Group,CN=Admini
msDs-masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Na
msDs-masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Na
msDs-masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Na
msDs-masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Na
masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Name,CN
masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN
msDS-IsDomainFor: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-N
msDS-IsDomainFor: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-N
msDs-masteredBy: CN=NTDS Settings,CN=SGFS3,CN=Servers,CN=Default-First-Site-Na
msDs-masteredBy: CN=NTDS Settings,CN=SGFS2,CN=Servers,CN=Default-First-Site-Na
root@ucs1:/home/dmadmin#
root@ucs1:/home/dmadmin# /usr/share/univention-samba4/scripts/purge_s4_computer.py --computername=SGFS2.sgvfr.lan
Samba 4 computer account 'SGFS2.sgvfr.lan' not found.
root@ucs1:/home/dmadmin# /usr/share/univention-samba4/scripts/purge_s4_computer.py --computername=SGFS2
Samba 4 computer account 'SGFS2' not found.
#4

Hey,

well… I don’t have a nice out-of-the-box solution for that. What I’d do now is:

  • to comb through the S4 LDAP looking for either of those two removed servers
  • remove all objects that clearly reference one of those two servers and no other server (e.g. CN=Public Folder Store (SGFS2),CN=First Storage Group,… might be one of those) via ldapdelete,
  • for all other objects that include references to the two removed servers but also to other servers: remove the attributes that reference the removed servers via ldapmodify and corresponding LDIF files (e.g. masteredBy: CN=NTDS Settings,CN=SGFS3,… is likely such a case — here you don’t want to delete the whole object that contains said masteredBy attribute, but only the masteredBy attributes that mention both servers),
  • see if that changes things

Kind regards,
mosu

#5

First, the WERR_FILE_NOT_FOUND error means that the server cannot be contacted (like everything’s a file in unix…)

Then, from the output of your LDAP search against Samba/AD you can see that - amongst other things - there are still “NTDS Settings” objects in the CN=Configuration “partition” of the Samba/AD. This would be a more precise search:

univention-s4search --cross-ncs objectClass=nTDSDSA objectGUID

Now, the purge_s4_computer script says that the corresponding computer account object cannot be found any more. Maybe it has been removed before by other means? A quick look into the code of that script shows that it first looks for (&(objectClass=computer)(sAMAccountName=SGFS2$)) and that seems to be gone already, which currently causes the script to assume that there is nothing else to do. Maybe we could improve that script in the future, to also cover your case. In the mean time you may check the output of the following command:

univention-s4search --cross-ncs (&(objectClass=server)(CN=SGFS2))" serverReference \
                              | ldapsearch-wrapper

Then you may check if the object DN listed in serverReference is really already deleted (to confirm my brief analysis based on the script code). If it is gone and you are really sure that you also want to cleanup the corresponding objectClass=server object, then you may consider to do it like this:

ldbdel -H /var/lib/samba/private/sam.ldb \
  CN=SGFS2,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=sgvfr,DC=lan \
  --recursive

After that, you may want to tell Samba/AD to re-calculate the replication topology, even though that should also happen automatically after a couple of minutes:

samba-tool drs kcc

You should also check for and clean up leftover DNS records:

/usr/share/univention-samba4/scripts/check_essential_samba4_dns_records.sh | grep -i SGFS2

That’s all that I can think of right now, I hope it helps.
Cheers,
Arvid

1 Like
#6

Hi @sgvfr, I’m exactly in your situation. I did an AD takeover from Zentyal and now I have a warning on UCS System Diagnostic similar to yours.

`samba-tool drs showrepl` returned a problem with the replication.
Inbound 'DC=ForestDnsZones,DC=intranet,DC=lan': error during DRS replication from Default-First-Site-Name/MOJITO (WERR_FILE_NOT_FOUND)
Inbound 'DC=DomainDnsZones,DC=intranet,DC=lan': error during DRS replication from Default-First-Site-Name/MOJITO (WERR_FILE_NOT_FOUND)
Inbound 'CN=Configuration,DC=intranet,DC=lan': error during DRS replication from Default-First-Site-Name/MOJITO (WERR_FILE_NOT_FOUND)
Inbound 'DC=intranet,DC=lan': error during DRS replication from Default-First-Site-Name/MOJITO (WERR_FILE_NOT_FOUND)
Inbound 'CN=Schema,CN=Configuration,DC=intranet,DC=lan': error during DRS replication from Default-First-Site-Name/MOJITO (WERR_FILE_NOT_FOUND)

@Moritz_Bunkus I tried with:

/usr/share/univention-samba4/scripts/purge_s4_computer.py --computername=MOJITO
Samba 4 computer account 'MOJITO' not found.

@requate I tried with:

univention-s4search --cross-ncs "(&(objectClass=server)(CN=MOJITO))" serverReference \ | ldapsearch-wrapper
# record 1
dn: CN=MOJITO,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=intranet,DC=lan

# returned 1 records
# 1 entries
# 0 referrals

And after with:

 ldbdel -H /var/lib/samba/private/sam.ldb \ CN=MOJITO,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=intranet,DC=lan \ recursive
delete of ' CN=MOJITO,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=intranet,DC=lan' failed - (Not allowed on non-leaf) subtree_delete: Unable to delete a non-leaf node (it has 1 children)!
delete of '(null)' failed - (Invalid DN syntax) ldb_delete: invalid dn '(null)'

Now my question is: how can I delete MOJITO and his children?

(I saw here a similar bug I didn’t understand very well https://forge.univention.org/bugzilla/show_bug.cgi?id=33882)

#7

Hey,

you can add -r to the ldbdel command to let it delete recursively.

Kind regards,
mosu

#8

Thanks @Moritz_Bunkus

ldbdel -r -H /var/lib/samba/private/sam.ldb \ CN=MOJITO,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=intranet,DC=lan
Deleted 2 records

worked perfectly!

Problem solved! Thanks again.

#9

@requate

Excellent! thank you so much, that was exactly what needed to be done.

I am so glad I switched to UCS… . Zentyal support was non-existent.

Mastodon