DNS problem, after upgrade

Moritz_Bunkus · March 29, 2018, 11:17am

Hey,

what are the actual error messages in /var/log/univention/connector-s4.log for those entries?

Kind regards,
mosu

codedmind · March 29, 2018, 11:18am

tail -f /var/log/univention/connector-s4.log
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 326, in delete_ext_s
    resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all=1,timeout=self.timeout)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 514, in result3
    resp_ctrl_classes=resp_ctrl_classes
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 521, in result4
    ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  File "/usr/lib/python2.7/dist-packages/ldap/ldapobject.py", line 106, in _ldap_call
    result = func(*args,**kwargs)
NOT_ALLOWED_ON_NONLEAF: {'info': '00002015: subtree_delete: Unable to delete a non-leaf node (it has 27 children)!', 'desc': 'Operation not allowed on non-leaf'}
29.03.2018 12:19:15,873 LDAP        (PROCESS): sync from ucs:   Resync rejected file: /var/lib/univention-connector/s4/1522319929.837764
29.03.2018 12:19:15,881 LDAP        (PROCESS): sync from ucs: [           dns] [    delete] DC=@,DC=ccm.local,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local
29.03.2018 12:19:15,901 LDAP        (WARNING): sync failed, saved as rejected

Moritz_Bunkus · March 29, 2018, 11:32am

Hey,

wow, that looks like your whole DNS structure has been removed on one side. Please post the output of the following:

univention-ldapsearch -b zoneName=$(ucr get domainname),cn=dns,$(ucr get ldap/base) dn|ldapsearch-wrapper
univention-s4search --cross-ncs dc=@ dn

Kind regards,
mosu

codedmind · March 29, 2018, 11:35am

Yes… i’m trying solve the issue of kerberos critical like in this post (Critical: Check kerberos authenticated DNS update (on DC Master))

First command

# extended LDIF
#
# LDAPv3
# base <zoneName=ccm.local,cn=dns,dc=ccm,dc=local> with scope subtree
# filter: (objectclass=*)
# requesting: dn
#

# ccm.local, dns, ccm.local
dn: zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# CCMDC01, ccm.local, dns, ccm.local
dn: relativeDomainName=CCMDC01,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# feldc01, ccm.local, dns, ccm.local
dn: relativeDomainName=feldc01,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _gc._tcp, ccm.local, dns, ccm.local
dn: relativeDomainName=_gc._tcp,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# gc._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=gc._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _kpasswd._tcp, ccm.local, dns, ccm.local
dn: relativeDomainName=_kpasswd._tcp,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _kpasswd._udp, ccm.local, dns, ccm.local
dn: relativeDomainName=_kpasswd._udp,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _kerberos._tcp, ccm.local, dns, ccm.local
dn: relativeDomainName=_kerberos._tcp,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _kerberos._udp, ccm.local, dns, ccm.local
dn: relativeDomainName=_kerberos._udp,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# DomainDnsZones, ccm.local, dns, ccm.local
dn: relativeDomainName=DomainDnsZones,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# ForestDnsZones, ccm.local, dns, ccm.local
dn: relativeDomainName=ForestDnsZones,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.dc._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.dc._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.gc._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.gc._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _kerberos._tcp.dc._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=_kerberos._tcp.dc._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.DomainDnsZones, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.DomainDnsZones,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.ForestDnsZones, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.ForestDnsZones,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _gc._tcp.Default-First-Site-Name._sites, ccm.local, dns, ccm.local
dn: relativeDomainName=_gc._tcp.Default-First-Site-Name._sites,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.Default-First-Site-Name._sites, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.Default-First-Site-Name._sites,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# 5150540b-6efc-4159-abb7-f4452288f2e2._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=5150540b-6efc-4159-abb7-f4452288f2e2._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _kerberos._tcp.Default-First-Site-Name._sites, ccm.local, dns, ccm.local
dn: relativeDomainName=_kerberos._tcp.Default-First-Site-Name._sites,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.Default-First-Site-Name._sites.dc._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.Default-First-Site-Name._sites.dc._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.Default-First-Site-Name._sites.gc._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.Default-First-Site-Name._sites.gc._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=_kerberos._tcp.Default-First-Site-Name._sites.dc._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.Default-First-Site-Name._sites.DomainDnsZones,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# _ldap._tcp.17435187-b154-4c14-a46f-69ab309d1823.domains._msdcs, ccm.local, dns, ccm.local
dn: relativeDomainName=_ldap._tcp.17435187-b154-4c14-a46f-69ab309d1823.domains._msdcs,zoneName=ccm.local,cn=dns,dc=ccm,dc=local

# search result
search: 3
result: 0 Success

# numResponses: 28
# numEntries: 27

Second

univention-s4search --cross-ncs dc=@ dn
# record 1
dn: DC=@,DC=ccm.local,CN=MicrosoftDNS,DC=DomainDnsZones,DC=ccm,DC=local

# record 2
dn: DC=@,DC=RootDNSServers,CN=MicrosoftDNS,DC=DomainDnsZones,DC=ccm,DC=local

# record 3
dn: DC=@,DC=_msdcs.ccm.local,CN=MicrosoftDNS,DC=ForestDnsZones,DC=ccm,DC=local

# record 4
dn: DC=@,DC=RootDNSServers,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 5
dn: DC=@,DC=motasc.gogest.pt,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 6
dn: DC=@,DC=esxi.local,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 7
dn: DC=@,DC=vcenter6.esxi.local,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 8
dn: DC=@,DC=120.168.192.in-addr.arpa,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 9
dn: DC=@,DC=100.168.192.in-addr.arpa,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 10
dn: DC=@,DC=unifi.int.mota-sc.com,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# returned 10 records
# 10 entries
# 0 referrals

In system diagnostic right now besides the samba replication and rejects i have kerberos critical warning

Errors occured while running `kinit` or `nsupdate`.
`nsupdate` check for domain ccm.local failed (CCMDC01.ccm.local).
`nsupdate` check for domain ccm.local failed (CCMDC01).

codedmind · March 29, 2018, 11:45am

@Moritz_Bunkus any sugestion how repair this? or what files shoud i restore/compare

Thanks

Moritz_Bunkus · March 29, 2018, 11:53am

Hey,

I’m not too sure about what to do about this. The situation looks like this:

On the S4 side there used to be a domain cmm.local beneath the CN=System sub-tree. That one has been removed. Now the connector tries to remove the domain on the OpenLDAP side, too. This is evident from the output of univention-s4connector-list-rejected and the univention-s4search you did.
On the OpenLDAP side the domain entry itself still contains children and can therefore not be removed. That’s evident from the log file and the univention-ldapsearch you ran. In fact, those entries must not be removed from the OpenLDAP side, otherwise all your DNS entries will be gone.
Now it gets interesting: on the S4 side the domain ccm.local does still exist, albeit under the DC=DomainDnsZones sub-tree — and that’s the location I would have expected the domain to reside under in the first place.

Based on those facts I would probably remove those S4 connector rejects (not the LDAP entries — only the reject files!).

However, before you do that: you said that you have a second DC, running Samba4, too? Can you please post the output of univention-s4search --cross-ncs dc=@ dn from that second DC?

How was the setup created in the first place? By an AD takeover per chance?

Afterwards we can tackle the Kerberos problem.

codedmind · March 29, 2018, 11:59am

@Moritz_Bunkus

Here the result from the comand in the second DC, is a dc that is in other site

FELDC01:~# univention-s4search --cross-ncs dc=@ dn
# record 1
dn: DC=@,DC=ccm.local,CN=MicrosoftDNS,DC=DomainDnsZones,DC=ccm,DC=local

# record 2
dn: DC=@,DC=RootDNSServers,CN=MicrosoftDNS,DC=DomainDnsZones,DC=ccm,DC=local

# record 3
dn: DC=@,DC=_msdcs.ccm.local,CN=MicrosoftDNS,DC=ForestDnsZones,DC=ccm,DC=local

# record 4
dn: DC=@,DC=RootDNSServers,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 5
dn: DC=@,DC=esxi.local,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 6
dn: DC=@,DC=vcenter6.esxi.local,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 7
dn: DC=@,DC=120.168.192.in-addr.arpa,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# record 8
dn: DC=@,DC=100.168.192.in-addr.arpa,CN=MicrosoftDNS,CN=System,DC=ccm,DC=local

# returned 8 records
# 8 entries
# 0 referrals

Yes, the first DC was created almost 3 years ago with ad takeover on a windows server 2003

Moritz_Bunkus · March 29, 2018, 12:02pm

Hey,

it’s possible that the entries for ccm.local beneath CN=System were due to that AD takeover. I think I’d go ahead with the removal of the reject files in this case.

You do have update-to-date backups from the DC, right?

Kind regards,
mosu

codedmind · March 29, 2018, 12:07pm

Yes i have… but the domain from the ad takeover is the same, as previous… CCM, nver was CMM

Just to clarify…

The main DC is a VM and i have daily snapshots. If possible i would like to restore the dns files and not the all the VM

Moritz_Bunkus · March 29, 2018, 12:08pm

Yeah, cmm was a typo on my end. Sorry about that. I always meant to talk about ccm(.local).

codedmind · March 29, 2018, 1:01pm

I don’t know how to do it… right know i’m having issues that i don’t have before… can’t login in intranet (my guest is because ad /dns issues)

Anyone can indicate the files where the dns entrys are stored? in samba side… because from ldap side appear to be ok

Moritz_Bunkus · March 29, 2018, 1:07pm

Hey,

I’ve posted a link above to a support article where removing those rejects is explained.

Kind regards,
mosu

codedmind · March 29, 2018, 1:15pm

But removing the rejects don’t will be destroy even more the dns structure?

Is my understanding that for some reason the dns entries disappear from samba but should stay there… if i remove the rejects that will not interfer with the ldap dns

Moritz_Bunkus · March 29, 2018, 1:20pm

Hey,

you’re only removing the reject files which tell the S4 connector to re-try applying those modifications. You’re not removing the LDAP objects themselves.

Kind regards,
mosu

codedmind · March 29, 2018, 1:29pm

But my rejects are from ldap… or not?

/var/log/univention# univention-s4connector-list-rejected

UCS rejected

    1:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.525067

    2:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.563395

    3:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.575114

    4:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.623042

    5:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.644692

    6:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.656966

    7:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.707915

    8:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.729991

    9:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.742196

   10:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.790098

   11:   UCS DN: zoneName=ccm.local,cn=dns,dc=ccm,dc=local
          S4 DN: dc=@,dc=ccm.local,cn=microsoftdns,cn=system,DC=ccm,DC=local
         Filename: /var/lib/univention-connector/s4/1522319929.837764


S4 rejected


        last synced USN: 278132

Moritz_Bunkus · March 29, 2018, 1:31pm

Hey,

again, the article I’ve linked to tells you how to remove UCS rejects. Only the files where the intended modifications are stored in are removed, not the objects they’re supposed to be applied to.

m.

codedmind · March 29, 2018, 1:40pm

Now i have the following

Found 1 UCS rejects and 0 S4 rejects. See Univention Support Database - How to deal with s4-connector rejects for more information.
UCS rejected:
UCS DN: ;unknown, S4 DN: not found, Filename: /var/lib/univention-connector/s4/.1522319929.525067.swp

Moritz_Bunkus · March 29, 2018, 1:44pm

Hey,

well, don’t use vim in that directory… Exit vim, and the swap file should vanish.

m.

codedmind · March 29, 2018, 1:56pm

I don’t use vim… use nano, and nothing is open…

Moritz_Bunkus · March 29, 2018, 1:58pm

That’s a swap file that’s usually created by vim. If you don’t use vim, maybe someone else does. If no one has vim open on the machine and the file still exists, remove it.