4.2-1 upgrade Samba4 DNS reverse lookup zone has broken (corrupted?) PTR records

openldap
dns
ucs-4-2
s4-connector

#1

Found a new issue after upgrade to 4.2-1.

This one seems related to DNS resolution. DNS information in the OpenLDAP directory appears to be correct, however reverse lookup zones in samba4 DNS under RSAT show badly formatted records (nonsense extra octet in IP address - which confuses RSAT). Assuming came with upgrade (will check logs) as I wouldn’t think RSAT input validation would allow a bad IP entry and we haven’t edited any entries after upgrade.

Additionally these PTR records are not being synced by the s4-connector.

cat /var/log/univention/connector-s4-status.log
Mon Jul 10 19:04:10 2017
--------------------------------------
try to sync 0 changes from UCS
done:
Changes from UCS: 0 (0 saved rejected)
--------------------------------------
--------------------------------------
try to sync 0 changes from S4
done:
Changes from S4:  0 (8 saved rejected)
--------------------------------------
--------------------------------------
Sync 0 rejected changes from UCS
restored 0 rejected changes
--------------------------------------
--------------------------------------
Sync 8 rejected changes from S4 to UCS
restored 0 rejected changes
--------------------------------------
- sleep 5 seconds (0/10 until resync) -
 univention-s4connector-list-rejected

UCS rejected


S4 rejected

    1:    S4 DN: DC=11,DC=20.20.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: relativedomainname=11,zonename=20.20.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
    2:    S4 DN: DC=@,DC=30.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=30.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
    3:    S4 DN: DC=@,DC=40.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=40.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
    4:    S4 DN: DC=@,DC=anotherreplaceddomain,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=anotherreplaceddomain,cn=dns,DC=removedrealdomain,dc=com,dc=au
    5:    S4 DN: DC=@,DC=display.anotherreplaceddomain.com.au,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=display.anotherreplaceddomain.com.au,cn=dns,DC=removedrealdomain,dc=com,dc=au
    6:    S4 DN: DC=@,DC=testing.anotherreplaceddomain.com.au,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=testing.anotherreplaceddomain.com.au,cn=dns,DC=removedrealdomain,dc=com,dc=au
    7:    S4 DN: DC=7,DC=20.20.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: <not found>
    8:    S4 DN: DC=7,DC=20.20.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: <not found>

        last synced USN: 299387

samba-tool dns query seems OK?

# samba-tool dns query dcmaster 20.10.in-addr.arpa 20 ALL  -U administrator
Password for [administrator]:
  Name=, Records=0, Children=0
  Name=10, Records=1, Children=0
    PTR: dcmaster.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=11, Records=0, Children=0
  Name=12, Records=0, Children=0
  Name=13, Records=0, Children=0
  Name=14, Records=0, Children=0
  Name=15, Records=0, Children=0
  Name=17, Records=0, Children=0
  Name=18, Records=0, Children=0
  Name=19, Records=0, Children=0
  Name=2, Records=0, Children=0
  Name=20, Records=0, Children=0
  Name=21, Records=0, Children=0
  Name=22, Records=0, Children=0
  Name=23, Records=0, Children=0
  Name=24, Records=0, Children=0
  Name=25, Records=0, Children=0
  Name=26, Records=0, Children=0
  Name=27, Records=0, Children=0
  Name=28, Records=0, Children=0
  Name=29, Records=1, Children=0
    PTR: device1.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=30, Records=1, Children=0
    PTR: device2.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=32, Records=1, Children=0
    PTR: device3.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=33, Records=1, Children=0
    PTR: device4.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=34, Records=1, Children=0
    PTR: device5.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=35, Records=1, Children=0
    PTR: device6.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=36, Records=1, Children=0
    PTR: device7.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=37, Records=1, Children=0
    PTR: device8.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=38, Records=1, Children=0
    PTR: device9.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=39, Records=1, Children=0
    PTR: device10.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=5, Records=0, Children=0
  Name=6, Records=1, Children=0
    PTR: device11.replaceddomain.com.au (flags=f0, serial=1, ttl=900)
  Name=7, Records=1, Children=0
    PTR: device12.replaceddomain.com.au (flags=f0, serial=1, ttl=900)

nslookup from linux and windows give same result

nslookup 10.20.20.6
Server:         10.20.20.10
Address:        10.20.20.10#53

** server can't find 6.20.20.10.in-addr.arpa: NXDOMAIN

Forward lookups work though.

So it seems the Samba4 has had some entries corrupted or synced badly and now it and Openldap/bind are out of sync.

How do I manually edit one side to match the other to allow sync to re-establish? Ideally I’d like to accept the Openldap reverse zone as accurate and overwrite the samba4 one.


One side effect of the above is that I think this has affected the univention-mount-homedir script that runs on a 10 min cronjob (and via pam common-session I think). Every time it runs it seems it failed on resolving the DNS name of the fileserver and hung the mount.nfs call in ‘D’ process state as it can’t resolve the PTR records for the fileserver.

This manifested as high load avg numbers (200+) with low CPU usage and built up over the weekend after the upgrade until all memory and swap space ran out this morning and the server went completely unresponsive. Additionally I think the OOM killer shutdown named and other UCS services rather than the hundreds of mount.nfs in ‘D’ state so that everything gradually stopped working.

Since rebooting both DCs I’ve made sure the process trees launching that script don’t execute and the graphs have returned to normal. I haven’t yet worked out what part of UCS is causing user homedirs to be mounted on the DCs? I don’t have any NFS mount policies and don’t really understand why user homedirs would need to be mounted on a DC.

So we’re limping along with S4 DNS in a degraded state but would really like to sort this out in case something like kerberos timeouts or machine password renewals or some other active directory function that needs good DNS starts to go wrong.

Hope someone can help.


Ghost DNS PTR in RSAT that show up in UCS UDM - no rejects - suspect dnsRecord RANK flags wrong
#2

Tried modifying a PTR record in UCS DNS and watched the S4 connector sync it to samba4 DNS.

It shows up in RSAT with the extra octet in the name 10.255.0.0.1 instead of 10.255.0.1. Also, looks like I can’t delete these strange objects with RSAT (“The record cannot be deleted. The record does not exist”).

Looking at the DomainDNSZones naming context in ADSI it looks like the name and dc, attributes seem to be correct.

I wonder if the binary dnsRecord attribute is getting written by samba or the s4-connector incorrectly? It would need some binary checking referenced against the protocol docs to find out. Hoping you guys can see a bug there though before I go to that effort.


#3

Hey,

In our domain (4.2-1, too) reverse lookups continue to work just fine fortunately, so it’s not a general issue all UCS users hit.

I assume that your DNS backend is set to Samba 4, right? Please check with ucr get dns/backend which should output samba4.

Next try resolving one of those S4 rejects by re-syncing that entry from the UCS side to the S4 side. This SDB entry shows you how to do that. Be sure to sync in the correct direction; you’ll need to use the resync_object_from_ucs.py script with a UCS DN for one of the affected entries.

Then watch the connector log file /var/log/univention/connector-s4.log (not the …-status.log one) for the result. Post any error message here, please.

Kind regards,
mosu


#4

Hi Moritz, yes it is samba4 as the backend.

I’ve discovered that part of it is the RSAT MMC has some sort of cosmetic display issue. When displaying a reverse zone with subfolders for different subnets I get the bad IP address formatting. All I have to do is to refresh the view on the reverse zone folder and the subfolders disappear and the entries display correctly. Great.

I thought it might have been something to do with the win10 RSAT tools interacting with samba4, but trying it out on an old win7 with RSAT that previously worked with UCS 4.1-4 also displays the same behaviour so I wonder if its something to do with the RPC from samba4 back to the RSAT? But as you say you’re not seeing this issue?

So now I have RSAT displaying some entries correctly but under one reverse zone I still only have one PTR record visible while in ADSI edit all 6 or so PTRs are there (and reverse lookups work for the missing entries).

I used the univention ldap search utilities to compare some of the entries and as far as I can tell the decoded DNS record is correctly stored in samba4.

So seems like two issues, directories out of sync or s4 is not displaying DNS properly and this cosmetic RSAT issue that seems to have a workaround.

I will try your suggested actions and let you know how it goes.

Thanks.


#5

Haven’t yet upgraded to the latest errata, but I do wonder if this has anything to do with my PTR issues. WIll be upgrading this afternoon as part of the SSL issue you’ve helped me with too; so we’ll see.
https://forge.univention.org/bugzilla/show_bug.cgi?id=43072


#6

I’ve seen and read the announcement for the errata update yesterday and thought of your issue, too, but it didn’t seem to fit exactly. However, it sure won’t hurt to update; fewer bugs are always better :wink:


#7

I do have some rejected entries

univention-s4connector-list-rejected

UCS rejected


S4 rejected

    1:    S4 DN: DC=@,DC=30.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=30.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
    2:    S4 DN: DC=@,DC=40.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=40.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
    3:    S4 DN: DC=@,DC=brisbane.anotherreplaceddomain.local,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=brisbane.anotherreplaceddomain.local,cn=dns,DC=removedrealdomain,dc=com,dc=au
    4:    S4 DN: DC=@,DC=display.removedrealdomain.com.au,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=display.removedrealdomain.com.au,cn=dns,DC=removedrealdomain,dc=com,dc=au
    5:    S4 DN: DC=@,DC=testing.removedrealdomain.com.au,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
         UCS DN: zonename=testing.removedrealdomain.com.au,cn=dns,DC=removedrealdomain,dc=com,dc=au

        last synced USN: 375939

The logs look the same for each.

04.08.2017 16:37:51,499 LDAP        (PROCESS): sync to ucs: Resync rejected dn: DC=@,DC=testing.removedrealdomain.com.au,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
04.08.2017 16:37:51,507 LDAP        (PROCESS): sync to ucs:   [           dns] [    modify] zonename=testing.removedrealdomain.com.au,cn=dns,DC=removedrealdomain,dc=com,dc=au
04.08.2017 16:37:51,513 LDAP        (ERROR  ): Unknown Exception during sync_to_ucs
04.08.2017 16:37:51,514 LDAP        (ERROR  ): Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/s4connector/__init__.py", line 1543, in sync_to_ucs
    result = self.property[property_type].ucs_sync_function(self, property_type, object)
  File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/dns.py", line 1682, in con2ucs
    ucs_zone_create(s4connector, object, dns_type)
  File "/usr/lib/pymodules/python2.7/univention/s4connector/s4/dns.py", line 1436, in ucs_zone_create
    zone.modify()
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 347, in modify
    dn = self._modify(modify_childs, ignore_license=ignore_license, response=response)
  File "/usr/lib/pymodules/python2.7/univention/admin/handlers/__init__.py", line 857, in _modify
    self.lo.modify(self.dn, ml, ignore_license=ignore_license, serverctrls=serverctrls, response=response)
  File "/usr/lib/pymodules/python2.7/univention/admin/uldap.py", line 505, in modify
    raise univention.admin.uexceptions.ldapError(_err2str(msg), original_exception=msg)
ldapError: Type or value exists: nSRecord: value #0 provided more than once

ldapError: Type or value exists: nSRecord: value #0 provided more than once


#8

Interesting. Please show the output of the following:

univention-ldapsearch zonename=30.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
univention-s4search DC=@,DC=30.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au

Thanks.


#9

Here you go.

univention-ldapsearch zonename=30.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
# extended LDIF
#
# LDAPv3
# base <DC=removedrealdomain,dc=com,dc=au> (default) with scope subtree
# filter: zonename=30.10.in-addr.arpa,cn=dns,DC=removedrealdomain,dc=com,dc=au
# requesting: ALL
#

# search result
search: 3
result: 0 Success

# numResponses: 1
univention-s4search DC=@,DC=30.10.in-addr.arpa,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au
# Referral
ref: ldap://removedrealdomain.com.au/CN=Configuration,DC=removedrealdomain,DC=com,DC=au

# Referral
ref: ldap://removedrealdomain.com.au/DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au

# Referral
ref: ldap://removedrealdomain.com.au/DC=ForestDnsZones,DC=removedrealdomain,DC=com,DC=au

# returned 3 records
# 0 entries
# 3 referrals

When I look at the UMC and the RSAT there are different entries under each interface for the PTR records.

UMC has PTRs for 10.30.20.5,10.30.20.7,10.30.20.10,10.30.20.11

While RSAT has only PTRs for 10.30.20.10,10.30.20.11 visible in the DNS console, but all entries visible in UMC appear when looking at the S4 LDAP directly with ADSI edit.

Reverse lookups also work for the missing entries (10.30.20.5,10.30.20.7) which I assume are coming via samba4 and not Bind as that is the DNS backend.

So I wonder again if there is some bug with either the storage of the DNS records, or the way it feeds it via RPC to the DNS RSAT that causes those entries to be responded to by the DNS queries, but not to display properly in the RSAT DNS interface?

So these multiple overlapping issues in my various forum threads (SSL, SAML auth, Directories out of sync, PTRs, S4 sync) all have me scratching my head about what happened during the upgrade. Thankfully though UCS seems robust enough that the end-user visible functionality (logins, new users, moving LDAP objects) is still working, its only me being hampered so far.


#10

That entries aren’t visible in UMC is understandable when the sync of the PTR zones themselves isn’t working. You should try telling the S4 connector to resync the objects from Samba4 to UCS by running the following:

/usr/share/univention-s4-connector/resync_object_from_s4.py DC=@,DC=testing.removedrealdomain.com.au,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au

Then watch the S4 connector’s log file and post what happens with that object within the next minute.

Please also post the output of the following:

univention-s4search DC=@,DC=testing.removedrealdomain.com.au,CN=MicrosoftDNS,DC=DomainDnsZones,DC=removedrealdomain,DC=com,DC=au