KDC Erreichbarkeit kritisch

lebernd · February 19, 2018, 2:20pm

Dear Community,

I’m having trouble with reaching the KDC server on an Openvpn setup: local UCS-Master (with Samba4), vserver UCS-Slave (without Samba).
The system diagnostic page is showing me two errors

The KDC isn’t reachable under tcp/udp master.domain.name:88
Checking Kerberos DNS Updates: Fehler traten auf bei der Ausführung von kinit oder nsupdate.
kinit für den Principal slave-host$ mit der Password Datei /etc/machine.secret ist fehlgeschlagen.

I have tried to fix this by following a renewal todo for /etc/machine.secret but that went wrong and so I did a rejoin running univention-join. When I run this command from the UCS-slave I get an initial fail and a success on the second try. After I rejoin the system I have to adjust some network settings as the system will join with my.public.ip.adr. My network looks like

root@slave-host:~# ip a           
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1         
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00                                            
    inet 127.0.0.1/8 scope host lo                         
       valid_lft forever preferred_lft forever                                             
    inet6 ::1/128 scope host                                                                     
       valid_lft forever preferred_lft forever                                       
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000       
    link/ether MAC brd ff:ff:ff:ff:ff:ff                         
    inet my.public.ip.adr brd my.public.ip.255 scope global eth0  
       valid_lft forever preferred_lft forever             
    inet6 fe80:some::thing/64 scope link  
       valid_lft forever preferred_lft forever 
3: eth0.vlannr@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000          
    link/ether MAC brd ff:ff:ff:ff:ff:ff     
    inet my.local.ip.adr/31 brd my.local.ip.adr scope global eth0.vlannr    
       valid_lft forever preferred_lft forever
    inet6 fe80:some::thing/64 scope link
       valid_lft forever preferred_lft forever  
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100   
    link/none   
    inet my.openvpn.ip.adr peer master.openvpn.ip.adr/32 scope global tun0   
       valid_lft forever preferred_lft forever   
    inet6 fe80::f4a1:4e6c:599c:12c4/64 scope link flags 800  
       valid_lft forever preferred_lft forever   
5: docker0: ....

On the UCS-Master I don’t have any issues when running the system diagnostics.
I can search ldap, I can ping the master, samba4 is running on the master:

root@slave-host:~# nmap master.domain.name -p 88                          
                                                                                                                        
Starting Nmap 6.47 ( http://nmap.org ) at 2018-02-19 15:17 CET                                                         
Nmap scan report for master.domain.name (master.local.ip.adr)                                                          
Host is up (0.029s latency).                                                                                            
PORT   STATE SERVICE                                                                                                    
88/tcp open  kerberos-sec                                                                                               
                                                                                                                        
Nmap done: 1 IP address (1 host up) scanned in 0.58 seconds

Kind regards,
Bernd

knebb · February 20, 2018, 8:09am

Hi,

yes, sounds someway strange. I do not have a clue either but for troubleshooting purposes I guess we should check some things.

With nmap you tested TCP. What about UDP (-sU)?
Did you test with the VPN-IPs? Btw: what VPN setup do you use? Direct connection to the master or do you have a firewall acting as VPN-Server (and so bridging the full local network)? Could you post the routing table on your slave (and master)?

And please add the output of the following commands:

kinit --keytab="/var/lib/samba/private/dns.keytab" dns-$(hostname) || echo $?

echo -ne "server $(ucr get ldap/master)\nprereq yxdomain $(hostname -f)\nsend\n" | nsupdate -d -g -t15

/KNEBB

lebernd · February 20, 2018, 8:47pm

Hello again knebb

I did’t post the UDP because I was quite sure of the result:

PORT   STATE         SERVICE              
88/udp open|filtered kerberos-sec

The kinit command is working on the master, giving a ticket. On the slave of course not - there is no samba.
The second command is working too - anything special to look for? I think the entries are all master related having a lot of key-strings in it…

When I change the KDC in the registry on the slave to the slave, I can run:

root@slave-host:~# kinit --password-file=/etc/machine.secret $(hostname)\$@$(ucr get kerberos/realm)          
root@slave-host:~# klist                        
Credentials cache: FILE:/tmp/krb5cc_0         
        Principal: slave-host/slave-host.myucs.domain.name@MYUCS.DOMAIN.NAME   
                                                            
  Issued                Expires               Principal                                                                
Feb 20 21:36:39 2018  Feb 21 07:36:37 2018  krbtgt/MYUCS.DOMAIN.NAME@MYUCS.DOMAIN.NAME

Changing back to the master it fails:

kinit: krb5_get_init_creds: Client (slave-host$@MYUCS.DOMAIN.NAME) unknown

The setup is a server2server configuration: slave is connecting to the master, both running openvpn. So the route is vpn-based, but as no other services fail, I don’t think it has something to do with the network and more with the AD/LDAP on one side and LDAP on the other?! (But I can post more routes if you insist… )

Thank you for helping,
Bernd

Moritz_Bunkus · February 21, 2018, 7:58am

Moin,

Da auf dem Slave kein Samba4 installiert ist, ist zumindest diese Meldung mit ziemlicher Sicherheit ein false positive. Bedeutet: ja, kinit mit einem Maschinenaccount wird nicht funktionieren, weil’s den Maschinenaccount schlicht nicht gibt und spezifisch dieser Test gar nicht ausgeführt werden sollte.

Die »KDC isn’t reachable«-Fehlermeldung ist davon aber unabhängig. Hier muss geprüft werden, ob TCP- und UDP-Verbindungen vom Slave zu dem Host, der im SRV-Record _kerberos._tcp.$domain (siehe host -t srv _kerberos._tcp.$(ucr get domainname)), auf Port 88 möglich sind. Dafür:

Auf beiden Seiten tcpdump entsprechend laufen lassen
Mit z.B. nc Verbindungsaufbau testen
Schauen, wo Pakete rein- und rausgehen
Firewallregeln begutachten
Schauen, wo das VPN intern terminiert und ob da Firewall dazwischen funkt

Gruß
mosu

lebernd · February 21, 2018, 8:12pm

nAbend,

über den false positive -thread bin ich auch schon gestolpert und habe auch v.a. wg. des anderen Fehlers gepostet.

Also:

root@slave:~# host -t srv _kerberos._tcp.$(ucr get domainname)
_kerberos._tcp.domain.name has SRV record 0 100 88 master.domain.name.

Sieht gut aus?!
Und dann, wie poste ich das? Nach hosts getrennt. tcpdump beim Knöpfchen drücken in der Systemdiagnose - KDC Erreichbarbeit.
Slave:

root@slave:~# tcpdump -i any dst master.domain.name and port 88                                                                                                                                                                                      
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:04:43.414122 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [S], seq 2168163237, win 29200, options [mss 1460,sackOK,TS val 257123306 ecr 0,nop,wscale 7], length 0
20:04:43.441788 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [.], ack 535582276, win 229, options [nop,nop,TS val 257123313 ecr 321664509], length 0
20:04:43.443563 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [P.], seq 0:177, ack 1, win 229, options [nop,nop,TS val 257123313 ecr 321664509], length 177
20:04:43.473150 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [.], ack 180, win 237, options [nop,nop,TS val 257123321 ecr 321664517], length 0
20:04:43.475365 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [R.], seq 177, ack 180, win 237, options [nop,nop,TS val 257123321 ecr 321664517], length 0
20:04:43.483370 IP slave.vpn.ip.adr.35999 > master.domain.name.kerberos:  v5

root@slave:~# tcpdump -i any dst master.domain.name and port 88
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:28:18.403821 IP slave.vpn.ip.adr.42862 > master.domain.name.kerberos: Flags [.], ack 190811325, win 229, options [nop,nop,TS val 257477053 ecr 322018250], length 0
20:28:18.404839 IP slave.vpn.ip.adr.42862 > master.domain.name.kerberos: Flags [P.], seq 0:177, ack 1, win 229, options [nop,nop,TS val 257477054 ecr 322018250], length 177
20:28:18.434153 IP slave.vpn.ip.adr.42862 > master.domain.name.kerberos: Flags [.], ack 180, win 237, options [nop,nop,TS val 257477061 ecr 322018257], length 0
20:28:18.435062 IP slave.vpn.ip.adr.42862 > master.domain.name.kerberos: Flags [R.], seq 177, ack 180, win 237, options [nop,nop,TS val 257477061 ecr 322018257], length 0
20:28:18.460885 IP slave.vpn.ip.adr.45731 > master.domain.name.kerberos:  v5
^C
5 packets captured
6 packets received by filter
0 packets dropped by kernel

Master:

root@master:~# tcpdump -w kerbmaster.dump -i any dst port 88
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
^C6 packets captured
12 packets received by filter
0 packets dropped by kernel
root@master:~# tcpdump -r kerbmaster.dump 
reading from file kerbmaster.dump, link-type LINUX_SLL (Linux cooked)
20:04:43.425688 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [S], seq 2168163237, win 29200, options [mss 1349,sackOK,TS val 257123306 ecr 0,nop,wscale 7], length 0
20:04:43.453314 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [.], ack 535582276, win 229, options [nop,nop,TS val 257123313 ecr 321664509], length 0
20:04:43.454869 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [P.], seq 0:177, ack 1, win 229, options [nop,nop,TS val 257123313 ecr 321664509], length 177
20:04:43.484586 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [.], ack 180, win 237, options [nop,nop,TS val 257123321 ecr 321664517], length 0
20:04:43.486935 IP slave.vpn.ip.adr.40508 > master.domain.name.kerberos: Flags [R.], seq 177, ack 180, win 237, options [nop,nop,TS val 257123321 ecr 321664517], length 0
20:04:43.521854 IP slave.vpn.ip.adr.35999 > master.domain.name.kerberos:  v5

root@master:~# tcpdump -i any src port 88
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:28:18.417235 IP master.domain.name.kerberos > slave.vpn.ip.adr.42862: Flags [.], ack 1413791447, win 235, options [nop,nop,TS val 322018257 ecr 257477054], length 0
20:28:18.419215 IP master.domain.name.kerberos > slave.vpn.ip.adr.42862: Flags [P.], seq 0:179, ack 1, win 235, options [nop,nop,TS val 322018257 ecr 257477054], length 179
20:28:18.494722 IP master.domain.name.kerberos > slave.vpn.ip.adr.45731:

@knebb also auch die route:

root@slave:~# ip route get "$(dig master.domain.name +short)"                                                                                                                                                                      
master.eth0.ip.adr via master.vpn.ip.adr dev tun0  src slave.vpn.ip.adr 
    cache 
root@slave:~# traceroute master.domain.name
traceroute to master.domain.name (master.eth0.ip.adr), 30 hops max, 60 byte packets
 1  master.domain.name (master.eth0.ip.adr)  29.306 ms  35.628 ms  35.621 ms

Mein erster Gebrauch von tcpdump - also ich hoffe ich habe das einigermaßen richtig benutzt… Any hint to the error already?

Kind regards,
Bernd

knebb · February 22, 2018, 7:39am

Hi,

yes, nearly perfect for first use

Next time, it is helpful to have full communication on a single dump. So instead af “dst port 88” I would prefer “port 88”- then you will see the replies, too. But that’s just my gust.

May I recall the issue?
The system diagnostic tells you:
* The KDC isn’t reachable under tcp/udp master.domain.name:88
* Checking Kerberos DNS Updates: Fehler traten auf bei der Ausführung von kinit oder nsupdate.
* kinit für den Principal slave-host$ mit der Password Datei /etc/machine.secret ist fehlgeschlagen.

And on your slave there is no samba4 installed?

This indeed looks like a mislead error message- the system check script tries to get a Kerberos ticket. But as there is no samba installed it fails. Which does not matter as you do not need Samba.I am unsure but there mightt be and additional check implemented in UCS 4.3: “Only check this if Samba is installed”.

From what you have posted all routes, network settings and firewall rules seem to be fine.

Besides of this do you have any issues?

/KNEBB

lebernd · February 22, 2018, 4:00pm

Hi,

I would rather go along with @Moritz_Bunkus that only the last issue - meaning the last two on your list are false positive, but I actually don’t see much trouble resulting from the KDC issue right now.
But also - I don’t have this issue on another domain, where all servers are in the same LAN. (the dns-kerberos issue too is listed there)

I don’t know if it is or may be related, but I have a question about the primary interface on the slave host and the impact to the join process. As mentioned in the first post

After I rejoin the system I have to adjust some network settings as the system will join with my.public.ip.adr.

Leaving the primary interface untouched, the slave will join with the public-IP address and the first external DNS will be overwritten by the master value. Which of course makes no sense for the slave-host. I’m a little afraid to lose the internet-connection for the slave host by choosing the vlan-interface for the primary interface as it has no internet-gateway. But perhaps that’s the way to go and just edit/ register something for /etc/network/interfaces?

Thanks,
Bernd

lebernd · February 22, 2018, 4:52pm

well, hm:
The funny thing is, that kerberos (heimdal-kerberos) is working on the slave just fine (although not listed as running):

root@slave:~# kinit Administrator
Administrator@DOMAIN.NAME's Password: 
root@slave:~# klist
Credentials cache: FILE:/tmp/krb5cc_0
        Principal: Administrator@DOMAIN.NAME

  Issued                Expires               Principal
Feb 22 17:43:46 2018  Feb 23 03:43:41 2018  krbtgt/DOMAIN.NAME@DOMAIN.NAME

Slave-role servers seam to not get a host-entry on the master samba4 server (unlike member-servers). Checking with:

univention-s4search '(|(userPrincipalName=*)(servicePrincipalName=*))' userPrincipalName servicePrincipalName

on both domains. This may make sense as they are supposed to work ‘more independent’ from master-servers (local LDAP queries).
But anyway - strange error - as it is not shown on the LAN.

Still thinking about it,
Bernd

Gohmann · February 22, 2018, 6:46pm

Maybe it is a bug in the check script? See here:
https://forge.univention.org/bugzilla/show_bug.cgi?id=46266

lebernd · February 23, 2018, 10:20am

Thank you for the feedback.
I have looked at this before, but it was out of range for my tcp-knowledge :humble:
Ha missing icon !

Thank you,
Bernd