S4 Connector not working

s4-connector
ucs4

#1

History

My homelab had a Windows Server as PDC once (windc1). I migrated this to a physical UCS about two years ago (ucsdc1). Last May, I created a backup DC as a virtual machine inside a Proxmox server, switched off ucsdc1 and promoted the backup DC to PDC (following the steps described in the manual).
Since May 2017 latest, the S4 Connector on ucsdc2 is not working. After I saw this, I did rollbacks to the first snapshots (ucs version 4.2-0 errata15) and pulled the software back up to 4.2-3 errata 256.

Diagnosis

Running the “System-Fehlerdiagnose” (system error diagnosis), I get two issues:

  1. Problem: Unsynchronized S4 Connector Objects
    There is a Traceback and a link to “Univention Support Database - How to deal with s4-connector rejects
  2. Warning: Check File Permissions
    This is easily remedied with chmod 755 /var/cache/univention-samba4

Following the link and the suggested solution for problem #1, I get

root@mypdc:~# univention-s4connector-list-rejected
Failed to get SID from S4: 'objectSid'

This seems to be conclusive with the Traceback, as the S4 Connector service is not running on my system:

root@ucsdc2:~# service univention-s4-connector status -l
● univention-s4-connector.service - LSB: Univention S4 Connector
   Loaded: loaded (/etc/init.d/univention-s4-connector)
   Active: failed (Result: exit-code) since So 2017-12-31 03:26:06 CET; 10min ago
  Process: 2345 ExecStop=/etc/init.d/univention-s4-connector stop (code=exited, status=0/SUCCESS)
  Process: 1533 ExecStart=/etc/init.d/univention-s4-connector start (code=exited, status=0/SUCCESS)
 Main PID: 2343 (code=exited, status=1/FAILURE)

Dez 31 03:26:06 ucsdc2 univention-s4-connector[1533]: Starting Univention S4 Connector: univention-s4-connector.
Dez 31 03:26:06 ucsdc2 systemd[1]: univention-s4-connector.service: Supervising process 2343 which is not our child. We'll most likely not notice when it exits.
Dez 31 03:26:06 ucsdc2 systemd[1]: Started LSB: Univention S4 Connector.
Dez 31 03:26:06 ucsdc2 systemd[1]: univention-s4-connector.service: main process exited, code=exited, status=1/FAILURE
Dez 31 03:26:06 ucsdc2 univention-s4-connector[2345]: Stopping Univention S4 Connector: univention-s4-connectorstart-stop-daemon: warning: failed to kill 2343: No such process
Dez 31 03:26:06 ucsdc2 univention-s4-connector[2345]: .
Dez 31 03:26:06 ucsdc2 systemd[1]: Unit univention-s4-connector.service entered failed state.

I can’t get the connector to start or give me sufficient debug info to tackle the problem.

root@ucsdc2:~# tail -n 30 /var/log/univention/connector-s4-status.log
Sun Dec 31 03:26:06 2017
Failed to get SID from S4: 'objectSid'
root@ucsdc2:~# tail -n 20 /var/log/univention/connector-s4.log
31.12.2017 00:14:52,981 MAIN        (------ ): DEBUG_INIT
31.12.2017 00:14:53,5 LDAP        (PROCESS): Building internal group membership cache
31.12.2017 00:14:53,23 LDAP        (PROCESS): Internal group membership cache was created
31.12.2017 02:25:46,851 MAIN        (------ ): DEBUG_INIT
31.12.2017 02:25:46,906 LDAP        (PROCESS): Building internal group membership cache
31.12.2017 02:25:46,932 LDAP        (PROCESS): Internal group membership cache was created
31.12.2017 02:44:41,101 MAIN        (------ ): DEBUG_INIT
31.12.2017 02:44:41,124 LDAP        (PROCESS): Building internal group membership cache
31.12.2017 02:44:41,130 LDAP        (PROCESS): Internal group membership cache was created
31.12.2017 03:03:47,86 MAIN        (------ ): DEBUG_INIT
31.12.2017 03:03:47,137 LDAP        (PROCESS): Building internal group membership cache
31.12.2017 03:03:47,142 LDAP        (PROCESS): Internal group membership cache was created
31.12.2017 03:13:07,357 MAIN        (------ ): DEBUG_INIT
31.12.2017 03:13:07,381 LDAP        (PROCESS): Building internal group membership cache
31.12.2017 03:13:07,385 LDAP        (PROCESS): Internal group membership cache was created
31.12.2017 03:26:06,309 MAIN        (------ ): DEBUG_INIT
31.12.2017 03:26:06,336 LDAP        (PROCESS): Building internal group membership cache
31.12.2017 03:26:06,353 LDAP        (PROCESS): Internal group membership cache was created
31.12.2017 03:29:25,239 MAIN        (------ ): DEBUG_INIT
31.12.2017 03:33:40,258 MAIN        (------ ): DEBUG_INIT

The Univention System Check (found here) produced coherent results:

root@ucsdc2:~# python ./univention-system-check
running [dns]      - OK       - forward_and_reverse_dns_kdc.sh
running [dns]      - OK       - forward_dns_myself.sh
running [listener] - OK       - all_handlers_initialized.sh
running [listener] - OK       - replication.sh
running [basic]    - OK       - check_for_ntpd_process.sh
running [basic]    - OK       - univention_ldapsearch_machine_basic.sh
running [basic]    - OK       - univention_ldapsearch_machine_kerberos.sh
running [basic]    - OK       - check_nagios_status.py
running [basic]    - OK       - secure_apt_is_activated.sh
running [basic]    - OK       - check_for_dockerd_process.sh
running [basic]    - OK       - joinstatus.sh
running [basic]    - OK       - package_status.sh
running [samba]    - OK       - master_is_member_of_enterprise_domain_controllers.sh
running [samba]    - OK       - check_s4_connector_listener_active.sh
running [samba]    - OK       - disabled_drsuapi_adtakeover_incomplete.sh
running [samba]    - OK       - wbinfo_checks.sh
running [samba]    - OK       - krbtgt_has_rid_502.sh
running [samba]    - OK       - check_guid_msdcs_dns_alias.sh
running [samba]    - OK       - cn_system_exists_only_once.sh
running [samba]    - OK       - testjoin.sh
running [samba]    - FAILED   - hosts_sids_equal_in_ucs_and_samba.sh
running [samba]    - OK       - check_msds_keyversionnumber.sh
running [samba]    - FAILED   - check_s4_connector_autostart.sh
running [samba]    - OK       - maximum_password_age_smaller_999.sh
running [samba]    - FAILED   - check_s4_connector_rejects.sh
running [samba]    - OK       - check_samba_drs_replication.sh
running [samba]    - OK       - check_winbind_idmap_range.sh
running [samba]    - OK       - check_samba_processes.sh
running [samba]    - OK       - check_samba_domain_trust.sh
running [samba]    - OK       - check_smbclient_via_krb5_keytab.sh
running [samba]    - OK       - cn_idmap_exists.sh
running [samba]    - OK       - check_ddns_update.sh
running [samba]    - FAILED   - check_for_temporary_udm_sids.sh
running [samba]    - OK       - no_3000_mapping_in_net_cache.sh

Tests failed: 4
<--- snip (lots of gory details can be provided if this helps the cause) --->

FAILED #2 - check_s4_connector_autostart.sh
Looks like the main problem: the connector immediately and rather silently ceases to function shortly after invocation.

FAILED #1 - hosts_sids_equal_in_ucs_and_samba.sh
FAILED #4 - check_for_temporary_udm_sids.sh
No, the sids in ucs and samba are NOT equal, because all the windows pcs have a proper sid in samba and temporary sids in ucs. I assume that they got rejoined to the new pdc at a time when no connector took care of the synchronization to ucs.

FAILED #3 - check_s4_connector_rejects.sh
This is merely a follow-up failure, probably from the same Traceback as above.

Questions

  1. Is there a way to coerce the S4 Connector into providing more debug data?
  2. What are the steps to make the S4 Connector work again?

#2

Hi,

could you please provide the logfile from univention-system-check

/var/log/univention/univention-system-check.log

#3

Here you go …
univention-system-check.log (132.7 KB)
(alternative upload to pastebin.com)


#4

Hi,

thank you for the logfile.
I now need some more information, to get closer to the real cause of the problem.
Could you please provide the output of the following commands:

univention-s4search -s base objectclass=domain objectSid
ls -ld /var/lib/samba*
ls -l /var/lib/samba*/private/sam.ldb
for sam in /var/lib/samba*/private/sam.ldb; do ldbsearch -H "$sam" 1.1 | grep entries; done

#5

This is the session captured with script:

Script started on Mi 03 Jan 2018 15:15:27 CET
root@ucsdc2:~# univention-s4search -s base objectclass=domain objectSid
# record 1
dn: DC=local,DC=k-family,DC=net
objectSid: S-1-5-21-2082147068-637438979-1492575468

# returned 1 records
# 1 entries
# 0 referrals
root@ucsdc2:~# ls -ld /var/lib/samba*
drwxr-xr-x 11 root root 4096 Jan  3 15:16 /var/lib/samba
drwxr-xr-x  9 root root 4096 Mai  2  2017 /var/lib/samba3
root@ucsdc2:~# ls -l /var/lib/samba*/private/sam.ldb
-rw------- 1 root root 4247552 Mai  2  2017 /var/lib/samba/private/sam.ldb
root@ucsdc2:~# for sam in /var/lib/samba*/private/sam.ldb; do ldbsearch -H "$sam" 1.1 | grep entries; done
# 303 entries
root@ucsdc2:~# exit

Script done on Mi 03 Jan 2018 15:18:54 CET

Looks like the sam.ldb was not touched since May 2017 :confused:


#6

Hi,

the output looks not as I have expected: What is the output of

ucr search --brief connector/s4/ldap/base samba4/ldap/base

#7

It is

root@ucsdc2:~# ucr search --brief connector/s4/ldap/base samba4/ldap/base
connector/s4/ldap/base: 
samba4/ldap/base: DC=LOCAL,DC=K-FAMILY,DC=NET
root@ucsdc2:~#

Should I try and ucr set the connector/s4/ldap/base to the same string as the samba4/ldap/base?


#8

Success! after a brave

root@ucsdc2:~# ucr set connector/s4/ldap/base='DC=LOCAL,DC=K-FAMILY,DC=NET'
Setting connector/s4/ldap/base
root@ucsdc2:~# service univention-s4-connector start

The connector is up and running again!
After returning to my “latest” backup (before all the debugging began), everything is fine!

root@ucsdc2:~# python ./univention-system-check
running [dns]      - OK       - forward_and_reverse_dns_kdc.sh
running [dns]      - OK       - forward_dns_myself.sh
running [listener] - OK       - all_handlers_initialized.sh
running [listener] - OK       - replication.sh
running [basic]    - OK       - check_for_ntpd_process.sh
running [basic]    - OK       - univention_ldapsearch_machine_basic.sh
running [basic]    - OK       - univention_ldapsearch_machine_kerberos.sh
running [basic]    - OK       - check_nagios_status.py
running [basic]    - OK       - secure_apt_is_activated.sh
running [basic]    - OK       - check_for_dockerd_process.sh
running [basic]    - OK       - joinstatus.sh
running [basic]    - OK       - package_status.sh
running [samba]    - OK       - master_is_member_of_enterprise_domain_controllers.sh
running [samba]    - OK       - check_s4_connector_listener_active.sh
running [samba]    - OK       - disabled_drsuapi_adtakeover_incomplete.sh
running [samba]    - OK       - wbinfo_checks.sh
running [samba]    - OK       - krbtgt_has_rid_502.sh
running [samba]    - OK       - check_guid_msdcs_dns_alias.sh
running [samba]    - OK       - cn_system_exists_only_once.sh
running [samba]    - OK       - testjoin.sh
running [samba]    - OK       - hosts_sids_equal_in_ucs_and_samba.sh
running [samba]    - OK       - check_msds_keyversionnumber.sh
running [samba]    - OK       - check_s4_connector_autostart.sh
running [samba]    - OK       - maximum_password_age_smaller_999.sh
running [samba]    - OK       - check_s4_connector_rejects.sh
running [samba]    - OK       - check_samba_drs_replication.sh
running [samba]    - OK       - check_winbind_idmap_range.sh
running [samba]    - OK       - check_samba_processes.sh
running [samba]    - OK       - check_samba_domain_trust.sh
running [samba]    - OK       - check_smbclient_via_krb5_keytab.sh
running [samba]    - OK       - cn_idmap_exists.sh
running [samba]    - OK       - check_ddns_update.sh
running [samba]    - OK       - check_for_temporary_udm_sids.sh
running [samba]    - OK       - no_3000_mapping_in_net_cache.sh
root@ucsdc2:~# 

Thanks a lot!