Removing Active Directory App Broke Everything

Apps & App Center
, , ,
#1

Months ago I set up a Univention Nextcloud applicance, and it’s worked great through a couple Nextcloud major revision upgrades. I’ve got around 3TB of content on it now. I wanted (still want) to be able to access those files not only via the Nextcloud sync tool, but directly from my Windows PCs via CIFS. To try to accomplish this, I installed Samba 4 via the Active Directory Domain Controller app. Because I was not trying to use my Nextcloud appliance as a domain controller, it never got configured correctly, and I got busy with other things and I just left it, AD non-working but Nextcloud working like a champ.

Last weekend I decided that since I now have an AD Docker container appliance elsewhere on my network, I could just remove the AD from my Univention Nextcloud VM and join it to my new test domain. Unfortunately, removing the app appears to have resulted in my VM being unable to do all the Nextcloud things: Nextcloud clients can’t connect to the server, and I can attempt to login to Nextcloud but I get an error. I am able to log into the Univention interface, and I can go to App Center, but then when I try browsing apps (to reinstall the one I removed) I get “An unknown error with status code 502 occurred while connecting to the server, please try again later.”. The software update module tells me there are updates, but Package Updates loads indefinitely. Logging into Nextcloud returns the following:

The server was unable to complete your request.

If this happens again, please send the technical details below to the server administrator.

More details can be found in the server log.

### Technical details

* Remote Address: 10.1.10.111
* Request ID: hy3V0gptf37rnmgAugYb

The currently installed release version is 4.4-2 errata294. Nextcloud is 16.0.4-0. I still have SSH access to the VM, and the etc/hosts file looks like I would expect it to. I did not have the foresight to take a snapshot prior to removing the AD app. So from here, what are my options for troubleshooting and restoring whatever I’ve broken by removing the app? My goal is to have Nextcloud functional again, and it would be nice if I didn’t have to recreate the VM and move 3TB over, but I guess that’s always a final option…

#2

Hello @CinciTech,

well … that’s not what is supposed to happen when you uninstall an App :confused:

I can’t point my finger at the cause right now, so let’s gather some more insight in which condition your UCS Nextcloud Appliance actually is. Can you please run the following commands via SSH on the commandline and paste the output here?

univention-app info

univention-check-join-status

dpkg --audit

dig AXFR $(hostname -d)

ucr get dns/backend

univention-run-diagnostic-checks -t ALL

Best regards,
Michael Grandjean

#3

I agree, that’s not how it’s supposed to work. I’m just hopeful that either I’m helping reveal some really obscure bug, or if it’s something I screwed up that it’s not a lot of work for you to tell me where to unscrew it up. That said, I’m good for copypasting commands at least! :smiley:

Last login: Sat Nov  2 08:21:34 2019 from 10.1.10.111
root@nextcloud:~# univention-app info
UCS: 4.4-2 errata294
Installed: adconnector=12.0 letsencrypt=1.2.2-8 4.1/nextcloud=16.0.4-0
Upgradable: nextcloud
root@nextcloud:~# univention-check-join-status
Joined successfully
root@nextcloud:~# dpkg --audit
root@nextcloud:~# dig AXFR $(hostname -d)

; <<>> DiG 9.10.3-P4-Univention <<>> AXFR 337.ninja
;; global options: +cmd
; Transfer failed.
root@nextcloud:~# ucr get dns/backend
ldap
root@nextcloud:~#
#4

Thanks!

  1. So, you don’t have any broken packages and all join scripts are executed successfully. That’s good :slight_smile:

  2. adconnector is installed. This is a Connector App that can either read, write or sync users, groups etc. from an existing Active Directory (usually Microsoft AD) from/to UCS. Was this installed on purpose? Was the UCS appliance connected to an existing Active Directory before you installed the Samba Active Directory App? I guess not, but just to make sure.

  3. Your DNS resolution does not seem to be working. The dig command should have returned some DNS records. What does “systemctl status bind9” say? Also check “ps auxf | grep [n]amed”

  4. Also: Can you please also execute the last command I suggested above?
    -> univention-run-diagnostic-checks -t ALL

#5
  1. Yay!
  2. Is it possible that this got missed when I told my VM to remove the AD server app? I’m sure it was installed “on purpose”, although what I really meant to do was install Samba for CIFS, but all I found was ADDC available as an app, so it’s very possible that I just did this by mistake. I did not have an AD domain on my home network when I set this up, which is part of the reason it never got set up correctly.
  3. That’s consistent with what I suspected but couldn’t quantify. Per DHCP on my router (a Mikrotik smart switch running RouterOS) my DNS goes through a PiHole VM, (opensource virtual appliance that filters traffic going to blacklisted URLs). The IP of PiHole happens to be 10.1.10.3, and this is handed out to my network via DHCP. Anything PiHole doesn’t have a DNS entry for, it forwards on to Google’s public DNS (8.8.8.8). So what I could maybe use help on is knowing whether this means my Univention VM is not finding 10.1.10.3, or perhaps I need to add some hard-coded entries to the DNS config in 10.1.10.3 to make up for something that got removed, or maybe I need to add something to the HOSTS file or manually alter DNS on the univention VM?
  4. So apparently I’m not that good at copypasting commands. :confused: Here y’go:
root@nextcloud:~# univention-run-diagnostic-checks -t ALL
Domain Admin Login: Administrator
Password:
usage: univention-run-diagnostic-checks [-h] [--bindpwdfile BINDPWDFILE]
                                        [--username USERNAME] -t
                                        {all,00_check_server_password,01_ssh_con                       nection,02_certificate_check,03_check_notifier_replication,04_saml_certificate_c                       heck,10_gateway,11_nameserver,12_proxy,20_check_nameservers,21_check_join_status                       ,22_kdc_service,23_check_update_sites,30_disk_usage,31_file_permissions,32_secur                       ity_limits,40_samba_tool_dbcheck,41_samba_tool_showrepl,43_connectors4_rejects,4                       4_well_known_sid_check,45_heimdal_on_samba4_dc,46_kerberos_ddns_update,50_check_                       ucr_templates,51_hostname_check,52_mail_acl_sync,53_package_status,54_sources_li                       st_check,55_user_migration,56_univention_types,57_univention_server_role_windows                       }
                                        [{all,00_check_server_password,01_ssh_co                       nnection,02_certificate_check,03_check_notifier_replication,04_saml_certificate_                       check,10_gateway,11_nameserver,12_proxy,20_check_nameservers,21_check_join_statu                       s,22_kdc_service,23_check_update_sites,30_disk_usage,31_file_permissions,32_secu                       rity_limits,40_samba_tool_dbcheck,41_samba_tool_showrepl,43_connectors4_rejects,                       44_well_known_sid_check,45_heimdal_on_samba4_dc,46_kerberos_ddns_update,50_check                       _ucr_templates,51_hostname_check,52_mail_acl_sync,53_package_status,54_sources_l                       ist_check,55_user_migration,56_univention_types,57_univention_server_role_window                       s} ...]
univention-run-diagnostic-checks: error: argument -t: invalid choice: 'ALL' (cho                       ose from 'all', u'00_check_server_password', u'01_ssh_connection', u'02_certific                       ate_check', u'03_check_notifier_replication', u'04_saml_certificate_check', u'10                       _gateway', u'11_nameserver', u'12_proxy', u'20_check_nameservers', u'21_check_jo                       in_status', u'22_kdc_service', u'23_check_update_sites', u'30_disk_usage', u'31_                       file_permissions', u'32_security_limits', u'40_samba_tool_dbcheck', u'41_samba_t                       ool_showrepl', u'43_connectors4_rejects', u'44_well_known_sid_check', u'45_heimd                       al_on_samba4_dc', u'46_kerberos_ddns_update', u'50_check_ucr_templates', u'51_ho                       stname_check', u'52_mail_acl_sync', u'53_package_status', u'54_sources_list_chec                       k', u'55_user_migration', u'56_univention_types', u'57_univention_server_role_wi                       ndows')
root@nextcloud:~#
#6

The right command is

univention-run-diagnostic-checks -t all
#7

Doh, the result was long and I was running off to work, so I didn’t get a chance to look thru it to realize it was a mistype. Here’s the correct execution:

root@nextcloud:~# univention-run-diagnostic-checks -t all
Domain Admin Login: Administrator
Password:

You can find the logging messages of the diagnostic modules at /var/log/univention/management-console-module-diagnostic.log

ran 00_check_server_password successfully.

ran 01_ssh_connection successfully.

ran 02_certificate_check successfully.

ran 03_check_notifier_replication successfully.

########################## Start 04_saml_certificate_check ##########################
## Check failed: 04_saml_certificate_check - SAML certificate verification failed! ##
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/__init__.py", line 280, in execute
    result = execute(umc_module, **kwargs)
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/04_saml_certificate_check.py", line 76, in run
    test_identity_provider_certificate()
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/04_saml_certificate_check.py", line 89, in test_identity_provider_certificate
    for host in socket.gethostbyname_ex(sso_fqdn)[2]:
gaierror: [Errno -2] Name or service not known
########################### End 04_saml_certificate_check ###########################

ran 10_gateway successfully.

####################### Start 11_nameserver ########################
## Check failed: 11_nameserver - Nameserver(s) are not responsive ##
2 of the configured nameservers are not responding to DNS queries.
Please make sure the DNS settings in the {setup:network} are correctly set up.
If the problem persists make sure the nameserver is connected to the network and the forwarders are able to reach the internet (www.univention.de).

The nameserver 10.1.10.9 (UCR variable 'nameserver1') is not responsive:
All nameservers failed to answer the query nextcloud.crawford.local. IN A: Server 10.1.10.9 UDP port 53 answered SERVFAIL


The nameserver 10.1.10.253 (UCR variable 'dns/forwarder1') is not responsive:
A timeout occurred while reaching the nameserver (is it online?).
######################## End 11_nameserver #########################

ran 12_proxy successfully.

ran 20_check_nameservers successfully.

ran 21_check_join_status successfully.

################ Start 22_kdc_service ################
## Check failed: 22_kdc_service - KDC service check ##
No reachable KDCs were found.
################# End 22_kdc_service #################

######################## Start 23_check_update_sites #########################
## Check failed: 23_check_update_sites - Check resolving repository servers ##
The following FQDNs were not resolvable:
https://updates.software-univention.de
appcenter.software-univention.de
Please see {sdb} for troubleshooting DNS problems.
######################### End 23_check_update_sites ##########################

ran 30_disk_usage successfully.

################## Start 31_file_permissions ###################
## Check failed: 31_file_permissions - Check file permissions ##
File '/var/cache/univention-system-activation' has mode 777, 700 was expected.
################### End 31_file_permissions ####################

ran 32_security_limits successfully.

ran 40_samba_tool_dbcheck successfully.

ran 41_samba_tool_showrepl successfully.

ran 43_connectors4_rejects successfully.

ran 44_well_known_sid_check successfully.

ran 45_heimdal_on_samba4_dc successfully.

########################### Start 46_kerberos_ddns_update ############################
## Check failed: 46_kerberos_ddns_update - Check kerberos authenticated DNS updates ##
Errors occurred while running `kinit` or `nsupdate`.
`kinit` for principal nextcloud$ with password file /etc/machine.secret failed.
############################ End 46_kerberos_ddns_update #############################

ran 50_check_ucr_templates successfully.

ran 51_hostname_check successfully.

ran 52_mail_acl_sync successfully.

ran 53_package_status successfully.

######################## Start 54_sources_list_check #########################
## Check failed: 54_sources_list_check - Check errors in sources.list files ##
Found exception in '/etc/apt/sources.list.d/20_ucs-online-component.list': ConfigurationError: Configuration error: host is unresolvable
Please check the files for more details.
The error might be fixable by regenerating the sources.list.
######################### End 54_sources_list_check ##########################

ran 55_user_migration successfully.

ran 56_univention_types successfully.

ran 57_univention_server_role_windows successfully.

root@nextcloud:~#
#8

So, armed with the previous info (and a good night’s sleep), I have identified that the DNS settings are/were definitely pointing to an external nameserver which never (as far as I can remember) existed (10.1.10.253), as well as the local DC (10.1.10.9), which I haven’t fully got working for local name resolution. I went to network settings and removed the external nameserver, replaced it with my PiHole appliance, and I appear to at least be able to update the packages and I’ve upgraded Nextcloud to 16.0.5-0. I am still unable to log into Nextcloud. 11_nameserver is much happier now:

root@nextcloud:~# univention-run-diagnostic-checks --username Administrator -t all
Password:

You can find the logging messages of the diagnostic modules at /var/log/univention/management-console-module-diagnostic.log

ran 00_check_server_password successfully.

ran 01_ssh_connection successfully.

ran 02_certificate_check successfully.

ran 03_check_notifier_replication successfully.

########################## Start 04_saml_certificate_check ##########################
## Check failed: 04_saml_certificate_check - SAML certificate verification failed! ##
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/__init__.py", line 280, in execute
    result = execute(umc_module, **kwargs)
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/04_saml_certificate_check.py", line 76, in run
    test_identity_provider_certificate()
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/04_saml_certificate_check.py", line 89, in test_identity_provider_certificate
    for host in socket.gethostbyname_ex(sso_fqdn)[2]:
gaierror: [Errno -2] Name or service not known
########################### End 04_saml_certificate_check ###########################

ran 10_gateway successfully.

ran 11_nameserver successfully.

ran 12_proxy successfully.

ran 20_check_nameservers successfully.

ran 21_check_join_status successfully.

################ Start 22_kdc_service ################
## Check failed: 22_kdc_service - KDC service check ##
No reachable KDCs were found.
################# End 22_kdc_service #################

ran 23_check_update_sites successfully.

ran 30_disk_usage successfully.

################## Start 31_file_permissions ###################
## Check failed: 31_file_permissions - Check file permissions ##
File '/var/cache/univention-system-activation' has mode 777, 700 was expected.
################### End 31_file_permissions ####################

ran 32_security_limits successfully.

ran 40_samba_tool_dbcheck successfully.

ran 41_samba_tool_showrepl successfully.

ran 43_connectors4_rejects successfully.

ran 44_well_known_sid_check successfully.

ran 45_heimdal_on_samba4_dc successfully.

########################### Start 46_kerberos_ddns_update ############################
## Check failed: 46_kerberos_ddns_update - Check kerberos authenticated DNS updates ##
Errors occurred while running `kinit` or `nsupdate`.
`kinit` for principal nextcloud$ with password file /etc/machine.secret failed.
############################ End 46_kerberos_ddns_update #############################

ran 50_check_ucr_templates successfully.

ran 51_hostname_check successfully.

ran 52_mail_acl_sync successfully.

ran 53_package_status successfully.

ran 54_sources_list_check successfully.

ran 55_user_migration successfully.

ran 56_univention_types successfully.

ran 57_univention_server_role_windows successfully.

root@nextcloud:~#
#9

Worked with it a little bit more last night, and it seems my certificates are wrong. They were right when I thought I was going to use mylastname.local as my domain (the Samba AD app made me choose something), but later I’ve opted to use 337.ninja as my internal domain (on the external Docker Container Samba DC). I’ve made it as far as trying to join the new domain, but it tells me “The domain name of the AD server does not match the local UCS domain name. For the AD member mode, it is necessary to setup a UCS system with the same domain name as the AD server.”

So, any direction on what I need to do in order to update my UCS VM with the new domain name? Do I need to regenerate certificates as well?

#10

Googling for awhile turns up many requests for being able to change the hostname and/or domain name. I’ve seen a script that was written for UCS3/4 but doesn’t appear to work for me. Otherwise it seems that everyone defaults to this being a humongous change for all apps and thus a bad idea. Which is truly a shame, since it’s pretty easy to rename a Windows server.

So, I have a semi-working USC VM with happy DNS but I’m not getting any love from Nextcloud anymore. And short of further direction I am going to assume the only option I have left is to set up another Nextcloud VM and move the files.

Mastodon