Problem: Überprüfung der SSL Zertifikate

UCS - Univention Corporate Server
,
#1

I see a problem in the system-checks on a UCS memberserver but I don’t know what it is. The link on the web-ui is pointing me to the renewal thread, but the certificate is still valid for some years.
On the console I get this error traceback:

root@member:/# univention-run-diagnostic-checks -t 02_certificate_check
Domain Admin Login: Administrator
Password:

You can find the logging messages of the diagnostic modules at /var/log/univention/management-console-module-diagnostic.log

############################ Start 02_certificate_check ###########################
## Check failed: 02_certificate_check - Überprüfe Gültigkeit der SSL Zertifikate ##
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/__init__.py", line 280, in execute
    result = execute(umc_module, **kwargs)
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/02_certificate_check.py", line 289, in run
    all_certificates))
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/02_certificate_check.py", line 267, in verify_from_master
    with download_tempfile(root_ca_uri) as root_ca, download_tempfile(crl_uri) as crl:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/pymodules/python2.7/univention/management/console/modules/diagnostic/plugins/02_certificate_check.py", line 239, in download_tempfile
    response = requests.get(url, stream=True)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 247, in resolve_redirects
    **adapter_kwargs
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
SSLError: HTTPSConnectionPool(host='member.domain.tdl', port=443): Max retries exceeded with url: /ucs-root-ca.crt (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
############################# End 02_certificate_check ############################

Best,
Bernd

Check the validity of the SSL certificates failure
#2

Looks like the system diagnostics tries to retrieve the CA certificate stored on that server via HTTPS which fails. The most likely explanation is that the SSL certificate present to the client doesn’t match the expected server name.

Did you modify your Apache configuration somehow? Are you running Let’s Encrypt on that server? What’s the output of univention-check-templates?

#3

Oh ya… I tried to get rid of the nextcloud warning about missing .well-known entries. So I added to default-ssl something to the templates:

### https://docs.nextcloud.com/server/15/admin_manual/issues/general_troubleshooting.html#service-discovery
<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule ^/\.well-known/host-meta /nextcloud/public.php?service=host-meta [QSA,L]
  RewriteRule ^/\.well-known/host-meta\.json /nextcloud/public.php?service=host-meta-json [QSA,L]
  RewriteRule ^/\.well-known/webfinger /nextcloud/public.php?service=webfinger [QSA,L]
  RewriteRule ^/\.well-known/carddav /nextcloud/remote.php/dav/ [R=301,L]
  RewriteRule ^/\.well-known/caldav /nextcloud/remote.php/dav/ [R=301,L]
</IfModule>

The univention-check-templates only shows me some email and SNMP related entries.

I will check on the nextcloud-entry later.

Thanks for your help,
Bernd

#4

You modified default-ssl.conf? The one in /etc/apache2/sites-available/default-ssl.conf or any of the templates in /etc/univention/templates/files/etc/apache2/sites-available/ssl.d/?

What’s the output of curl -v https://$(hostname -f)/ucs-root-ca.crt > /dev/null ?

#5

I have added a conf-file to /etc/univention/templates/files/etc/apache2/sites-available/ssl.d/ and an info-file too.

The output of the curl command is:

*   Trying IP-ADDRESS ...
* TCP_NODELAY set
* Connected to ucs-member.domain.intern (IP-ADDRESS) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=DE; ST=DE; L=DE; O=domain; OU=Univention Corporate Server; CN=ucs-member.domain.intern; emailAddress=ssl@domain.intern
*  start date: Mar 23 20:39:22 2017 GMT
*  expire date: Mar 22 20:39:22 2022 GMT
*  subjectAltName: host "ucs-member.domain.intern" matched cert's "ucs-member.domain.intern"
*  issuer: C=DE; ST=DE; L=DE; O=domain; OU=Univention Corporate Server; CN=Univention Corporate Server Root CA (ID=fsDtIgFY); emailAddress=ssl@domain.intern
*  SSL certificate verify ok.
> GET /ucs-root-ca.crt HTTP/1.1
> Host: ucs-member.domain.intern
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 22 Oct 2019 16:01:13 GMT
< Server: Apache/2.4.25 (Univention)
< Strict-Transport-Security: max-age=15552000
< Last-Modified: Thu, 23 Mar 2017 20:36:22 GMT
< ETag: "79c-54b6bd47f71a2"
< Accept-Ranges: bytes
< Content-Length: 1948
< Content-Type: application/x-x509-ca-cert
<
* Curl_http_done: called premature == 0
* Connection #0 to host ucs-member.domain.intern left intact

Which seams fine to me.

Anyway I will try that text without the nextcloud part.

#6

Well that’s unfortunate - the same without my nextcloud-.well-known code.

Another idea?

#7

Hey,

you’re actually using two different host names. In your original post it’s member.domain.tdl:

and in your last one it’s ucs-member.domain.intern:

I’m guessing you’ve replaced the actual host & domain names in both posts, but please make absolutely sure they’re actually the same in both situations.

#8

Yes, they are - my mistake replacing them differently.

#9

Great. Some more commands to test. Please post their output.

python2 -c "import requests; import pprint; pprint.pprint(requests.get('https://$(ucr get hostname).$(ucr get domainname)/ucs-root-ca.crt'))"
test $(hostname -f) = $(ucr get hostname).$(ucr get domainname) && echo OK
#10

For the commands:

root@ucs-member:~/debug# python2 -c "import requests; import pprint; pprint.pprint(requests.get('https://$(ucr get hostname).$(ucr get domainname)/ucs-root-ca.crt'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='ucs-member.domain.intern', port=443): Max retries exceeded with url: /ucs-root-ca.crt (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
1 root@ucs-member:~/debug# test $(hostname -f) = $(ucr get hostname).$(ucr get domainname) && echo OK
OK

So - same error and OK.
If I look at line 514 of /usr/local/lib/python2.7/dist-packages/requests/adapters.py I see:

512            if isinstance(e.reason, _SSLError):
513                # This branch is for urllib3 v1.22 and later.
514                raise SSLError(e, request=request)

and then:

root@ucs-member:~/debug# dpkg -l | grep urllib3
ii  python-urllib3                                      1.19.1-1                                                            all          HTTP library with thread-safe connection pooling for Python
ii  python3-urllib3                                     1.19.1-1                                                            all          HTTP library with thread-safe connection pooling for Python3

Can this be somehow related? Anyway, this is also the case on systems where the check is fine.

#11

Well… Maybe try updating the CA certificate store:

update-ca-certificates

Also try accessing any other web server via SSL in Python, e.g. Google:

python2 -c "import requests; import pprint; pprint.pprint(requests.get('https://www.google.com/'))"

If the former doesn’t help & the latter works, let’s look at which Python modules are actually loaded:

strace -o open.txt -e open python2 -c "import requests; import pprint; pprint.pprint(requests.get('https://$(ucr get hostname).$(ucr get domainname)/ucs-root-ca.crt'))"

Then post open.txt here (it’ll be rather big).

You may have to install the strace package if the command isn’t found.

#12 
root@ucs-member:~/debug# update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

updates of cacerts keystore disabled.
done.

root@ucs-member:~/debug# python2 -c "import requests; import pprint; pprint.pprint(requests.get('https://www.google.com/'))"
<Response [200]>

and the indeed long open.txt was too long for the forum software to paste. So: open.txt (384,0 KB)

a lot of No such...

#13

Thanks.

All of those No such file or directory errors are absolutely OK.

What’s not fine, though, is that you seem to have a copy of urllib3 and requests lying around in /usr/local/…:

open("/usr/local/lib/python2.7/dist-packages", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
open("/usr/local/lib/python2.7/dist-packages/requests/__init__.py", O_RDONLY) = 3
open("/usr/local/lib/python2.7/dist-packages/requests/__init__.pyc", O_RDONLY) = 4
open("/usr/local/lib/python2.7/dist-packages/urllib3/__init__.py", O_RDONLY) = 4
open("/usr/local/lib/python2.7/dist-packages/urllib3/__init__.pyc", O_RDONLY) = 5
open("/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", O_RDONLY) = 5
open("/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.pyc", O_RDONLY) = 6
…
open("/usr/local/lib/python2.7/dist-packages/certifi/cacert.pem", O_RDONLY) = 4
open("/usr/local/lib/python2.7/dist-packages/requests/api.py", O_RDONLY) = 3
open("/usr/local/lib/python2.7/dist-packages/requests/api.py", O_RDONLY) = 3
open("/usr/local/lib/python2.7/dist-packages/requests/sessions.py", O_RDONLY) = 3
open("/usr/local/lib/python2.7/dist-packages/requests/sessions.py", O_RDONLY) = 3
open("/usr/local/lib/python2.7/dist-packages/requests/adapters.py", O_RDONLY) = 3

I’m pretty sure that you’ve installed those for some reason at one time or another. Please purge any custom Python code from /usr/local and try those manual requests again (meaning running python2 -c … from the command line). If that works, restart the server (too many Python services running that would have to be listed manually) and see if the check works now.

#14

Ok.

Just removing them rm ... ? Or do I check for installed packages?

#15

No regular Debian package will ever install stuff into /usr/local. Simply delete the contents of /usr/local/lib/python2.7

1 Like
#16

Hm, well I think I know now… I’m using checkmk to monitor the system and the new docker monitoring agent needs some pip install docker to work.

#17

So I have the choice between running the monitoring software and getting errors in the univention-check?
Ok - I’ve tested this on another server where I’ve installed the agent and I get the same mistake.

Is it worth trying to remove the files from /usr/local and installing python-docker instead?

Edit: To answer this incase someone will read until here.
Removing /usr/local/lib/python2.7/dist-packages solves the issue. Thank you very much @Moritz_Bunkus

And will make the checkmk agent crit telling it needs pip install docker
What also works for me is: univention-install python-docker python-certifi python-backports.ssl-match-hostname
The other needed packages are already installed (on my system - you can check what packages got installed into dist-packages).

Edit2: Or you do it in some python-way. univention-install python-virtualenv

# cd /usr/lib/check_mk_agent
# virtualenv env
# source /usr/lib/check_mk_agent/env/bin/activate
(env)# pip install docker
(env)# deactivate
# nano path/to/mk_docker.py

Change the shebang to

#!/usr/lib/check_mk_agent/env/bin/python

Save. Works too.

Mastodon