HTTPS issue after update to UCS 4.4-0

FishboneSA · April 2, 2019, 7:42am

Hi, after update from UCS 4.3 I can no longer access UCS via HTTPS
Nextcloud and OpenProject have the same issue.

Chrome reports ERR_CONNECTION_REFUSED

[root@]# curl -v https://something.mydomain.com
* About to connect() to something.mydomain.com port 443 (#0)
*   Trying 192.168.2.235... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host

Server not listening on port 443

root@:~# netstat -tan | grep LISTEN
tcp        0      0 127.0.0.1:55555         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:389             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:139           0.0.0.0:*               LISTEN
tcp        0      0 192.168.2.235:139       0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:587           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:11212           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:749             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:1743            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:465           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:7636            0.0.0.0:*               LISTEN
tcp        0      0 172.17.42.1:53          0.0.0.0:*               LISTEN
tcp        0      0 192.168.2.235:53        0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:8090          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:636             0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:445           0.0.0.0:*               LISTEN
tcp        0      0 192.168.2.235:445       0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:7389            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:8095            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:32767           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:32768           0.0.0.0:*               LISTEN
tcp        0      0 172.17.42.1:7777        0.0.0.0:*               LISTEN
tcp        0      0 192.168.2.235:7777      0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:2049            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:5666            0.0.0.0:*               LISTEN
tcp6       0      0 :::5443                 :::*                    LISTEN
tcp6       0      0 :::40003                :::*                    LISTEN
tcp6       0      0 :::389                  :::*                    LISTEN
tcp6       0      0 ::1:139                 :::*                    LISTEN
tcp6       0      0 :::6669                 :::*                    LISTEN
tcp6       0      0 :::6670                 :::*                    LISTEN
tcp6       0      0 :::111                  :::*                    LISTEN
tcp6       0      0 :::80                   :::*                    LISTEN
tcp6       0      0 :::7636                 :::*                    LISTEN
tcp6       0      0 :::53                   :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::5432                 :::*                    LISTEN
tcp6       0      0 :::636                  :::*                    LISTEN
tcp6       0      0 :::7389                 :::*                    LISTEN
tcp6       0      0 ::1:445                 :::*                    LISTEN
tcp6       0      0 :::8095                 :::*                    LISTEN
tcp6       0      0 :::32767                :::*                    LISTEN
tcp6       0      0 :::40000                :::*                    LISTEN
tcp6       0      0 :::32768                :::*                    LISTEN
tcp6       0      0 :::40001                :::*                    LISTEN
tcp6       0      0 :::7777                 :::*                    LISTEN
tcp6       0      0 :::2049                 :::*                    LISTEN
tcp6       0      0 :::40002                :::*                    LISTEN
tcp6       0      0 :::5666                 :::*                    LISTEN

Port 443 set to ACCEPT on INPUT and DOCKER chains on firewall

d:/var/log/apache2# systemctl status apache2
â apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-04-02 09:39:59 SAST; 24s ago
  Process: 19584 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 17367 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
  Process: 19590 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 19595 (apache2)
    Tasks: 7 (limit: 4915)
   Memory: 38.9M
      CPU: 164ms
   CGroup: /system.slice/apache2.service
           ââ19595 /usr/sbin/apache2 -k start
           ââ19596 /usr/sbin/apache2 -k start
           ââ19597 /usr/sbin/apache2 -k start
           ââ19598 /usr/sbin/apache2 -k start
           ââ19599 /usr/sbin/apache2 -k start
           ââ19600 /usr/sbin/apache2 -k start
           ââ19650 /usr/sbin/apache2 -k start

Apr 02 09:39:58 bbfnextcloud systemd[1]: Starting The Apache HTTP Server...
Apr 02 09:39:59 bbfnextcloud systemd[1]: Started The Apache HTTP Server.

:/var/log/apache2# cat error.log
[Tue Apr 02 06:25:13.000920 2019] [mpm_prefork:notice] [pid 19490] AH00163: Apache/2.4.25 (Univention) mod_wsgi/4.5.11 Python/2.7 configured -- resuming normal operations
[Tue Apr 02 06:25:13.000987 2019] [core:notice] [pid 19490] AH00094: Command line: '/usr/sbin/apache2'
[Tue Apr 02 09:39:58.472087 2019] [mpm_prefork:notice] [pid 19490] AH00169: caught SIGTERM, shutting down
[Tue Apr 02 09:39:58.551552 2019] [suexec:notice] [pid 19594] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Tue Apr 02 09:40:00.004532 2019] [mpm_prefork:notice] [pid 19595] AH00163: Apache/2.4.25 (Univention) mod_wsgi/4.5.11 Python/2.7 configured -- resuming normal operations
[Tue Apr 02 09:40:00.004603 2019] [core:notice] [pid 19595] AH00094: Command line: '/usr/sbin/apache2'

Could anyone help point me in the right direction?

Moritz_Bunkus · April 2, 2019, 8:43am

Hey,

did you ever modify the templates used for Apache? Check by running univention-check-templates ; no output would indicate that everything’s OK on that front. If something is output, please post it here.

Next verify that the Univention Apache packages are still installed properly by running dpkg -l univention-apache ; post its output here.

PS: when you paste log content or output of programs here, please surround that by two lines containing solely three backticks:

```
content goes here
```

That way the forum will not try to interpret the content as Markdown and keep it as-is. Thanks.

FishboneSA · April 2, 2019, 8:49am

Hi Moritz

Thanks for the reply

univention-check-templates has no output

root@:/var/log/apache2# dpkg -l univention-apache
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                  Version                         Architecture                    Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii  univention-apache                                     11.0.1-1A~4.4.0.201812211048    all                             UCS - Apache2 configuration

Thanks for the help regarding the backticks.

Moritz_Bunkus · April 2, 2019, 10:47am

So far, so good.

Now please restart Apache and post the output of the latter commands:

systemctl restart apache2.service
lsof -PniTCP:443
lsof -aPni -c apache2
grep -Er '^[^#]*Listen' /etc/apache2/
ls -l /etc/apache2/mods-enabled/

FishboneSA · April 2, 2019, 10:53am

apache2 service restarted

lsof -PniTCP:443 generates no output.

root@:/var/log/apache2# lsof -aPni -c apache2
COMMAND  PID     USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
apache2 9261     root    4u  IPv6 1275910      0t0  TCP *:80 (LISTEN)
apache2 9262 www-data    4u  IPv6 1275910      0t0  TCP *:80 (LISTEN)
apache2 9262 www-data   11u  IPv6 1275687      0t0  TCP 192.168.2.235:80->192.168.2.16:60301 (ESTABLISHED)
apache2 9263 www-data    4u  IPv6 1275910      0t0  TCP *:80 (LISTEN)
apache2 9264 www-data    4u  IPv6 1275910      0t0  TCP *:80 (LISTEN)
apache2 9265 www-data    4u  IPv6 1275910      0t0  TCP *:80 (LISTEN)
apache2 9266 www-data    4u  IPv6 1275910      0t0  TCP *:80 (LISTEN)
apache2 9272 www-data    4u  IPv6 1275910      0t0  TCP *:80 (LISTEN)

root@:/var/log/apache2# grep -Er '^[^#]*Listen' /etc/apache2/
/etc/apache2/ports.conf.debian:Listen 80
/etc/apache2/ports.conf.debian: Listen 443
/etc/apache2/ports.conf.debian: Listen 443
/etc/apache2/ports.conf:Listen 80
/etc/apache2/mods-available/ssl.conf:   Listen 443

root@:/var/log/apache2# ls -l /etc/apache2/mods-enabled/
total 0
lrwxrwxrwx 1 root root 36 Aug  2  2018 access_compat.load -> ../mods-available/access_compat.load
lrwxrwxrwx 1 root root 30 Nov 23 09:20 actions.conf -> ../mods-available/actions.conf
lrwxrwxrwx 1 root root 30 Nov 23 09:20 actions.load -> ../mods-available/actions.load
lrwxrwxrwx 1 root root 28 Aug  2  2018 alias.conf -> ../mods-available/alias.conf
lrwxrwxrwx 1 root root 28 Aug  2  2018 alias.load -> ../mods-available/alias.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 auth_basic.load -> ../mods-available/auth_basic.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 authn_core.load -> ../mods-available/authn_core.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 authn_file.load -> ../mods-available/authn_file.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 authnz_pam.conf -> ../mods-available/authnz_pam.conf
lrwxrwxrwx 1 root root 33 Aug  2  2018 authnz_pam.load -> ../mods-available/authnz_pam.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 authz_core.load -> ../mods-available/authz_core.load
lrwxrwxrwx 1 root root 38 Aug  2  2018 authz_groupfile.load -> ../mods-available/authz_groupfile.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 authz_host.load -> ../mods-available/authz_host.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 authz_user.load -> ../mods-available/authz_user.load
lrwxrwxrwx 1 root root 32 Aug  2  2018 autoindex.conf -> ../mods-available/autoindex.conf
lrwxrwxrwx 1 root root 32 Aug  2  2018 autoindex.load -> ../mods-available/autoindex.load
lrwxrwxrwx 1 root root 26 Nov 23 09:20 cgi.load -> ../mods-available/cgi.load
lrwxrwxrwx 1 root root 30 Aug  2  2018 deflate.conf -> ../mods-available/deflate.conf
lrwxrwxrwx 1 root root 30 Aug  2  2018 deflate.load -> ../mods-available/deflate.load
lrwxrwxrwx 1 root root 26 Aug  2  2018 dir.conf -> ../mods-available/dir.conf
lrwxrwxrwx 1 root root 26 Aug  2  2018 dir.load -> ../mods-available/dir.load
lrwxrwxrwx 1 root root 26 Aug  2  2018 env.load -> ../mods-available/env.load
lrwxrwxrwx 1 root root 30 Aug  2  2018 expires.load -> ../mods-available/expires.load
lrwxrwxrwx 1 root root 29 Aug  2  2018 filter.load -> ../mods-available/filter.load
lrwxrwxrwx 1 root root 30 Aug  2  2018 headers.load -> ../mods-available/headers.load
lrwxrwxrwx 1 root root 27 Aug  2  2018 mime.conf -> ../mods-available/mime.conf
lrwxrwxrwx 1 root root 27 Aug  2  2018 mime.load -> ../mods-available/mime.load
lrwxrwxrwx 1 root root 34 Aug  2  2018 mpm_prefork.conf -> ../mods-available/mpm_prefork.conf
lrwxrwxrwx 1 root root 34 Aug  2  2018 mpm_prefork.load -> ../mods-available/mpm_prefork.load
lrwxrwxrwx 1 root root 34 Aug  2  2018 negotiation.conf -> ../mods-available/negotiation.conf
lrwxrwxrwx 1 root root 34 Aug  2  2018 negotiation.load -> ../mods-available/negotiation.load
lrwxrwxrwx 1 root root 29 Nov 23 09:20 php7.0.conf -> ../mods-available/php7.0.conf
lrwxrwxrwx 1 root root 29 Nov 23 09:20 php7.0.load -> ../mods-available/php7.0.load
lrwxrwxrwx 1 root root 28 Aug  2  2018 proxy.conf -> ../mods-available/proxy.conf
lrwxrwxrwx 1 root root 36 Aug  2  2018 proxy_connect.load -> ../mods-available/proxy_connect.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 proxy_http.load -> ../mods-available/proxy_http.load
lrwxrwxrwx 1 root root 28 Aug  2  2018 proxy.load -> ../mods-available/proxy.load
lrwxrwxrwx 1 root root 37 Nov 23 11:06 proxy_wstunnel.load -> ../mods-available/proxy_wstunnel.load
lrwxrwxrwx 1 root root 33 Aug  2  2018 reqtimeout.conf -> ../mods-available/reqtimeout.conf
lrwxrwxrwx 1 root root 33 Aug  2  2018 reqtimeout.load -> ../mods-available/reqtimeout.load
lrwxrwxrwx 1 root root 30 Aug  2  2018 rewrite.load -> ../mods-available/rewrite.load
lrwxrwxrwx 1 root root 31 Aug  2  2018 setenvif.conf -> ../mods-available/setenvif.conf
lrwxrwxrwx 1 root root 31 Aug  2  2018 setenvif.load -> ../mods-available/setenvif.load
lrwxrwxrwx 1 root root 36 Aug  2  2018 socache_shmcb.load -> ../mods-available/socache_shmcb.load
lrwxrwxrwx 1 root root 29 Aug  2  2018 status.conf -> ../mods-available/status.conf
lrwxrwxrwx 1 root root 29 Aug  2  2018 status.load -> ../mods-available/status.load
lrwxrwxrwx 1 root root 29 Nov 23 09:20 suexec.load -> ../mods-available/suexec.load
lrwxrwxrwx 1 root root 32 Aug  2  2018 unique_id.load -> ../mods-available/unique_id.load
lrwxrwxrwx 1 root root 27 Aug  2  2018 wsgi.conf -> ../mods-available/wsgi.conf
lrwxrwxrwx 1 root root 27 Aug  2  2018 wsgi.load -> ../mods-available/wsgi.load

Moritz_Bunkus · April 2, 2019, 1:14pm

The SSL module isn’t enabled. Try:

a2enmod ssl
a2ensite default-ssl
systemctl restart apache2.service

Additionally please post the output of ls /etc/apache2/sites-enabled/ ; maybe there are other sites that should be enabled but aren’t for you.

FishboneSA · April 2, 2019, 2:35pm

You, Sir, are a hero

That did the trick.
I noticed from the previous output that the SSL module wasn’t enabled and ran a2enmod ssl but forgot about the second command.

Output requested…

root@:~# ls -l /etc/apache2/sites-enabled/
total 0
lrwxrwxrwx 1 root root 35 Aug  2  2018 000-default.conf -> ../sites-available/000-default.conf
lrwxrwxrwx 1 root root 35 Apr  2 16:01 default-ssl.conf -> ../sites-available/default-ssl.conf
lrwxrwxrwx 1 root root 34 Aug  2  2018 univention.conf -> ../sites-available/univention.conf
lrwxrwxrwx 1 root root 41 Apr  1 07:54 univention-portal.conf -> ../sites-available/univention-portal.conf
lrwxrwxrwx 1 root root 39 Nov 23 09:20 univention-saml.conf -> ../sites-available/univention-saml.conf
lrwxrwxrwx 1 root root 50 Mar 30 18:59 univention-server-overview.conf -> ../sites-available/univention-server-overview.conf

Any reason you can think of why this would have happened?

Moritz_Bunkus · April 2, 2019, 2:53pm

Those other links look fine, none seems to be missing (check with the contents of /etc/apache2/sites-available if unsure).

From the top of my head I don’t have an idea what might have caused the symlinks to disappear.

FishboneSA · April 2, 2019, 3:05pm

Thanks again.
I really appreciate the time and effort spent on helping me.

All the docker apps are running fine now with SLL along with the UCS management portal.

Moritz_Bunkus · April 2, 2019, 3:06pm

You’re quite welcome.