Letsencrypt failed after 4.4-9 to 5.02 UCS upgrade

UCS - Univention Corporate Server
#1

I upgraded my fully updated 4.4-9 server to 5.02 using the UCS web interface. Most things went pretty well. I needed to edit a couple of virtual hosts to get apache2 to load and I needed to add a UCR variable for postfix timeout. That said, I cannot get letsencrypt to work. I can’t even get it to uninstall and start over.

Unfortunately, this conversion happened over the day that the existing letsencrypt cert expired. So, nothing is easy…

Everything appears to work from the UCS web interface up to the point where the screen sales “Registering account…” At that point the system seems ‘stuck’. No further activity and it doesn’t complete the update.

Checking the appcenter.log:
10584 actions.configure 22-08-30 15:17:32 [ DEBUG]: Calling configure
10584 actions.configure.progress 22-08-30 15:17:32 [ DEBUG]: 0
10584 actions.configure 22-08-30 15:17:32 [ INFO]: Configuring letsencrypt=2.0.0-2
10584 settings 22-08-30 15:17:32 [ INFO]: Setting letsencrypt/domains to ‘sp-svr01.infolocity.net www.ivyinfosys.com www.omegafsi.com www.pennreserve.com ucs-sso.infolocity.net www.byteflight.net’
10584 settings 22-08-30 15:17:32 [ INFO]: Setting letsencrypt/services/apache2 to ‘true’
10584 settings 22-08-30 15:17:32 [ INFO]: Setting letsencrypt/services/dovecot to ‘false’
10584 settings 22-08-30 15:17:32 [ INFO]: Setting letsencrypt/services/postfix to ‘true’
10584 settings 22-08-30 15:17:32 [ INFO]: Setting letsencrypt/status to ‘“detail”: “JWS has an invalid anti-replay nonce: \“0001bCbS7WDrN4h0EXKHNVKB8MMVgyqIs3w1qETbduxkIOw\””,’
10584 settings 22-08-30 15:17:32 [ INFO]: Setting letsencrypt/staging to ‘false’
10584 actions.configure 22-08-30 15:17:35 [ DEBUG]: Calling /var/cache/univention-appcenter/appcenter.software-univention.de/5.0/letsencrypt_20211006103329.configure_host settings --version 2.0.0-2 --error-file /tmp/tmpf3eu98dq --locale en
10584 actions.configure 22-08-30 15:17:36 [ INFO]: WARNING: UCR variable letsencrypt/domains does not match domains in CSR.
10584 actions.configure 22-08-30 15:17:36 [ INFO]: Removing domain.csr…
10584 actions.configure 22-08-30 15:17:36 [ INFO]: Creating domain.csr…
10584 actions.configure 22-08-30 15:17:36 [ INFO]: Multi domain mode
10584 actions.configure 22-08-30 15:17:36 [ WARNING]: run-parts: executing /etc/univention/letsencrypt/setup.d//apache2
10584 actions.configure 22-08-30 15:17:37 [ INFO]: Setting apache2/ssl/certificate
10584 actions.configure 22-08-30 15:17:37 [ INFO]: Setting apache2/ssl/key
10584 actions.configure 22-08-30 15:17:37 [ INFO]: Multifile: /etc/apache2/sites-available/default-ssl.conf
10584 actions.configure 22-08-30 15:17:37 [ INFO]: Module: kopano-cfg
10584 actions.configure 22-08-30 15:17:37 [ WARNING]: run-parts: executing /etc/univention/letsencrypt/setup.d//dovecot
10584 actions.configure 22-08-30 15:17:37 [ WARNING]: run-parts: executing /etc/univention/letsencrypt/setup.d//postfix
10584 actions.configure 22-08-30 15:17:38 [ INFO]: Setting mail/postfix/ssl/key
10584 actions.configure 22-08-30 15:17:38 [ INFO]: Setting mail/postfix/ssl/certificate
10584 actions.configure 22-08-30 15:17:38 [ INFO]: Setting mail/postfix/ssl/cafile
10584 actions.configure 22-08-30 15:17:38 [ INFO]: Multifile: /etc/postfix/main.cf
10584 actions.configure 22-08-30 15:17:38 [ INFO]: Module: kopano-cfg
10584 actions.configure 22-08-30 15:17:39 [ INFO]: Tue 30 Aug 2022 03:17:39 PM EDT
10584 actions.configure 22-08-30 15:17:39 [ INFO]: Refreshing certificate for following domains:
10584 actions.configure 22-08-30 15:17:39 [ INFO]: sp-svr01.infolocity.net www.ivyinfosys.com www.omegafsi.com www.pennreserve.com ucs-sso.infolocity.net www.byteflight.net
10584 actions.configure 22-08-30 15:17:39 [ INFO]: Parsing account key…
10584 actions.configure 22-08-30 15:17:39 [ INFO]: Parsing CSR…
10584 actions.configure 22-08-30 15:17:39 [ INFO]: Found domains: www.ivyinfosys.com, ucs-sso.infolocity.net, sp-svr01.infolocity.net, www.omegafsi.com, www.byteflight.net, www.pennreserve.com
10584 actions.configure 22-08-30 15:17:39 [ INFO]: Getting directory…
10584 actions.configure 22-08-30 15:19:49 [ INFO]: Directory found!
10584 actions.configure 22-08-30 15:19:49 [ INFO]: Registering account…
root@sp-svr01:/var/log/univention#

And it just sits are “Registering account…” Thinking maybe a fresh install might help,I tried uninstalling using the UCS interface. Same result. I tried using the shell command with univention-app remove letsencrypt and it also gets stuck at “Registering account…”

Suggestions to solve this are greatly appreciated!

John

Migrate from a local domain to a public one
#2

My 2 cents here is not about fixing your existing issue - I have had my fair share of issues in the past - and all have lead me to stop using LE completely. The majority of the instances we have running are all behind firewalls on private networks and inaccessible from the public side. The LE client on UCS requires port 80 to be open and pointing to the UCS instance.

We’ve switched to using acme.sh using DNS verification of certs and a simple bash script to move the certs into place and update via ucr. If interested, I’ll be happy to share what we’ve done.

If you are able to roll back to 4.4-9 from a backup, you might have better success working through updates, upgrade to 5.01, updates again, and the final upgrade to 5.02. Skipping directly to the latest build has never worked for us in the past. Univention actually encourages the stepped approach.

#3

After much research, my primary problem was related to IPv6. The symptoms included delayed performance of command line utilities such as whois and traceroute. The IPv6 problems caused all sorts of issues like unable to reach the update servers at appcenter.software-univention.de.

The ‘fix’ was to make a change in /etc/gai.conf around line 54 where the notes state that “For sites that prefer IPv4 connections change the last line line to” and then uncomment “precedence ::ffff:0:0/96 100”

Once done and saved, the letsencrypt application worked as intended and ran without the “nonce” error, and, the other IPv6 issues cleared up.

The IPv6 problem masked the fact that my SSL cert was not updating and I did not notice that the expiration was the same day I was trying to upgrade from 4.4-9 to 5.

Lesson learned.

Mastodon