Update Master / Slave fails

It should work out OK if you re-join all other servers afterwards. Otherwise the state of the LDAP servers won’t be in sync which will prove problematic almost right away.

In fact, I would probably restore the master, join all the other servers (e.g. the slave), remove the init scripts that prevented the upgrade the last time and only then upgrade.

allright…
Backup master restored, slave rejoined succesfully, now start all over again with the upgrade…

This is the output of univention-upgrade now, so what exactly would you suggest next?
I don’t think I understand which scripts to delete here…

updater-log-ucs-master.txt (64.2 KB)

Thanks again for your help!
Sascha

There are still 4 packages, which displays a WARNING. These should have the “rc” state, that means that the packages are removed and left some configuration files (and init scripts) behind. You could verify that by the following command

dpkg -l | grep -v ^ii

So moving the init scripts to an other location will take you a step further.

Please do not move all init scripts of installed packages. Only the 4 which displays a WARNING could be moved. All other init scripts are still required and will be migrated to systemd-sysv.

ohhh…sorry for being so dumb…this figures now…
will report…
thanks!

I do really feel embarrassed… seems to work now…
Think i got you completely wrong when you said:

So, moving the old init scripts should be fine.

:man_facepalming:

Thanks
Sascha

Yeah, saying “old init scripts” was probably too ambiguous. I really meant only those old init scripts that warnings were printed for during the first upgrade attempt (console-screen.sh, plucs, portmap, univention-ad-connector). Purging existing packages in state rc as @peichert has said will most likely get rid of exactly those scripts and should definitely be done before the upgrade attempt. Maybe… purge them first, then check if any of those four init scripts I’ve listed still exist and move them out of the way. Then upgrade.

well, we have an upgraded and ready to use Master now, however now the slave has become a disaster.
I had the same issue with the unneeded scripts in /etc/init.d and moved 4 scripts out of the way.
For no reason i can see right now, the upgrade after running pretty well stopped and now i have a messed up system.
As this is the mail server, i would really prefer not to restore a backup, if not absolutely needed.

these are the problems now, kindly asking for help:



dpkg --audit
Die folgenden Pakete wurden entpackt, aber noch nicht konfiguriert.
Sie müssen mit dpkg --configure oder dem Konfigurations-Menüeintrag in
dselect konfiguriert werden, damit sie ordnungsgemäß funktionieren:
 slapd                OpenLDAP server (slapd)

Die folgenden Pakete sind nur halb konfiguriert, wahrscheinlich durch
Probleme während der ersten Konfiguration. Die Konfiguration sollte mit
dpkg --configure <Paket> oder mit dem Konfigurations-Menüeintrag in
dselect erneut versucht werden:
 plymouth             boot animation, logger and I/O multiplexer
dpkg --configure -a
plymouth (0.9.0-9A~4.2.0.201703231915) wird eingerichtet ...
update-initramfs: deferring update (trigger activated)
update-rc.d: error: expected NN after start
usage: update-rc.d [-n] [-f] <basename> remove
       update-rc.d [-n] <basename> defaults [NN | SS KK]
       update-rc.d [-n] <basename> start|stop NN runlvl [runlvl] [...] .
       update-rc.d [-n] <basename> disable|enable [S|2|3|4|5]
		-n: not really
		-f: force

The disable|enable API is not stable and might change in the future.
dpkg: Fehler beim Bearbeiten des Paketes plymouth (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
dpkg: Abhängigkeitsprobleme verhindern Konfiguration von slapd:
 slapd hängt ab von libldap-2.4-2 (= 2.4.42+dfsg-2.224.201706201314); aber:
  Version von libldap-2.4-2:amd64 auf dem System ist 2.4.42+dfsg-2.A~4.2.0.201703081826.
 slapd hängt ab von libperl5.14 (>= 5.14.2); aber:
  Paket libperl5.14 ist nicht installiert.

dpkg: Fehler beim Bearbeiten des Paketes slapd (--configure):
 Abhängigkeitsprobleme - verbleibt unkonfiguriert
Trigger für initramfs-tools (0.115~bpo70+1.35.201410091545) werden verarbeitet ...
update-initramfs: Generating /boot/initrd.img-4.1.0-ucs227-amd64
W: plymouth: The plugin label.so is missing, the selected theme might not work as expected.
W: plymouth: You might want to install the plymouth-themes package to fix this.
Fehler traten auf beim Bearbeiten von:
 plymouth
 slapd

after rejoining the slave to the master now i have this:

dpkg --configure -a
plymouth (0.9.0-9A~4.2.0.201703231915) wird eingerichtet ...
update-initramfs: deferring update (trigger activated)
update-rc.d: error: expected NN after start
usage: update-rc.d [-n] [-f] <basename> remove
       update-rc.d [-n] <basename> defaults [NN | SS KK]
       update-rc.d [-n] <basename> start|stop NN runlvl [runlvl] [...] .
       update-rc.d [-n] <basename> disable|enable [S|2|3|4|5]
		-n: not really
		-f: force

The disable|enable API is not stable and might change in the future.
dpkg: Fehler beim Bearbeiten des Paketes plymouth (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
slapd (2.4.42+dfsg-2.A~4.2.0.201703081826) wird eingerichtet ...
File: /etc/init.d/slapd
Multifile: /etc/ldap/slapd.conf
  Backing up /etc/ldap/slapd.conf in /var/backups/slapd-2.4.42+dfsg-2.A~4.2.0.201703081826... done.
[ ok ] Stopping ldap server(s): slapd ...done.
[....] Check database: ...[info] back-bdb was linked against 5.3, but db5.3-util package does not seem to be installed.
[info] Skipping /usr/bin/db5.3_recover.
[FAIL] Starting ldap server(s): slapd ...failed.
[info] 5993618f bdb(dc=holz-joki,dc=de): BDB1538 Program version 5.3 doesn't match environment version 5.1 5993618f bdb_db_open: database "dc=holz-joki,dc=de" cannot be opened, err -30969. Restore from backup! 5993618f backend_startup_one (type=bdb, suffix="dc=holz-joki,dc=de"): bi_db_open failed! (-30969) slap_startup failed.
invoke-rc.d: initscript slapd, action "start" failed.
dpkg: Fehler beim Bearbeiten des Paketes slapd (--configure):
 Unterprozess installiertes post-installation-Skript gab den Fehlerwert 1 zurück
dpkg: Abhängigkeitsprobleme verhindern Konfiguration von plymouth-themes:
 plymouth-themes hängt ab von plymouth (= 0.9.0-9A~4.2.0.201703231915); aber:
  Paket plymouth ist noch nicht konfiguriert.

dpkg: Fehler beim Bearbeiten des Paketes plymouth-themes (--configure):
 Abhängigkeitsprobleme - verbleibt unkonfiguriert
Trigger für initramfs-tools (0.115~bpo70+1.35.201410091545) werden verarbeitet ...
update-initramfs: Generating /boot/initrd.img-4.1.0-ucs227-amd64
Fehler traten auf beim Bearbeiten von:
 plymouth
 slapd
 plymouth-themes

Will leave it for today…
thanks for helping, really appreciated!
Sascha

I… don’t know how to continue the upgrade at that point, to be honest.

I would still recommend some kind of restoration. However, the only truly important part is that the system files are restored, not the mail server content. Meaning you should be able to do something like this:

  1. backup the current content of the mail server’s database and all associated directories with content (e.g. for Kopano you’d backup both the database and the directory containing the attachments; with Dovecot you’d backup all the maildirs); maybe even create a full backup at this point, just to be sure
  2. restore from backup to before the update
  3. restore the mail server content from the backup in step 1
  4. purge any package still in rc state (see dpkg -l | grep -i '^rc')
  5. remove the init files that the update process threw warnings about
  6. rejoin the slave
  7. retry the upgrade

Sorry if I don’t have a better way, but the “restore & retry” way seems way safer and easier to me than trying to fiddle around with LDAP components where the database tools & libraries don’t match.

Yeah…seems legit…
I would suggest the following:

  • Restore a backup from yesterday to another vm
  • copy the contents of /var/spool/dovecot/ to the restored machine from the messed up one.
  • then proceed with the upgrade…someday…maybe next week…sometime…:slight_smile:

The changes of one day in the OX DB are pretty negligible, i would consider, dovecot is the important part.
Only thing i would like to know:
Anything i must be aware of, when copying the content of /var/spoo/dovecot?
Would that work out of the box in your eyes, or do i miss something here?

thanks
Sascha

You’re right, I have not clearly pointed to move only the 4 with a WARNING neither I said to move all of them. But some posts above it were written. I’m very sorry for the follow up problems. I will try to be more specific in future.

For your Slave, I hope you also move only init scripts with a WARNING and verify that the package is in “rc” state, as @Moritz_Bunkus pointed out.

From the error message it seems that “plymouth” is missing some information. And for “ldap” a dependency is missing (libldap is still from UCS 4.1 and libperl was not updated).

Hey peichert, no worries plz, all good!
the master is up and running, so we are happy here now. :smiley:
As for the slave, of course i only removed the WARNING scripts this time, but the problem was something else this time…
During the update the VM simply stopped working, due to a bug in qcow2 and snapshots, renderering the Vdisk corrupt…so something new, not having anything to do with UCS…but thats the way it goes sometimes, or as they say in germany…“haste kacke am schuh, haste kacke am schuh”… LOL!
So the problem here now is, that the Upgrade progress stopped suddenly and completely unexpected, and now we have an inconsistent system state.

The question is, do you think it can be repaired (in a decent amount of work and time) or would you consider it best to restore the backup and then rsync the mail-spool directory, as i suggested?
And if the second is a good idea, do i miss something when recovering dovecot mails here?

Thanks
Sascha

In general one can try resolve a failed upgrade manually, if one know the problem, and do it package by package to fix it. After resolving the situation run univention-upgrade and the setup should continue. That could take many many hours and is not recommended, because also some UCR-Values (version and errata level) has to be adjusted to do the trick. Also by installing packages manually “dpkg” gets confused and did not see the package as a automatically installed package by some dependency. So later auto-remove will not work as expected to remove all unneeded packages.

But as you don’t know the state of your harddisk in the VM, because the qcow2 is damaged, its seems the best to start there and resolve problems at KVM level first. If the physical harddisk is damaged, this could happen again in the future. You could check the SMART values and syslog of your virtualization node.

No, don’t wory about the corrupted VD, i fixed the disk and resolved the underlying problem.
The disk now works perfectly.

Allright,
Restored the slave from the backup, rsynced /var/spool/dovecot from the defunct VM to the restored VM, et voilá…it all works again.

:confetti_ball::sparkler::fireworks::sparkler::sparkler::fireworks::fireworks::confetti_ball::tada:

Will retry the update this weekend, hopefully better vibrations around me then…
Thanks for all your incredible help!

Cheers
Sascha

Glad to hear that! Good luck with the next try.

ok, just to come to a conclusion here…updates ran fine, systems up to date, rc purged…
All good!

YEAH!

wish all of you a great weekend!
Sascha

1 Like

I’m usually not a fan of the inflation of superlatives and the resulting hyperbole, but this is truly great!

One lesson I’ve learned from this myself is to always check packages in “removed but configuration still present” state (dpkg -l | grep '^rc') before doing big upgrades. It’s now part of my personal TODO list for UCS upgrades.

1 Like

yup! Hallelujah!
thanks again for your help…

Really interessting. We had similar issues trying to upgrade in a test environment, a while ago. I think it’s time for the next try.

Mastodon