Update to UCS 4.3-2 broke my system due to not loading crc32c driver

UCS - Univention Corporate Server
#1

I was running UCS 4.3-0 errata 28 on Debian Stretch in a Synology Virtual machine.

I finally gave in to the notification that this version was no longer receiving security updates and installed the latest release 4.3-2 through the UCS managament console.

In the window messages started flowing about all the packages that were being upgraded, so far so good.
After a while, connection to the console was lost, that was excepted so the disconnect message said. After about half an hour, when there was no more system activity for quite a while, i rebooted the virtual machine.

Now the virtual machine wont reboot, possibly because the kernel was upgraded, and this error occurs in Grub before the mounts take place:

alg: hash: digest failed on test 1 for crc32c-intel: RET=126

Because of this error, no disks are mounted and the system does not start.

Troubleshooting reveals i should add ‘crc32c’ to the initramfs modules and rebuild initramfs (the kernel) , but i don’t know how to do that on the disk which i can only mount in rescue mode through the ucs installer disk.

Any help is welcome

#2

Hey,

after booting from your rescue CD (whichever that is) you’ll have to mount a lot of file systems, enter the mounted hierarchy, add the module to the list of modules to add to the initial ram disk and rebuild the initial ram disks. Do the following (adjust values as needed for your installation):

  1. Activate volume groups (only if LVM is used): vgchange -a y
  2. Mount the device containing your root file system: mount /dev/mapper/vg_ucs-vg_root /mnt
  3. Mount the device containing /boot (if /boot is on a separate file system which it certainly is if you’ve installed using LVM): mount /dev/sda1 /mnt/boot
  4. Mount the proc and sys pseudo file systems: mount -t proc none /mnt/proc ; mount -t sysfs none /mnt/sys
  5. Mount the /dev tree from your rescue CD into the mounted root hierarchy: mount /dev /mnt/dev -o bind
  6. Edit the file /mnt/etc/initramfs-tools/modules and add your module there (one module per line).
  7. Enter the mounted hierarchy: chroot /mnt /bin/bash
  8. Rebuild your initial ram disks: update-initramfs -k all -u
  9. Exit the chroot and reboot: exit followed by reboot

However, I’m not convinced that this is really the root cause of your machine not booting. What’s the very last thing you see before the boot process stalls?

Additionally, try enabling debug messages for your boot process. During the boot when you’re in the grub shell, edit the configuration and remove both quiet and splash from the linux … line and add debug instead. Then continue booting and see what additional information is output.

Kind regards
mosu

#3

Hi Mosu,

Thank you very much for your reply. I founbd many of the steps you mention already by googling for this kind of error, bu every time i had to give up because the rescue mode of the UCS Installer disk is very limited in the commands supplied. For instance, the command chroot is unknown…
I have started using the Knoppix LiveCD for booting, and will retry doing the steps you outlined (which are more detailed than any other instructions i found, thanks for that).
As a second approach i already started setting uop a virtual machine from scratch & installing all Kopano packages i need, and copying over the attachments and sql database, in case i couldn’t repair the Original system (i was able to mount the volume through the Knoppix LiveCD and copy the files to a network drive)crc32c .
Attached yuo find a screenshot of the console messages, and i have concluded that the crc32c-intel error message is the culprint (the other errors appear even on a new setup virtual machine)

#4

Hey,

thanks. Looks like your initramfs might be broken; that update-initramfs should take care of it.

BTW: I suggest you don’t use Knoppix but grml instead. It’s more lightweight, and it includes a nice command grml-chroot which you can use instead of my chroot above. The advantage is that it already takes care of mounting proc, sys and dev — meaning you can skip steps 4 and 5. Less typing, fewer possibilities for mistakes.

m.

#5

Hi,

I get as far as the chroot command, which gives:

chroot: failed to run the command ‘bin/bash’: No such file or directory

I have checked, and the complete /bin folder is missing. Can i just copy that over from somewhere else? In another VM, il learned its full of symlinks, so a plain copy probably wont work.

NB step 2 : my device name is /dev/mapper/vg_ucs-root

#6

Hi,

first, usually the boot menu offers to boot different kernel versions. Have you tried to boot to the previous one? This should at least cover the “crc32c” module issue.

Second, if you have been able to mount your vg_ucs-root LV and there is really no /bin we have to think of the filesystem being totally wrecked. In this case your only option is to restore.

Third, I am unsure about the reason for this. It might be the hard power-off you performed. Probably the updater was still running, just the web- frontend (UMC) timed out? Perhaps you can start the upgrade next time through command line (but on your virtual console, not through ssh) with univention-upgrade?

/CV

#7

Hi,

Unfortunately there is no previous kernel to chosse from the boot menu.

Currently i am trying to recreate a /bin folder to at least get mysql going so i can dump my kopano database. It has been decades since i last worked with databases from the command line. Its a innodb database so i cant just copy the /var/lib/mysql folder, but i have a hard time getting mysql running , especially getting it to find the databases in de /mnt/var/lib/mysql folder (i cant change the datadir in my.conf bceause that is readonly due to using a livecd to mount the /dev/mapper/vg_ucs-root.)

#8

Its a innodb database so i cant just copy the /var/lib/mysql folder

Why not?
You cannot do it while the database is running, indeed.

But as your database is not running anyways…

So go ahead an just copy these files (but yes, all of them!) to their new location, start the database server and you should be fine!

/CV

#9

ok, will try :slight_smile:

Last time i tried mysql wouldnt start afterwards…, but i think i didnt stop the server before copying. ill keep you posted.

#10

Well, copying the database worked, but getting Kopano back up running nicely is more diificult, mostly because directory permissions etc are not correct anymore. However i got far enough to do a kopano-backup of all user mail stores which i can restore into a fresh kopano install.

Lesson learned; first make an export of the virtual machine before updating…

Mastodon