Reboot/shutdown sometimes fails and hangs

UCS - Univention Corporate Server
, ,
#1

Sometimes reboot / shutdown / halt / poweroff hangs on shutdown and the last message is something like this:

systemd[1]: Reached target Shutdown.

As most services are stopped at this point, the VM respectively the host can only be rebooted by power-cycling it.
This can be caused by multiple things:

  • If services are started but do not have an associated stop action, systemd may not stop them, which prevents shutting down resources still used by those processes.
  • This may for example include swap memory: If too much memory is used processes get pushed to swap space; swap then cannot be disabled, because doing this requires pulling back all processes into memory, where they don’t fit into. Thus swapoff fails.
  • Another known issue is NFS: It cannot be shut down clearly when processes continue using it or if the NFS server is unreachable.

The newer version 241 of systemd has this issue fixed:

  • by using the newer version 2.33 of util-linux all file systems are umounted even when any NFS server is currently unreachable.
  • by not using blocking system calls systemd-shutdown will no longer block and timeout after some minutes.

A backport of systemd from Debian 10 Buster, which is the basis for UCS 5.0, is available for UCS 4.4. It includes the following packages:

  • systemd 241
  • util-linux 2.33.1
  • shadow 4.5
  • bash-completion 2.8
  • debhelper 12.1.1
  • dh-autoreconf 17
  • meson 0.49.0

Only the first 4 packages are required on UCS-4 systems to fix the issue, the later 3 are only required for building the other packages.
As those packages are quite low-level updating them might break other packages! So far we do not have observed any breaking issues. Nevertheless for now these packages are provided in an extra repository, which must be activated explicitly.

ucr set repository/online/component/systemd44{=yes,/version=4.4,/description=systemd-reboot-fix}
univention-upgrade

If those packages are deemed stable they might get released as errata for UCS 4.4 in the future. Any feedback on the quality of those backports is very much appreciated.

Note: When booting with the updated systemd package, systemd tries to activate a kernel feature that is not yet present in the kernel used in UCS 4.4. Thus, the following message can be seen in the dmesg output. It can be ignored, as a fallback is done if the option is not supported.
kernel: cgroup: cgroup2: unknown option "nsdelegate"

1 Like
#2
#3
Mastodon