UCS and systemd - the basics

ucs-4-2
systemd

#1

UCS used Linux’ sysvinit up to version 4.1. Since UCS 4.2, systemd is used as default init system and replaces sysvinit. For now, sysvinit is still installed and available as a fallback. It can be selected as the second entry in the grub boot menu:

The first entry in the grub menu (which is the default) uses systemd.

Systemd offers a lot of new features and while it provides a broad layer of backwards compatibility to sysvinit services and service commands, getting familiar with the systemd-style commands has its advantages.

Which services?

To get an impression which services are present on your system and which of those are actually started, use the command systemctl without any arguments:

root@ucs-7125:~# systemctl
output of systemctl
  UNIT                                                                                      LOAD   ACTIVE SUB       DESCRIPTION
  proc-sys-fs-binfmt_misc.automount                                                         loaded active waiting   Arbitrary Executable File Formats File System Automount Point
  sys-devices-pci0000:00-0000:00:01.1-ata1-host0-target0:0:0-0:0:0:0-block-sr0.device       loaded active plugged   QEMU_DVD-ROM
  sys-devices-pci0000:00-0000:00:03.0-virtio0-net-eth0.device                               loaded active plugged   Virtio network device
  sys-devices-pci0000:00-0000:00:04.0-sound-card0.device                                    loaded active plugged   82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller (QEMU Virtual Machine
  sys-devices-pci0000:00-0000:00:05.0-virtio1-virtio\x2dports-vport1p1.device               loaded active plugged   /sys/devices/pci0000:00/0000:00:05.0/virtio1/virtio-ports/vport1p1
  sys-devices-pci0000:00-0000:00:07.0-virtio2-block-vda-vda1.device                         loaded active plugged   /sys/devices/pci0000:00/0000:00:07.0/virtio2/block/vda/vda1
  sys-devices-pci0000:00-0000:00:07.0-virtio2-block-vda-vda2.device                         loaded active plugged   /sys/devices/pci0000:00/0000:00:07.0/virtio2/block/vda/vda2
  sys-devices-pci0000:00-0000:00:07.0-virtio2-block-vda-vda5.device                         loaded active plugged   LVM PV 0n7wg5-p5d4-atO8-3eXH-2rqT-dG6Q-fqk0lx on /dev/vda5 5
  sys-devices-pci0000:00-0000:00:07.0-virtio2-block-vda.device                              loaded active plugged   /sys/devices/pci0000:00/0000:00:07.0/virtio2/block/vda
  sys-devices-platform-serial8250-tty-ttyS1.device                                          loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS1
  sys-devices-platform-serial8250-tty-ttyS2.device                                          loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS2
  sys-devices-platform-serial8250-tty-ttyS3.device                                          loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS3
  sys-devices-pnp0-00:04-tty-ttyS0.device                                                   loaded active plugged   /sys/devices/pnp0/00:04/tty/ttyS0
  sys-devices-virtual-block-dm\x2d0.device                                                  loaded active plugged   /sys/devices/virtual/block/dm-0
  sys-devices-virtual-block-dm\x2d1.device                                                  loaded active plugged   /sys/devices/virtual/block/dm-1
  sys-devices-virtual-net-docker0.device                                                    loaded active plugged   /sys/devices/virtual/net/docker0
  sys-subsystem-net-devices-docker0.device                                                  loaded active plugged   /sys/subsystem/net/devices/docker0
  sys-subsystem-net-devices-eth0.device                                                     loaded active plugged   Virtio network device
  -.mount                                                                                   loaded active mounted   /
  boot.mount                                                                                loaded active mounted   /boot
  dev-hugepages.mount                                                                       loaded active mounted   Huge Pages File System
  dev-mqueue.mount                                                                          loaded active mounted   POSIX Message Queue File System
  run-rpc_pipefs.mount                                                                      loaded active mounted   /run/rpc_pipefs
  sys-kernel-debug.mount                                                                    loaded active mounted   Debug File System
  var-lib-docker-overlay.mount                                                              loaded active mounted   /var/lib/docker/overlay
  acpid.path                                                                                loaded active running   ACPI Events Check
  systemd-ask-password-console.path                                                         loaded active waiting   Dispatch Password Requests to Console Directory Watch
  systemd-ask-password-wall.path                                                            loaded active waiting   Forward Password Requests to Wall Directory Watch
  acpid.service                                                                             loaded active running   ACPI event daemon
  apache2.service                                                                           loaded active running   LSB: Apache2 web server
  atd.service                                                                               loaded active running   Deferred execution scheduler
  bind9.service                                                                             loaded active exited    LSB: bind9 Domain Name Server (DNS)
  cgroupfs-mount.service                                                                    loaded active exited    LSB: Set up cgroupfs mounts.
  console-setup.service                                                                     loaded active exited    LSB: Set console font and keymap
  cron.service                                                                              loaded active running   Regular background program processing daemon
  dbus.service                                                                              loaded active running   D-Bus System Message Bus
  docker.service                                                                            loaded active running   Docker Application Container Engine
  getty@tty1.service                                                                        loaded active running   Getty on tty1
  heimdal-kdc.service                                                                       loaded active running   LSB: Start KDC server
  ifplugd.service                                                                           loaded active running   LSB: Brings up/down network automatically
  inetd.service                                                                             loaded active running   Internet superserver
  kbd.service                                                                               loaded active exited    LSB: Prepare console
  kdm.service                                                                               loaded active running   LSB: X display manager for KDE
  keyboard-setup.service                                                                    loaded active exited    LSB: Set preliminary keymap
  kmod-static-nodes.service                                                                 loaded active exited    Create list of required static device nodes for the current kernel
  lvm2-monitor.service                                                                      loaded active exited    Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
  lvm2-pvscan@254:5.service                                                                 loaded active exited    LVM2 PV scan on device 254:5
  memcached.service                                                                         loaded active running   memcached daemon
  nagios-nrpe-server.service                                                                loaded active running   LSB: Start/Stop the Nagios remote plugin execution daemon
  networking.service                                                                        loaded active running   LSB: Raise network interfaces.
  nfs-common.service                                                                        loaded active running   LSB: NFS support files common to client and server
  nfs-kernel-server.service                                                                 loaded active exited    LSB: Kernel NFS server support
  nscd.service                                                                              loaded active running   LSB: Starts the Name Service Cache Daemon
  ntp.service                                                                               loaded active running   LSB: Start NTP daemon
  postfix.service                                                                           loaded active running   LSB: start and stop the Postfix Mail Transport Agent
  quota.service                                                                             loaded active exited    Initial Check File System Quotas
  rc-local.service                                                                          loaded active exited    /etc/rc.local Compatibility
  rpcbind.service                                                                           loaded active running   LSB: RPC portmapper replacement
  rsyslog.service                                                                           loaded active running   System Logging Service
  saslauthd.service                                                                         loaded active exited    LSB: saslauthd startup script
  slapd.service                                                                             loaded active running   LSB: OpenLDAP standalone server (Lightweight Directory Access Protocol)
  ssh.service                                                                               loaded active running   OpenBSD Secure Shell server
  stunnel4.service                                                                          loaded active exited    LSB: Start or stop stunnel 4.x (SSL tunnel for network daemons)
  systemd-fsck@dev-disk-by\x2duuid-61672b11\x2d7a1b\x2d4e63\x2db629\x2df943594ef924.service loaded active exited    File System Check on /dev/disk/by-uuid/61672b11-7a1b-4e63-b629-f943594ef924
  systemd-journald.service                                                                  loaded active running   Journal Service
  systemd-logind.service                                                                    loaded active running   Login Service
  systemd-modules-load.service                                                              loaded active exited    Load Kernel Modules
  systemd-random-seed.service                                                               loaded active exited    Load/Save Random Seed
  systemd-remount-fs.service                                                                loaded active exited    Remount Root and Kernel File Systems
  systemd-setup-dgram-qlen.service                                                          loaded active exited    Increase datagram queue length
  systemd-sysctl.service                                                                    loaded active exited    Apply Kernel Variables
  systemd-tmpfiles-setup-dev.service                                                        loaded active exited    Create Static Device Nodes in /dev
  systemd-tmpfiles-setup.service                                                            loaded active exited    Create Volatile Files and Directories
  systemd-udev-settle.service                                                               loaded active exited    udev Wait for Complete Device Initialization
  systemd-udev-trigger.service                                                              loaded active exited    udev Coldplug all Devices
  systemd-udevd.service                                                                     loaded active running   udev Kernel Device Manager
  systemd-update-utmp.service                                                               loaded active exited    Update UTMP about System Boot/Shutdown
  systemd-user-sessions.service                                                             loaded active exited    Permit User Sessions
  udev-finish.service                                                                       loaded active exited    Copy rules generated while the root was ro
  univention-directory-listener.service                                                     loaded active exited    LSB: Univention Directory Listener Daemon
  univention-directory-notifier.service                                                     loaded active exited    LSB: Univention Directory Notifier Daemon
  univention-directory-policy.service                                                       loaded active exited    LSB: Univention Directory Policy
  univention-firewall.service                                                               loaded active exited    LSB: Univention iptables configuration
● univention-maintenance.service                                                            loaded failed failed    LSB: Univention Updater
  univention-management-console-server.service                                              loaded active running   LSB: Univention Management Console Server
  univention-management-console-web-server.service                                          loaded active running   LSB: Univention Management Console Web Server
  univention-network-common.service                                                         loaded active exited    LSB: save DHCP address in LDAP
  univention-runit.service                                                                  loaded active running   LSB: Univention process supervision
  univention-saml.service                                                                   loaded active running   LSB: Univention Security Assertion Markup Language integration
  univention-system-setup-boot.service                                                      loaded active exited    LSB: Univention System Setup on boot
  -.slice                                                                                   loaded active active    Root Slice
  system-getty.slice                                                                        loaded active active    system-getty.slice
  system-lvm2\x2dpvscan.slice                                                               loaded active active    system-lvm2\x2dpvscan.slice
  system-systemd\x2dfsck.slice                                                              loaded active active    system-systemd\x2dfsck.slice
  system.slice                                                                              loaded active active    System Slice
  user.slice                                                                                loaded active active    User and Session Slice
  acpid.socket                                                                              loaded active running   ACPID Listen Socket
  dbus.socket                                                                               loaded active running   D-Bus System Message Bus Socket
  dm-event.socket                                                                           loaded active listening Device-mapper event daemon FIFOs
  docker.socket                                                                             loaded active running   Docker Socket for the API
  lvm2-lvmetad.socket                                                                       loaded active listening LVM2 metadata daemon socket
  syslog.socket                                                                             loaded active running   Syslog Socket
  systemd-initctl.socket                                                                    loaded active listening /dev/initctl Compatibility Named Pipe
  systemd-journald-dev-log.socket                                                           loaded active running   Journal Socket (/dev/log)
  systemd-journald.socket                                                                   loaded active running   Journal Socket
  systemd-shutdownd.socket                                                                  loaded active listening Delayed Shutdown Socket
  systemd-udevd-control.socket                                                              loaded active running   udev Control Socket
  systemd-udevd-kernel.socket                                                               loaded active running   udev Kernel Socket
  dev-mapper-vg_ucs\x2dswap_1.swap                                                          loaded active active    /dev/mapper/vg_ucs-swap_1
  basic.target                                                                              loaded active active    Basic System
  cryptsetup.target                                                                         loaded active active    Encrypted Volumes
  getty.target                                                                              loaded active active    Login Prompts
  graphical.target                                                                          loaded active active    Graphical Interface
  local-fs-pre.target                                                                       loaded active active    Local File Systems (Pre)
  local-fs.target                                                                           loaded active active    Local File Systems
  mail-transport-agent.target                                                               loaded active active    Mail Transport Agent
  multi-user.target                                                                         loaded active active    Multi-User System
  network-online.target                                                                     loaded active active    Network is Online
  network.target                                                                            loaded active active    Network
  nss-lookup.target                                                                         loaded active active    Host and Network Name Lookups
  paths.target                                                                              loaded active active    Paths
  remote-fs-pre.target                                                                      loaded active active    Remote File Systems (Pre)
  remote-fs.target                                                                          loaded active active    Remote File Systems
  rpcbind.target                                                                            loaded active active    RPC Port Mapper
  slices.target                                                                             loaded active active    Slices
  sockets.target                                                                            loaded active active    Sockets
  sound.target                                                                              loaded active active    Sound Card
  swap.target                                                                               loaded active active    Swap
  sysinit.target                                                                            loaded active active    System Initialization
  timers.target                                                                             loaded active active    Timers
  systemd-tmpfiles-clean.timer                                                              loaded active waiting   Daily Cleanup of Temporary Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

131 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

systemctl is the commandline interface to query or send commands to systemd. To list also inactive units, use systemctl --all.

Alternatively, systemd can also list all unit files:

systemctl list-unit-files

Working with services

Check the status of a service:

root@ucs-7125:~# systemctl status apache2.service
example output of systemctl status
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2)
   Active: active (running) since Di 2017-04-04 21:26:50 CEST; 3 weeks 0 days ago
  Process: 19639 ExecReload=/etc/init.d/apache2 reload (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/apache2.service
           ├─ 4905 /usr/sbin/apache2 -k start
           ├─19666 /usr/sbin/apache2 -k start
           ├─19667 /usr/sbin/apache2 -k start
           ├─19668 /usr/sbin/apache2 -k start
           ├─19669 /usr/sbin/apache2 -k start
           ├─19690 /usr/sbin/apache2 -k start
           ├─19723 /usr/sbin/apache2 -k start
           ├─20576 /usr/sbin/apache2 -k start
           ├─20577 /usr/sbin/apache2 -k start
           ├─20579 /usr/sbin/apache2 -k start
           └─20664 /usr/sbin/apache2 -k start

Apr 26 06:25:25 master systemd[1]: Reloading LSB: Apache2 web server.
Apr 26 06:25:26 master apache2[4500]: Reloading web server: apache2.
Apr 26 06:25:26 master systemd[1]: Reloaded LSB: Apache2 web server. 

We can also use the unit commands stop, start and restart in the same fashion.
The schema is always the same:

systemctl <UNIT COMMAND> <UNIT NAME> 

systemd comes with autocompletion support for unit commands and for unit names - just TAB along!

Enabling and disabling services

systemd also manages which services are started automatically and which are not. Disabling a service prevents automatic starting:

root@ucs-7125:~# systemctl disable ntp.service
Synchronizing state for ntp.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d ntp defaults
Executing /usr/sbin/update-rc.d ntp disable

Enabling it again:

root@ucs-7125:~# systemctl enable ntp.service
Synchronizing state for ntp.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d ntp defaults
Executing /usr/sbin/update-rc.d ntp enable

A disabled service can still be started manually. To also prevent a manual start, the unit command mask must be used:

root@ucs-7125:~# systemctl mask ntp.service 
Created symlink from /etc/systemd/system/ntp.service to /dev/null.

Unmasking:

root@ucs-7125:~# systemctl unmask ntp.service
Removed symlink /etc/systemd/system/ntp.service.

System commands

Besides units, we can also interact with the whole system:

systemctl rescue                          Enter system rescue mode
systemctl poweroff                        Shut down and power-off the system
systemctl reboot                          Shut down and reboot the system

Analyzing boot

If you run into boot problems, e.g. booting your machine becomes very slow suddenly, systemd has you covered:
systemd-analyze can analyze the boot process:

root@ucs-7125:~# systemd-analyze time
Startup finished in 6.991s (kernel) + 18.678s (userspace) = 25.669s
root@ucs-7125:~# systemd-analyze critical-chain
example output of systemd-analyze critical chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @18.672s
└─multi-user.target @18.672s
  └─apache2.service @16.696s +1.975s
    └─univention-management-console-web-server.service @7.840s +8.849s
      └─univention-management-console-server.service @6.848s +991ms
        └─slapd.service @4.579s +2.267s
          └─basic.target @4.562s
            └─sockets.target @4.562s
              └─docker.socket @4.559s +2ms
                └─sysinit.target @4.559s
                  └─nfs-common.service @4.490s +68ms
                    └─rpcbind.target @4.490s
                      └─rpcbind.service @4.452s +37ms
                        └─network-online.target @4.452s
                          └─network.target @4.452s
                            └─networking.service @820ms +3.631s
                              └─local-fs.target @819ms
                                └─var-lib-docker-overlay.mount @8.026s
                                  └─local-fs-pre.target @753ms
                                    └─systemd-remount-fs.service @735ms +15ms
                                      └─keyboard-setup.service @218ms +516ms
                                        └─systemd-udevd.service @198ms +13ms
                                          └─systemd-tmpfiles-setup-dev.service @158ms +34ms
                                            └─kmod-static-nodes.service @147ms +9ms
                                              └─system.slice @145ms
                                                └─-.slice @145ms

systemd can also tell you which process took how long to start. This can be done with systemd-analyze blame.
We can also get a nice SVG image of this via systemd-analyze plot > systemd-analyze.svg

How about logs? :books:

systemd comes with a system service that collects and stores logging data called journald. By now, UCS does not yet make use of journald actively (e.g. Univention services still use their own logging mechanism via univention.debug), but we can still use it for most Debian-based services, e.g. ntp or the NRPE daemon:

root@ucs-7125:~# journalctl --unit=nagios-nrpe-server.service 
Summary
-- Logs begin at So 2017-04-23 18:21:40 CEST, end at So 2017-04-23 18:54:34 CEST. --
Apr 23 18:21:49 ucs-7125 systemd[1]: Starting LSB: Start/Stop the Nagios remote plugin execution daemon...
Apr 23 18:21:50 ucs-7125 nagios-nrpe-server[1363]: Starting nagios-nrpe: nagios-nrpe.
Apr 23 18:21:50 ucs-7125 systemd[1]: PID file /var/run/nagios/nrpe.pid not readable (yet?) after start.
Apr 23 18:21:50 ucs-7125 nrpe[1466]: Starting up daemon
Apr 23 18:21:50 ucs-7125 systemd[1]: Started LSB: Start/Stop the Nagios remote plugin execution daemon.
Apr 23 18:21:50 ucs-7125 nrpe[1466]: Server listening on 0.0.0.0 port 5666.
Apr 23 18:21:50 ucs-7125 nrpe[1466]: Server listening on :: port 5666.
Apr 23 18:21:50 ucs-7125 nrpe[1466]: Listening for connections on port 0
Apr 23 18:21:50 ucs-7125 nrpe[1466]: Allowing connections from: 192.168.122.227

journalctl also allows us the specify a time range:

root@ucs-7125:~# journalctl --unit=nagios-nrpe-server.service \
                            --since "2017-04-23 16:45" \
                            --until "2017-04-23 18:55" 

This is quite handy compared to searching and grep-ing for a timestamp in a strange date format in a text log file.

Of course we can also follow the log, just like tail -f:

root@ucs-7125:~# journalctl -u nscd.service --follow

What else?

Did I miss something? Do you have additional tips and tricks? Feel free to add them in the comments.