Problem: UCS Consumes A Lot Of Memory

Problem: UCS consumes a lot of memory

Even though not much processes are running your UCS system consumes a lot of memory which is not useed by processes or buffers/cache.
It might look as follows:

root@ucs:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           141G         60G         72G         15M        8,6G         80G
Swap:          375G          0B        375G

So there are 60G of memory used but top does not show any process using so much memory:

top - 13:07:21 up 285 days, 19:53,  2 users,  load average: 1,08, 1,00, 1,00
Tasks: 282 total,   3 running, 279 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4,6 us,  5,9 sy,  0,0 ni, 88,8 id,  0,1 wa,  0,0 hi,  0,6 si,  0,0 st
KiB Mem : 14854566+total, 76275712 free, 63281320 used,  8988636 buff/cache
KiB Swap: 39389184+total, 39389184+free,        0 used. 84186600 avail Mem
 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
13706 root      20   0  458032 258232   2624 S   0,0  0,2  47:34.64 rsyslogd
7562 listener  20   0 2636816  53028  27624 S   0,0  0,0   0:01.22 univention-dire
25959 root      20   0   85172  48068   5300 R  47,2  0,0   0:01.44 apt-get
19332 root      20   0 1382992  37464  29508 S   0,0  0,0  20:28.46 libvirtd
  420 root      20   0  144524  36996  36400 S   1,0  0,0 665:11.79 systemd-journa

Solution

Linux kernel has some issues with releasing ressources:

The second number reported by the following command should be well below 500:

root@ucs:~ # grep '^memory' /proc/cgroups

Check whether pam_systemd is currently used by the PAM stack:

root@ucs:~ # grep pam_systemd /etc/pam.d/common-session
session    optional   pam_systemd.so

The module is active as it isn’t commented out; it’s triggering a memory leak in the kernel.

Fixed by: updating to at least UCS 4.4 Errata 191 which deactivates the pam_systemd module by default. You may have to run ucr commit /etc/pam.d/common-session manually after the update. Verify that the module is commented-out with grep pam_systemd /etc/pam.d/common-session

If the update cannot be applied at the moment, you can work around it by modifying the PAM configuration files directly:

root@ucs:~ # sed -i '/pam_systemd/s/^session/#&/'  /etc/univention/templates/files/etc/pam.d/common-session.d/10univention-pam_common /etc/pam.d/common-session

Caution: this modifies a UCS template file, preventing newer versions from being installed automatically. You will have to reset that template back to its original content manually whenever you’ll be able to update to 4.4 Errata 191 or later.

Whichever route you chose, you have to reboot the server as it’s the only way to free the leaked kernel memory.