Problem: UCS consumes a lot of memory
Even though not much processes are running your UCS system consumes a lot of memory which is not useed by processes or buffers/cache.
It might look as follows:
root@ucs:~# free -h
total used free shared buff/cache available
Mem: 141G 60G 72G 15M 8,6G 80G
Swap: 375G 0B 375G
So there are 60G of memory used but top does not show any process using so much memory:
top - 13:07:21 up 285 days, 19:53, 2 users, load average: 1,08, 1,00, 1,00
Tasks: 282 total, 3 running, 279 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4,6 us, 5,9 sy, 0,0 ni, 88,8 id, 0,1 wa, 0,0 hi, 0,6 si, 0,0 st
KiB Mem : 14854566+total, 76275712 free, 63281320 used, 8988636 buff/cache
KiB Swap: 39389184+total, 39389184+free, 0 used. 84186600 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13706 root 20 0 458032 258232 2624 S 0,0 0,2 47:34.64 rsyslogd
7562 listener 20 0 2636816 53028 27624 S 0,0 0,0 0:01.22 univention-dire
25959 root 20 0 85172 48068 5300 R 47,2 0,0 0:01.44 apt-get
19332 root 20 0 1382992 37464 29508 S 0,0 0,0 20:28.46 libvirtd
420 root 20 0 144524 36996 36400 S 1,0 0,0 665:11.79 systemd-journa
Solution
Linux kernel has some issues with releasing ressources:
The second number reported by the following command should be well below 500:
root@ucs:~ # grep '^memory' /proc/cgroups
Check whether pam_systemd is currently used by the PAM stack:
root@ucs:~ # grep pam_systemd /etc/pam.d/common-session
session optional pam_systemd.so
The module is active as it isn’t commented out; it’s triggering a memory leak in the kernel.
Fixed by: updating to at least UCS 4.4 Errata 191 which deactivates the pam_systemd module by default. You may have to run ucr commit /etc/pam.d/common-session manually after the update. Verify that the module is commented-out with grep pam_systemd /etc/pam.d/common-session
If the update cannot be applied at the moment, you can work around it by modifying the PAM configuration files directly:
root@ucs:~ # sed -i '/pam_systemd/s/^session/#&/' /etc/univention/templates/files/etc/pam.d/common-session.d/10univention-pam_common /etc/pam.d/common-session
Caution: this modifies a UCS template file, preventing newer versions from being installed automatically. You will have to reset that template back to its original content manually whenever you’ll be able to update to 4.4 Errata 191 or later.
Whichever route you chose, you have to reboot the server as it’s the only way to free the leaked kernel memory.