Problem
The swap of a virtualization host was 100% used, although there was still free main memory available. This did not cause a problem, but the customer’s monitoring scripts reported an alarm.
Analysis
Linux swaps pages to the hard disk as a precaution if they are not required. This makes it possible to make memory available quickly in the event of increased memory requirements. Otherwise, the system would first have to start swapping, which reduces performance. On virtualization hosts in particular, there are many memory pages that are rarely used due to the virtual machines, which is why the swap was used intensively here, even though there was still sufficient main memory available.
In addition, the main memory created by the swapped pages can be used as a cache.
In this respect, this is perfectly normal system behavior. In this specific case, the server had 256GB of main memory and only just under 4GB of swap configured. RedHat, for example, recommends 20% of the physical RAM as swap. In this case, that would have been approx. 50GB instead of the actual 4GB! See also: Do we really need swap on modern systems?
The customer was also unable to increase the size of the swap due to partitioning (instead of LVM). However, switching off the swap completely can quickly lead to problems in the event of memory bottlenecks, which is why this procedure is not recommended.
Solution
Reduction of the “swappiness”, i.e. how quickly the kernel swaps pages.
Insert in /etc/sysctl.conf:
vm.swappiness=1
After that use:
sysctl -p
According to the documentation, the swap is then only swapped in an “emergency”, without giving up the swap reserve completely. See: Memory paging - Wikipedia
This meant that the swap on the system was almost always empty and was only used in the event of an extreme memory bottleneck and then alerted.