Problem:Swap full 100% on virtualization

Knowledge Base Community
, , ,
1

Problem

The swap of a virtualization host was 100% used, although there was still free main memory available. This did not cause a problem, but the customer’s monitoring scripts reported an alarm.

Analysis

Linux swaps pages to the hard disk as a precaution if they are not required. This makes it possible to make memory available quickly in the event of increased memory requirements. Otherwise, the system would first have to start swapping, which reduces performance. On virtualization hosts in particular, there are many memory pages that are rarely used due to the virtual machines, which is why the swap was used intensively here, even though there was still sufficient main memory available.

In addition, the main memory created by the swapped pages can be used as a cache.

In this respect, this is perfectly normal system behavior. In this specific case, the server had 256GB of main memory and only just under 4GB of swap configured. RedHat, for example, recommends 20% of the physical RAM as swap. In this case, that would have been approx. 50GB instead of the actual 4GB! See also: Do we really need swap on modern systems?

The customer was also unable to increase the size of the swap due to partitioning (instead of LVM). However, switching off the swap completely can quickly lead to problems in the event of memory bottlenecks, which is why this procedure is not recommended.

Solution

Reduction of the “swappiness”, i.e. how quickly the kernel swaps pages.

Insert in /etc/sysctl.conf:

vm.swappiness=1

After that use:

sysctl -p

According to the documentation, the swap is then only swapped in an “emergency”, without giving up the swap reserve completely. See: Memory paging - Wikipedia

This meant that the swap on the system was almost always empty and was only used in the event of an extreme memory bottleneck and then alerted.

Hier ist eine weiterführende, professionell strukturierte Ergänzung für den bestehenden Knowledge Base Artikel in Englisch:

Additional Considerations for Swap and Memory Management on UCS Virtualization Hosts

Recommended Server Sizing

Proper dimensioning of server resources is crucial to maintain system performance and prevent swap-related alerts. Although a minimum of 1.5 GB of RAM per virtual machine is often specified, we recommend that no UCS server should be operated with less than 8 GB of RAM, regardless of its specific use case.

https://docs.software-univention.de/ucsschool-scenarios/5.0/de/network-and-hardware.html#infrastructure-and-hardware-requirements-servers-primary-directory-node

For environments with approximately 100 users, a minimum of 8 GB RAM is considered essential. For UCS 5.2, we recommend the following baseline configurations when running as a Domain Controller:

  • Up to 50 users: 8 GB RAM and 8 GB swap for AD DC, DNS, NTP, and Self Service Portal.
  • 50–100 users: 8–16 GB RAM and 8–16 GB swap.
  • Over 100 users: At least 16 GB RAM and 16 GB swap, especially if additional services are deployed.

Note: Inadequate sizing significantly amplifies performance-related issues and can lead to critical system conditions more rapidly.

Performance Optimization and Immediate Measures

To sustainably improve your server’s stability and performance, we recommend the following prioritized actions:

  1. Hardware Upgrade
    Increase your server’s physical memory to at least 8 GB, preferably 16 GB, and adjust swap space proportionally (e.g., 8 GB swap for 8 GB RAM). Adequate physical memory is the foundation for stable operation and helps mitigate memory-related issues.

  2. Comprehensive System Updates
    Ensure that all UCS 5.2 updates are applied, particularly for the Samba and Kerberos (krb5) packages. Many memory leaks and stability issues are resolved in newer versions.

  3. Adjusting vm.swappiness
    To reduce aggressive swap usage and prioritize RAM, lower the swappiness value (e.g., to 10).
    Temporary adjustment:

    sysctl vm.swappiness=10

    Permanent adjustment:
    Add or modify the following line in /etc/sysctl.conf:

    vm.swappiness = 10

    Then apply the change:

    sysctl -p

    For systems where swap is used preemptively (as described in the linked article), a value of 1 may be more appropriate to ensure swap is only used in emergency cases.

Diagnosing Potential Memory Leaks

If persistent or unusual swap usage is suspected to be caused by a memory leak, conduct a systematic investigation:

  1. System-wide Memory Check

    • Identify memory-intensive processes:

      top -o %MEM

      or

      ps aux --sort=-%mem | head

    • Get an overview of physical and swap usage:

      free -m

  2. Monitor Process Memory Growth Over Time
    Observe memory usage trends:

    watch -n 2 'ps -o pid,cmd,%mem,%cpu --sort=-%mem | head'

    Watch for processes whose %MEM steadily increases over hours or days—this may indicate a memory leak.

  3. Check for File Descriptor Leaks
    Monitor open file descriptors of suspicious

lsof -p <PID> | wc -l

Repeat this periodically (every 10–30 seconds). A continuous increase in open descriptors can signal a leak.

Summary

  • Ensure sufficient RAM and swap allocation for your UCS virtualization host according to the expected workload.
  • Adjust vm.swappiness to fine-tune swap behavior, particularly in virtualized environments.
  • Apply regular system updates to mitigate known memory-related issues.
  • Perform memory leak diagnostics if swap usage patterns appear abnormal or persistent.