Problem: 32bit UCS Very Slow

Knowledge Base Community
, , , ,
#1

Problem

A UCS server acts very sluggish. Even the execution of top or ps axuf takes a noticeable amount of time. Some services log a couple of errors.

Environment

The server in question is a 32bit UCS with PAE kernel and more than 4GB of memory.

Solution

  1. Upgrade
    Replace the server with a 64bit version which does not have the limitations of a 32bit system.
  2. Optimize settings
    Reset the values to the default ones to improve reaction time by adding the following lines to /etc/sysctl.conf and reboot the server.
vm.dirty_background_ratio=10
vm.dirty_ratio=60
vm.dirty_expire_centisecs=3000
vm.dirty_writeback_centisecs=500
vm.dirtytime_expire_seconds=43200

Root Cause

Part 1

This is a well-known phenomenon of the Linux kernel: 32-bit systems can only make IO within the first 4 GiB; for IO above this, these pages must be recopied into a range <4 GiB, which puts an additional load on the system.

Part 2

The Linux kernel calculates different limits based on the RAM size from when it becomes active and writes “dirty” memory pages back to the hard disk.

/proc/sys/vm/dirty_background_ratio:40

40% of the 8 GiB = 3.2 GiB may be dirty without triggering a write back in the background.

/proc/sys/vm/dirty_ratio:60

60% of the 8 GiB = 4.8 GiB may be dirty before the Linux kernel forces a synchronous write: If a process makes new memory pages dirty by writing, this process must wait for the (/all?) data to be written. This blocks the process and leads to waiting times (D-State)

/proc/sys/vm/dirty_expire_centisecs:3000

Dirty memory pages may remain in memory for 30 seconds before they are written back. The Linux kernel speculates that this will bundle multiple accesses to one page and save it work, because it does not have to write back the data immediately after each access.
A crash would mean that the changes of the last 30s would be lost.

/proc/sys/vm/dirty_writeback_centisecs:60000

The time interval in which the Linux kernel looks to see if there is something to do.

/proc/sys/vm/dirtytime_expire_seconds:43200

The time interval until dirty inodes have to be written back.

Here especially /proc/sys/vm/dirty_writeback_centisecs:60000 might be the reason for this behaviour. It means, the kernel check only every 10minutes if dirty pages have to be written back to disk. This value is not the default value for UCS and has been set somehow.

Mastodon