Problem: IO very slow

Problem

At one customer the write rate to the hard disks is very slow. Is a virtual 32bit UCS system running on a 64bit KVM host.

On the server:

srv:~# dd if=/dev/zero of=/home/foo bs=1M count=100 && rm /home/foo
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0876205 s, 1.2 GB/s

But on the VM it looks like this

root@vm:~# dd if=/dev/zero of=/home/foo bs=1M count=100 && rm /home/foo 100+0 records in
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 67.1148 s, 1.6 MB/s

Analysis

  • correct driver (virtio) was installed
  • customer had some customizations of kernel parameters, but all of them were recommended for VMs (e.g. elevator=noop)
  • no special IO load on the VM or on the host
  • only VM running on the host (so no side effects)
  • Internal timing also fails, as the duration of the above test was also noticeable (0.08s vs. 67s!)
  • a little bit performance loss is normal with VMs, but not to this extent
  • Test installed a 32bit UCS version --> no error found

Solution

customer installed a 64bit kernel in the VM -> running

Not a nice and especially not upgradeable solution, but it worked then. System has to be moved to 64bit at some point anyway.

Mastodon