UVMM Live Migration over NFS fails due to different numeric IDs

pmhahn · September 24, 2019, 3:10pm

Problem description

While a virtual machine (VM) is running its image files are owned by user libvirt-qemu and group libvirt-qemu to prevent other processes from accessing those files concurrently. Until UCS-4.2 the numeric user ID (UID) and group ID (GID) were allocated dynamically during Debian package installation. This leads to the named user having different numerical IDs, which breaks Live migration over Network File System: NFS works only with the numeric IDs. So while a VM is running on one host, a second host cannot access the same file as it has a different numeric mapping.
To solve this Debian switched to a static mapping after allocating UID 64055 and GID 64055 permanently to that user / group. This was fixed in Debian Jessie 8, which was imported into UCS-4.2. Since then the static IDs are used.

If hosts were setup bevor UCS-4.2 they still use their dynamic mapping. Migration between them works by accident if (and only if) each of them allocated the same numeric IDs to the user / group. If a host has a different mapping, VMs cannot be migrated to or from that host. This gets worse as soon as new hosts are setup with UCS-4.2 or later, as they definitely will use the new static mapping, which is different.

Solution

To solve this all hosts should be converted to the static allocation: On each host the numeric IDs must be switched to 64055.

Check that the UID 64055 and GID 64055 are unused in your domain:
```
$ getent passwd 64055
libvirt-qemu:x:64055:113:Libvirt Qemu,,,:/var/lib/libvirt:/bin/false
$ getent group 64055
libvirt-qemu:x:64055:libvirt-qemu
```
Also verify that no user / group exists in LDAP using those IDs:
```
$ udm users/user list --filter uidNumber=64055
uidNumber=64055
$ udm groups/group list --filter gidNumber=64055
gidNumber=64055
```
If any (other) user or group is returned, the IDs are already taken otherwise. In that case you either have to change that user / group to some other ID or you have to pick a free UID / GID instead of 64055, whichever is easier for you.
To get a list of affected hosts run the following command on all your hosts:
```
$ LANG=C id libvirt-qemu
uid=109(libvirt-qemu) gid=106(kvm) groups=106(kvm),24(cdrom),114(libvirt-qemu)
    ^^^                                                      ^^^
# THIS IS FROM A BROKEN SYSTEM
```
Please note that the other groups kvm and cdrom might use other numeric group IDs then shown here, as they are still assigned dynamically per host. This is okay as those groups are only used to regulate access to local devices below /dev/, which are per-host anyway.
1. Hosts with uid=64055 and groups=64055 already use the new static UID/GID assignment and are okay - nothing needs fixing there.
2. Otherwise they were setup with the old schema using dynamic UID/GID, which must be fixed.
For each affected host first check that there is no running process currently using those IDs. The following command must show no output:
```
$ pgrep -l -U libvirt-qemu
19258 qemu-system-x86
```
If there are still process, see section Details below on how to stop the VMs.

Change the IDs:

$ usermod -u 64055 libvirt-qemu
$ groupmod -g 64055 libvirt-qemu

Afterwards verify the IDs using the id command again:

$ LANG=C id libvirt-qemu
uid=64055(libvirt-qemu) gid=106(kvm) groups=106(kvm),24(cdrom),64055(libvirt-qemu)
    ^^^^^                                                      ^^^^^
# A FIXED SYSTEM SHOULD LOOK LIKE THIS

Restart the libvirt daemon as it caches the numeric ID mapping internally:
```
$ service libvirtd restart
```

Proceed with the next hosts until all are converted to the static schema.

Details

As this change can only be performed when there it no running VM, you first have to shutdown all your VMs or must migrate all VMs to other hosts.

Using shutdown or suspend-to-disk

If your service level agreement (SLA) permits a down-time, shutting down the VMs is often easier. Either trigger the procedure from within your guest operating system, or use virsh shutdown $VM or the equivalent UVMM action to initiate an ACPI Power Button Event from the host system itself.
As an alternative you can also use virsh managedsave $VM or the equivalent UVMM action to initiate a suspend-to-disk, which allows you to save the running state of the VM.

Then perform the steps listed above on the host before starting the VMs again either using virsh start $VM or UVMM.

Using migration

We have a recursive problem here: To fix migration we need migration itself.
The trick here is that we must give multiple libvirt-qemu users (with different numeric UIDs), which are in different groups libvirt-qemu (again with different numeric GIDs) access to the same file. The idea therefore is to make the image files world-writable - at least for the duration of the migration:

On the source host:
1.1 Use virsh domblklist --details --domain $VM to get the list of images files. Only change images with Type=file and Device=disk as ISO image files should be world-readable and are accessed read-only anyway.
1.2 Use chmod 666 /var/lib/libvirt/images/$IMAGE to make the file world-writable.
Migrate the VM using virsh migrate --live -persistent --undefinesource --verbose --domain $VM --desturi qemu://$HOSTNAME.$DOMAINNAME/system or the equivalent UVMM action. (The virsh variant does not work when the VM has snapshots - use UVMM then).
On the target host:
3.1 Use chmod 640 /var/lib/libvirt/images/$IMAGE to reset the file permissions to accessible to only the user.

The following command gives an example on how the changing of file permissions can be simplified:

$ virsh domblklist --details "$VM"|awk '$1=="file"&&$2=="disk"{print $4}'|xargs -r chmod -v 0640

Repeat those steps until no VM is running on the source host any more.
Then perform the steps from Solution above to change the UID and GID of libvirt-qemu using usermod and groupmod.

Variant using Access Control Lists (ACL)

Making your files world-writable is a security issue as anybody having access to your host systems can modify the file. Therefore you should limit the time while the images are in that state to a minimum. Think about restricting access to your host systems while you perform the steps above.
As an alternative you can use ACLs to give individual numeric UIDs access to your image files:

$ setfacl -m u:$UID:rw- /var/lib/libvirt/images/...

Replace $UID by the dynamic UIDs found above in step 1 and repeat that for each different UID and file (you can specify multiple -m u:$UID:rw- on the command line as well as multiple files using wildcards).

The downside is that not all NFS servers support this out-of-the-box or not at all.

After converting all hosts to the static schema you then should remove those extra ACLs again with

$ setfacl --remove-all /var/lib/libvirt/images/*

Cleanup

On shutdown of a VM libvirtd by default gives back the image files to the original owner. Due to crashes (or other issues) it might happen that after changing the numeric IDs on all hosts, you have files still using the old IDs, which are then no longer associated with a name.
You can use find /var/lib/libvirt/ -nouser -o -nogroup (or a similar command) to search for such files. Then you can - for example - use chown libvirt-qemu:libvirt-qemu /var/lib/libvirt/... to change their ownership to the new UID:GID.

References

Bug #45886: libvirt-qemu local uid gid number depends on installation order

Christian_Voelker · September 25, 2019, 1:45pm