[PATCH 0/1] vfio: fix KVM reference leak on QEMU CPR

From: Andrey Drobyshev

Date: Fri Nov 20 2026 - 10:48:38 EST


When a VM has a VFIO device and QEMU does a CPR (CheckPoint and Restart)
live update, QEMU creates a new VM and keeps the device open. The
device holds a reference to the VM it was opened with, so the old VM is
not destroyed. It stays until the device is closed, which usually means
for as long as the guest runs.

It is one stale VM, not one per update. It is also visible: after
cpr-exec the kernel logs "kvm: debugfs: duplicate directory <pid>-<fd>",
and the new VM gets no debugfs directory.

This patch makes the device take that reference only if its driver uses
it.

Tested on v7.4-rc3 with QEMU 11.1.50, cpr-exec and the legacy VFIO
container, in a nested setup: the L1 VM has a spare virtio-net device
behind a virtual IOMMU and passes it to an L2 VM with vfio-pci. The
script below runs cpr-exec and counts the kvm_create_vm() and
kvm_destroy_vm() calls:

$ cat kvmref.sh

> #!/bin/bash
>
> QEMU=/path/to/build/qemu-system-x86_64
> DEV=0000:08:00.0
> MON="/var/tmp/mon.sock"
> STATE="/var/tmp/vm.state"
> TRACING=/sys/kernel/tracing
>
> hmp() {
> echo "$1" | socat - "UNIX-CONNECT:${MON}" | tr -d '\r' \
> | grep -v -e '^QEMU ' -e '^(qemu)'
> }
>
> bind_vfio() {
> dev="/sys/bus/pci/devices/${DEV}"
> modprobe vfio-pci
> echo vfio-pci > "${dev}/driver_override"
> [[ -e "${dev}/driver" ]] && echo "${DEV}" > "${dev}/driver/unbind"
> echo "${DEV}" > /sys/bus/pci/drivers_probe
> }
>
> trace_off() {
> echo nop > "${TRACING}/current_tracer"
> echo > "${TRACING}/set_ftrace_filter"
> }
>
> trace_on() {
> trace_off
> echo 'kvm_create_vm kvm_destroy_vm' > "${TRACING}/set_ftrace_filter"
> echo function > "${TRACING}/current_tracer"
> echo > "${TRACING}/trace"
> }
>
> # How many VMs were created and destroyed so far
> report() {
> created="$(grep -c 'kvm_create_vm' "${TRACING}/trace")"
> destroyed="$(grep -c 'kvm_destroy_vm' "${TRACING}/trace")"
> printf '%-22s created %s, destroyed %s, alive %s\n' \
> "$1:" "${created}" "${destroyed}" "$((created - destroyed))"
> }
>
> main() {
> mode="${1:-vfio}"
> cycles="${2:-1}"
>
> local -a args=(
> "${QEMU}" -accel kvm -display none
> -machine q35,aux-ram-share=on,memory-backend=ram
> -object memory-backend-memfd,id=ram,size=512M,share=on
> -monitor "unix:${MON},server=on,wait=off"
> )
>
> if [[ "${mode}" == vfio ]]; then
> bind_vfio
> args+=(-device "vfio-pci,host=${DEV}")
> fi
>
> trace_on
> "${args[@]}" &
> pid=$!
> sleep 2
> report 'QEMU started'
>
> for ((i = 1; i <= cycles; i++)); do
> hmp 'migrate_set_parameter mode cpr-exec'
> hmp "migrate_set_parameter cpr-exec-command ${args[*]} -incoming file:${STATE}"
> hmp "migrate file:${STATE}"
> sleep 3
> report "after cpr-exec ${i}"
> done
> echo "pid ${pid} is still $(readlink "/proc/${pid}/exe"), $(hmp 'info status')"
>
> hmp quit
> wait "${pid}"
> sleep 1
> report 'QEMU exited'
> trace_off
> }
>
> main "$@"

Control run:

$ ./kvmref.sh novfio 3
QEMU started: created 1, destroyed 0, alive 1
after cpr-exec 1: created 2, destroyed 1, alive 1
after cpr-exec 2: created 3, destroyed 2, alive 1
after cpr-exec 3: created 4, destroyed 3, alive 1
pid 6057 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited: created 4, destroyed 4, alive 0

Before patch:

$ ./kvmref.sh vfio 3
QEMU started: created 1, destroyed 0, alive 1
after cpr-exec 1: created 2, destroyed 0, alive 2
after cpr-exec 2: created 3, destroyed 1, alive 2
after cpr-exec 3: created 4, destroyed 2, alive 2
pid 7607 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited: created 4, destroyed 4, alive 0

After this patch:

$ ./kvmref.sh vfio 3
QEMU started: created 1, destroyed 0, alive 1
after cpr-exec 1: created 2, destroyed 1, alive 1
after cpr-exec 2: created 3, destroyed 2, alive 1
after cpr-exec 3: created 4, destroyed 3, alive 1
pid 6164 is still /path/to/build/qemu-system-x86_64, VM status: running
QEMU exited: created 4, destroyed 4, alive 0

Andrey Drobyshev (1):
vfio: Take the KVM reference only for drivers that read device->kvm

drivers/gpu/drm/i915/gvt/kvmgt.c | 1 +
drivers/s390/crypto/vfio_ap_ops.c | 1 +
drivers/vfio/device_cdev.c | 3 ++-
drivers/vfio/group.c | 3 ++-
drivers/vfio/pci/vfio_pci_core.c | 1 +
drivers/vfio/vfio_main.c | 6 ++++--
include/linux/vfio.h | 2 ++
7 files changed, 13 insertions(+), 4 deletions(-)

--
2.47.1