[PATCH v5 00/21] KVM: s390: enable zPCI for interpretive execution

From: Matthew Rosato
Date: Mon Apr 04 2022 - 17:51:43 EST


Note: in this version, all IOMMU changes have been removed to be pursued
as a follow-on / in conjunction with IOMMUFD. This series proposes to
add only exploitation of the interpretive execution facilities.

Enable interpretive execution of zPCI instructions + adapter interruption
forwarding for s390x KVM vfio-pci. This is done by triggering a routine
when the VFIO group is associated with the KVM guest, transmitting to
firmware a special token (GISA designation) to enable that specific guest
for interpretive execution on that zPCI device. Load/store interpreation
enablement is then controlled by userspace (based upon whether or not a
SHM bit is placed in the virtual function handle). Adapter Event
Notification interpretation is controlled from userspace via a new KVM
ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of
interrupts to guests, we can reduce the frequency of guest SIE exits for
zPCI.

>From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by
default if available in kernel+qemu.

Will reply with a link to the associated QEMU series.

Changelog v4->v5:
- Remove all IOMMU changes - this series will continue to use vfio type1
for mapping; RPCIT enhancements will be pursued as a follow-on in
coordination with the IOMMUFD project
- Remove most of the ops from the proposed KVM ioctl -- combine the notion
of 'start' and 'enable interpretation' into a single operation that
occurs in response to assigning a KVM association to the VFIO group
- a new kvm ioctl is still being used to register/unregister for adapter
event notification forwarding. But teach it to use pin_user_pages
instead of gfn_to_page and perform accounting.
- Because we now attempt to enable interpretation for all devices (even
if the virtual PCI device has a SHM bit on) avoid setting the GISA for
the device and the ECB bits for the guest on hardware without SHM.
- found a bug with "s390/pci: stash associated GISA designation", namely
that CLP SET PCI FN (disable) expects the gisa designation to always be
0. Now that we set the gisa earlier, vfio-pci can trip this when
triggering restore for ISM.
- Add a single routine to determine if interpretation support is usable
on this host, to be used in multiple spots.
- eliminated arch/s390/include/asm/kvm_pci.h, moving most elements into
arch/s390/kvm/pci.h with one exception to asm/kvm_host.h

Matthew Rosato (21):
s390/sclp: detect the zPCI load/store interpretation facility
s390/sclp: detect the AISII facility
s390/sclp: detect the AENI facility
s390/sclp: detect the AISI facility
s390/airq: pass more TPI info to airq handlers
s390/airq: allow for airq structure that uses an input vector
s390/pci: externalize the SIC operation controls and routine
s390/pci: stash associated GISA designation
s390/pci: stash dtsm and maxstbl
KVM: s390: pci: add basic kvm_zdev structure
KVM: s390: pci: do initial setup for AEN interpretation
KVM: s390: pci: enable host forwarding of Adapter Event Notifications
KVM: s390: mechanism to enable guest zPCI Interpretation
KVM: s390: pci: provide routines for enabling/disabling interrupt
forwarding
KVM: s390: pci: add routines to start/stop interpretive execution
KVM: vfio: add s390x hook to register KVM guest designation
vfio-pci/zdev: add function handle to clp base capability
vfio-pci/zdev: different maxstbl for interpreted devices
KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
KVM: s390: introduce CPU feature for zPCI Interpretation
MAINTAINERS: additional files related kvm s390 pci passthrough

Documentation/virt/kvm/api.rst | 45 +++
MAINTAINERS | 1 +
arch/s390/include/asm/airq.h | 7 +-
arch/s390/include/asm/kvm_host.h | 16 +
arch/s390/include/asm/pci.h | 10 +
arch/s390/include/asm/pci_clp.h | 9 +-
arch/s390/include/asm/pci_insn.h | 29 +-
arch/s390/include/asm/sclp.h | 4 +
arch/s390/include/asm/tpi.h | 13 +
arch/s390/include/uapi/asm/kvm.h | 1 +
arch/s390/kvm/Makefile | 1 +
arch/s390/kvm/interrupt.c | 95 ++++-
arch/s390/kvm/kvm-s390.c | 81 +++-
arch/s390/kvm/kvm-s390.h | 10 +
arch/s390/kvm/pci.c | 666 +++++++++++++++++++++++++++++++
arch/s390/kvm/pci.h | 86 ++++
arch/s390/pci/pci.c | 15 +
arch/s390/pci/pci_clp.c | 7 +
arch/s390/pci/pci_insn.c | 4 +-
arch/s390/pci/pci_irq.c | 48 ++-
drivers/s390/char/sclp_early.c | 4 +
drivers/s390/cio/airq.c | 12 +-
drivers/s390/cio/qdio_thinint.c | 6 +-
drivers/s390/crypto/ap_bus.c | 9 +-
drivers/s390/virtio/virtio_ccw.c | 6 +-
drivers/vfio/pci/vfio_pci_zdev.c | 11 +-
include/uapi/linux/kvm.h | 31 ++
include/uapi/linux/vfio_zdev.h | 7 +
virt/kvm/vfio.c | 35 +-
29 files changed, 1216 insertions(+), 53 deletions(-)
create mode 100644 arch/s390/kvm/pci.c
create mode 100644 arch/s390/kvm/pci.h

--
2.27.0