[PATCH 0/9] RFC: NVME VFIO mediated device

[Maxim Levitsky]

Oops, I placed the subject in the wrong place.

Best regards,
	Maxim Levitsky

On Tue, 2019-03-19 at 16:41 +0200, Maxim Levitsky wrote:
> Date: Tue, 19 Mar 2019 14:45:45 +0200
> Subject: [PATCH 0/9] RFC: NVME VFIO mediated device
> 
> Hi everyone!
> 
> In this patch series, I would like to introduce my take on the problem of
> doing 
> as fast as possible virtualization of storage with emphasis on low latency.
> 
> In this patch series I implemented a kernel vfio based, mediated device that 
> allows the user to pass through a partition and/or whole namespace to a guest.
> 
> The idea behind this driver is based on paper you can find at
> https://www.usenix.org/conference/atc18/presentation/peng,
> 
> Although note that I stared the development prior to reading this paper, 
> independently.
> 
> In addition to that implementation is not based on code used in the paper as 
> I wasn't being able at that time to make the source available to me.
> 
> ***Key points about the implementation:***
> 
> * Polling kernel thread is used. The polling is stopped after a 
> predefined timeout (1/2 sec by default).
> Support for all interrupt driven mode is planned, and it shows promising
> results.
> 
> * Guest sees a standard NVME device - this allows to run guest with 
> unmodified drivers, for example windows guests.
> 
> * The NVMe device is shared between host and guest.
> That means that even a single namespace can be split between host 
> and guest based on different partitions.
> 
> * Simple configuration
> 
> *** Performance ***
> 
> Performance was tested on Intel DC P3700, With Xeon E5-2620 v2 
> and both latency and throughput is very similar to SPDK.
> 
> Soon I will test this on a better server and nvme device and provide
> more formal performance numbers.
> 
> Latency numbers:
> ~80ms - spdk with fio plugin on the host.
> ~84ms - nvme driver on the host
> ~87ms - mdev-nvme + nvme driver in the guest
> 
> Throughput was following similar pattern as well.
> 
> * Configuration example
>   $ modprobe nvme mdev_queues=4
>   $ modprobe nvme-mdev
> 
>   $ UUID=$(uuidgen)
>   $ DEVICE='device pci address'
>   $ echo $UUID > /sys/bus/pci/devices/$DEVICE/mdev_supported_types/nvme-
> 2Q_V1/create
>   $ echo n1p3 > /sys/bus/mdev/devices/$UUID/namespaces/add_namespace #attach
> host namespace 1 parition 3
>   $ echo 11 > /sys/bus/mdev/devices/$UUID/settings/iothread_cpu #pin the io
> thread to cpu 11
> 
>   Afterward boot qemu with
>   -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/$UUID
>   
>   Zero configuration on the guest.
>   
> *** FAQ ***
> 
> * Why to make this in the kernel? Why this is better that SPDK
> 
>   -> Reuse the existing nvme kernel driver in the host. No new drivers in the
> guest.
>   
>   -> Share the NVMe device between host and guest. 
>      Even in fully virtualized configurations,
>      some partitions of nvme device could be used by guests as block devices 
>      while others passed through with nvme-mdev to achieve balance between
>      all features of full IO stack emulation and performance.
>   
>   -> NVME-MDEV is a bit faster due to the fact that in-kernel driver 
>      can send interrupts to the guest directly without a context 
>      switch that can be expensive due to meltdown mitigation.
> 
>   -> Is able to utilize interrupts to get reasonable performance. 
>      This is only implemented
>      as a proof of concept and not included in the patches, 
>      but interrupt driven mode shows reasonable performance
>      
>   -> This is a framework that later can be used to support NVMe devices 
>      with more of the IO virtualization built-in 
>      (IOMMU with PASID support coupled with device that supports it)
> 
> * Why to attach directly to nvme-pci driver and not use block layer IO
>   -> The direct attachment allows for better performance, but I will
>      check the possibility of using block IO, especially for fabrics drivers.
>   
> *** Implementation notes ***
> 
> *  All guest memory is mapped into the physical nvme device 
>    but not 1:1 as vfio-pci would do this.
>    This allows very efficient DMA.
>    To support this, patch 2 adds ability for a mdev device to listen on 
>    guest's memory map events. 
>    Any such memory is immediately pinned and then DMA mapped.
>    (Support for fabric drivers where this is not possible exits too,
>     in which case the fabric driver will do its own DMA mapping)
> 
> *  nvme core driver is modified to announce the appearance 
>    and disappearance of nvme controllers and namespaces,
>    to which the nvme-mdev driver is subscribed.
>  
> *  nvme-pci driver is modified to expose raw interface of attaching to 
>    and sending/polling the IO queues.
>    This allows the mdev driver very efficiently to submit/poll for the IO.
>    By default one host queue is used per each mediated device.
>    (support for other fabric based host drivers is planned)
> 
> * The nvme-mdev doesn't assume presence of KVM, thus any VFIO user, including
>   SPDK, a qemu running with tccg, ... can use this virtual device.
> 
> *** Testing ***
> 
> The device was tested with stock QEMU 3.0 on the host,
> with host was using 5.0 kernel with nvme-mdev added and the following
> hardware:
>  * QEMU nvme virtual device (with nested guest)
>  * Intel DC P3700 on Xeon E5-2620 v2 server
>  * Samsung SM981 (in a Thunderbolt enclosure, with my laptop)
>  * Lenovo NVME device found in my laptop
> 
> The guest was tested with kernel 4.16, 4.18, 4.20 and
> the same custom complied kernel 5.0
> Windows 10 guest was tested too with both Microsoft's inbox driver and
> open source community NVME driver
> (https://lists.openfabrics.org/pipermail/nvmewin/2016-December/001420.html)
> 
> Testing was mostly done on x86_64, but 32 bit host/guest combination
> was lightly tested too.
> 
> In addition to that, the virtual device was tested with nested guest,
> by passing the virtual device to it,
> using pci passthrough, qemu userspace nvme driver, and spdk
> 
> 
> PS: I used to contribute to the kernel as a hobby using the
>     maximlevitsky@xxxxxxxxx address
> 
> Maxim Levitsky (9):
>   vfio/mdev: add .request callback
>   nvme/core: add some more values from the spec
>   nvme/core: add NVME_CTRL_SUSPENDED controller state
>   nvme/pci: use the NVME_CTRL_SUSPENDED state
>   nvme/pci: add known admin effects to augument admin effects log page
>   nvme/pci: init shadow doorbell after each reset
>   nvme/core: add mdev interfaces
>   nvme/core: add nvme-mdev core driver
>   nvme/pci: implement the mdev external queue allocation interface
> 
>  MAINTAINERS                   |   5 +
>  drivers/nvme/Kconfig          |   1 +
>  drivers/nvme/Makefile         |   1 +
>  drivers/nvme/host/core.c      | 149 +++++-
>  drivers/nvme/host/nvme.h      |  55 ++-
>  drivers/nvme/host/pci.c       | 385 ++++++++++++++-
>  drivers/nvme/mdev/Kconfig     |  16 +
>  drivers/nvme/mdev/Makefile    |   5 +
>  drivers/nvme/mdev/adm.c       | 873 ++++++++++++++++++++++++++++++++++
>  drivers/nvme/mdev/events.c    | 142 ++++++
>  drivers/nvme/mdev/host.c      | 491 +++++++++++++++++++
>  drivers/nvme/mdev/instance.c  | 802 +++++++++++++++++++++++++++++++
>  drivers/nvme/mdev/io.c        | 563 ++++++++++++++++++++++
>  drivers/nvme/mdev/irq.c       | 264 ++++++++++
>  drivers/nvme/mdev/mdev.h      |  56 +++
>  drivers/nvme/mdev/mmio.c      | 591 +++++++++++++++++++++++
>  drivers/nvme/mdev/pci.c       | 247 ++++++++++
>  drivers/nvme/mdev/priv.h      | 700 +++++++++++++++++++++++++++
>  drivers/nvme/mdev/udata.c     | 390 +++++++++++++++
>  drivers/nvme/mdev/vcq.c       | 207 ++++++++
>  drivers/nvme/mdev/vctrl.c     | 514 ++++++++++++++++++++
>  drivers/nvme/mdev/viommu.c    | 322 +++++++++++++
>  drivers/nvme/mdev/vns.c       | 356 ++++++++++++++
>  drivers/nvme/mdev/vsq.c       | 178 +++++++
>  drivers/vfio/mdev/vfio_mdev.c |  11 +
>  include/linux/mdev.h          |   4 +
>  include/linux/nvme.h          |  88 +++-
>  27 files changed, 7375 insertions(+), 41 deletions(-)
>  create mode 100644 drivers/nvme/mdev/Kconfig
>  create mode 100644 drivers/nvme/mdev/Makefile
>  create mode 100644 drivers/nvme/mdev/adm.c
>  create mode 100644 drivers/nvme/mdev/events.c
>  create mode 100644 drivers/nvme/mdev/host.c
>  create mode 100644 drivers/nvme/mdev/instance.c
>  create mode 100644 drivers/nvme/mdev/io.c
>  create mode 100644 drivers/nvme/mdev/irq.c
>  create mode 100644 drivers/nvme/mdev/mdev.h
>  create mode 100644 drivers/nvme/mdev/mmio.c
>  create mode 100644 drivers/nvme/mdev/pci.c
>  create mode 100644 drivers/nvme/mdev/priv.h
>  create mode 100644 drivers/nvme/mdev/udata.c
>  create mode 100644 drivers/nvme/mdev/vcq.c
>  create mode 100644 drivers/nvme/mdev/vctrl.c
>  create mode 100644 drivers/nvme/mdev/viommu.c
>  create mode 100644 drivers/nvme/mdev/vns.c
>  create mode 100644 drivers/nvme/mdev/vsq.c
>