Re: [PATCH v1 07/14] vfio: Add a device notifier interface

[Jason Gunthorpe]

On Mon, Mar 08, 2021 at 02:48:30PM -0700, Alex Williamson wrote:
> Using a vfio device, a notifier block can be registered to receive
> select device events.  Notifiers can only be registered for contained
> devices, ie. they are available through a user context.  Registration
> of a notifier increments the reference to that container context
> therefore notifiers must minimally respond to the release event by
> asynchronously removing notifiers.
> 
> Signed-off-by: Alex Williamson <alex.williamson@xxxxxxxxxx>
>  drivers/vfio/Kconfig |    1 +
>  drivers/vfio/vfio.c  |   35 +++++++++++++++++++++++++++++++++++
>  include/linux/vfio.h |    9 +++++++++
>  3 files changed, 45 insertions(+)
> 
> diff --git a/drivers/vfio/Kconfig b/drivers/vfio/Kconfig
> index 90c0525b1e0c..9a67675c9b6c 100644
> +++ b/drivers/vfio/Kconfig
> @@ -23,6 +23,7 @@ menuconfig VFIO
>  	tristate "VFIO Non-Privileged userspace driver framework"
>  	select IOMMU_API
>  	select VFIO_IOMMU_TYPE1 if (X86 || S390 || ARM || ARM64)
> +	select SRCU
>  	help
>  	  VFIO provides a framework for secure userspace device drivers.
>  	  See Documentation/driver-api/vfio.rst for more details.
> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
> index c47895539a1a..7f6d00e54e83 100644
> +++ b/drivers/vfio/vfio.c
> @@ -105,6 +105,7 @@ struct vfio_device {
>  	struct list_head		group_next;
>  	void				*device_data;
>  	struct inode			*inode;
> +	struct srcu_notifier_head	notifier;
>  };
>  
>  #ifdef CONFIG_VFIO_NOIOMMU
> @@ -601,6 +602,7 @@ struct vfio_device *vfio_group_create_device(struct vfio_group *group,
>  	device->ops = ops;
>  	device->device_data = device_data;
>  	dev_set_drvdata(dev, device);
> +	srcu_init_notifier_head(&device->notifier);
>  
>  	/* No need to get group_lock, caller has group reference */
>  	vfio_group_get(group);
> @@ -1785,6 +1787,39 @@ static const struct file_operations vfio_device_fops = {
>  	.mmap		= vfio_device_fops_mmap,
>  };
>  
> +int vfio_device_register_notifier(struct vfio_device *device,
> +				  struct notifier_block *nb)
> +{
> +	int ret;
> +
> +	/* Container ref persists until unregister on success */
> +	ret =  vfio_group_add_container_user(device->group);

I'm having trouble guessing why we need to refcount the group to add a
notifier to the device's notifier chain? 

I suppose it actually has to do with the MMIO mapping? But I don't
know what the relation is between MMIO mappings in the IOMMU and the
container? This could deserve a comment?

> +void vfio_device_unregister_notifier(struct vfio_device *device,
> +				    struct notifier_block *nb)
> +{
> +	if (!srcu_notifier_chain_unregister(&device->notifier, nb))
> +		vfio_group_try_dissolve_container(device->group);
> +}
> +EXPORT_SYMBOL_GPL(vfio_device_unregister_notifier);

Is the SRCU still needed with the new locking? With a cursory look I
only noticed this called under the reflck->lock ?

Jason