RE: [PATCH 4/4] dmaengine: idxd: Use DMA API for in-kernel DMA with PASID

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Sent: Thursday, December 9, 2021 1:51 AM
> 
> > > > +	/*
> > > > +	 * Try to enable both in-kernel and user DMA request with PASID.
> > > > +	 * PASID is supported unless both user and kernel PASID are
> > > > +	 * supported. Do not fail probe here in that idxd can still be
> > > > +	 * used w/o PASID or IOMMU.
> > > > +	 */
> > > > +	if (iommu_dev_enable_feature(dev, IOMMU_DEV_FEAT_SVA) ||
> > > > +		idxd_enable_system_pasid(idxd)) {
> > > > +		dev_warn(dev, "Failed to enable PASID\n");
> > > > +	} else {
> > > > +		set_bit(IDXD_FLAG_PASID_ENABLED, &idxd->flags);
> > > >   	}
> > > Huh? How can the driver keep going if PASID isn't supported? I thought
> > > the whole point of this was because the device cannot do DMA without
> > > PASID at all?
> >
> > There are 2 types of WQ supported with the DSA devices. A dedicated WQ
> type
> > and a shared WQ type. The dedicated WQ type can support DMA with and
> without
> > PASID. The shared wq type must have a PASID to operate. The driver can
> > support dedicated WQ only without PASID usage when there is no PASID
> > support.
> 
> Can you add to the cover letter why does the kernel require to use the
> shared WQ?
> 
> Jason

Two reasons:

On native the shared WQ is useful when the kernel wants to offload
some memory operations (e.g. page-zeroing) to DSA. When #CPUs are
more than #WQs, this allows per-cpu lock-less submissions using
ENQCMD(PASID, payload) instruction.

In guest the virtual DSA HW may only contain a WQ in shared mode
(unchangeable by the guest) when the host admin wants to share
the limited WQ resource among many VMs. Then there is no choice
in guest regardless whether it's for user or kernel controlled DMA.

Thanks
Kevin