RE: [PATCH 1/4] ioasid: Reserve a global PASID for in-kernel DMA

[Tian, Kevin]

> From: Jacob Pan <jacob.jun.pan@xxxxxxxxxxxxxxx>
> Sent: Saturday, December 11, 2021 2:06 AM
> 
> Hi Jason,
> 
> On Fri, 10 Dec 2021 08:31:09 -0400, Jason Gunthorpe <jgg@xxxxxxxxxx>
> wrote:
> 
> > On Fri, Dec 10, 2021 at 09:06:24AM +0000, Jean-Philippe Brucker wrote:
> > > On Thu, Dec 09, 2021 at 10:14:04AM -0800, Jacob Pan wrote:
> > > > > This looks like we're just one step away from device drivers needing
> > > > > multiple PASIDs for kernel DMA so I'm trying to figure out how to
> > > > > evolve the API towards that. It's probably as simple as keeping a
> > > > > kernel IOASID set at first, but then we'll probably want to
> > > > > optimize by having multiple overlapping sets for each device driver
> > > > > (all separate from the SVA set).
> > > > Sounds reasonable to start with a kernel set for in-kernel DMA once
> > > > we need multiple ones. But I am not sure what *overlapping* sets
> mean
> > > > here, could you explain?
> > >
> > > Given that each device uses a separate PASID table, we could allocate
> > > the same set of PASID values for different device drivers. We just need
> > > to make sure that those values are different from PASIDs allocated for
> > > user SVA.
> >
> > Why does user SVA need global values anyhow?
> >
> Currently, we have mm.pasid for user SVA. mm is global. We could have per
> device PASID for dedicated devices (not shared across mm's), but that would
> make things a lot more complex. I am thinking multiple PASIDs per mm is
> needed, right?
> 
> For VT-d, the shared workqueue (SWQ) requires global PASIDs in that we
> cannot have two processes use the same PASID to submit work on a
> workqueue
> shared by the two processes. Each process's PASID must be unique to the
> SWQ's PASID table.
> 

Uniqueness is not the main argument of using global PASIDs for
SWQ, since it can be defined either in per-RID or in global PASID
space. No SVA architecture can allow two processes to use the
same PASID to submit work unless they share mm! 😊

IMO the real reason is that SWQ for user SVA must be accessed 
via ENQCMD instruction which fetches the PASID from a CPU MSR 
(managed as part of XSAVE state). This assumes a single PASID 
representing the address space for the user task regardless 
how many devices are attached to this task. Otherwise the kernel
doesn't know when to set which PASID if multiple PASIDs per mm.

Then this single PASID has to be global cross all attached devices
to ensure uniqueness.

Of course SWQ is only one configuration for user SVA. In theory
multiple per-device PASIDs are still feasible for non-SWQ cases.
But given mm is global and the SWQ restriction, it's not worthwhile
of adding that complexity...

Thanks
Kevin