Re: [PATCH v3 1/1] PCI: Add translated request only flag for pci_enable_pasid()

[Bjorn Helgaas]

On Mon, Jan 30, 2023 at 02:47:32PM -0400, Jason Gunthorpe wrote:
> On Mon, Jan 30, 2023 at 12:38:10PM -0600, Bjorn Helgaas wrote:
> 
> > Sorry, I'm still confused.  PCI_PASID_XLATED_REQ_ONLY is a
> > device-specific property, and you want to opt-in AMD graphics devices.
> > Where's the AMD graphics-specific change?  The current patch does
> > this:
> > 
> >   pdev_pri_ats_enable
> >     pci_enable_pasid(pdev, 0, PCI_PASID_XLATED_REQ_ONLY)
> > 
> > which looks like it does it for *all* devices below an AMD IOMMU,
> > without any device or driver input.
> 
> AMD GPU has a private interface to AMD IOMMU to support PASID support
> that only it uses.

What is it that makes this a private interface?  It seems like we will
end up enabling PASID on any device below an AMD v2 IOMMU.

I do see the DRM amdgpu_amdkfd_device_init() path that ends up at
domain_enable_v2().  Maybe that's it?

But amd_iommu_domain_alloc() also leads to domain_enable_v2(), and
that's pretty generic, so it looks like we set PD_IOMMUV2_MASK
whenever the IOMMU supports it.

And it seems like we call pci_enable_pasid() for all endpoints below
a v2 IOMMU.  Of course, if the endpoint doesn't have a PASID cap,
pci_enable_pasid() fails (and pdev_pri_ats_enable() probably splats
useless warnings when we try to disable PRI and PASID which haven't
been enabled).

I guess I'm trying to convince myself that no harm in enabling PASID
for any device below an AMD v2 IOMMU.  But I don't think a device is
*required* to use translated addresses with PASID, and if it uses
untranslated addresses with PASID, don't we need ACS to avoid
misrouting?

Bjorn