RE: [PATCH v12 1/1] vfio/nvgpu: Add vfio pci variant module for grace hopper

[Tian, Kevin]

> From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> Sent: Tuesday, October 24, 2023 10:29 PM
> 
> On Tue, 24 Oct 2023 14:03:25 +0000
> Ankit Agrawal <ankita@xxxxxxxxxx> wrote:
> 
> > >> > After looking at Yishai's virtio-vfio-pci driver where BAR0 is emulated
> > >> > as an IO Port BAR, it occurs to me that there's no config space
> > >> > emulation of BAR2 (or BAR3) here.  Doesn't this mean that QEMU
> registers
> > >> > the BAR as 32-bit, non-prefetchable?  ie. VFIOBAR.type & .mem64 are
> > >> > wrong?
> > >>
> > >> Maybe I didn't understand the question, but the PCI config space
> read/write
> > >> would still be handled by vfio_pci_core_read/write() which returns the
> > >> appropriate flags. I have checked that the device BARs are 64b and
> > >> prefetchable in the VM.
> > >
> > > vfio_pci_core_read/write() accesses the physical device, which doesn't
> > > implement BAR2.  Why would an unimplemented BAR2 on the physical
> device
> > > report 64-bit, prefetchable?
> > >
> > > QEMU records VFIOBAR.type and .mem64 from reading the BAR register
> in
> > > vfio_bar_prepare() and passes this type to pci_register_bar() in
> > > vfio_bar_register().  Without an implementation of a config space read
> > > op in the variant driver and with no physical implementation of BAR2 on
> > > the device, I don't see how we get correct values in these fields.
> >
> > I think I see the cause of confusion. There are real PCIe compliant BARs
> > present on the device, just that it isn't being used once the C2C
> > interconnect is active. The BARs are 64b prefetchable. Here it the lspci
> > snippet of the device on the host.
> > # lspci -v -s 9:1:0.0
> > 0009:01:00.0 3D controller: NVIDIA Corporation Device 2342 (rev a1)
> >         Subsystem: NVIDIA Corporation Device 16eb
> >         Physical Slot: 0-5
> >         Flags: bus master, fast devsel, latency 0, IRQ 263, NUMA node 0,
> IOMMU group 19
> >         Memory at 661002000000 (64-bit, prefetchable) [size=16M]
> >         Memory at 662000000000 (64-bit, prefetchable) [size=128G]
> >         Memory at 661000000000 (64-bit, prefetchable) [size=32M]
> >
> > I suppose this answers the BAR sizing question as well?
> 
> Does this BAR2 size match the size we're reporting for the region?  Now
> I'm confused why we need to intercept the BAR2 region info if there's
> physically a real BAR behind it.  Thanks,
> 

same confusion.

probably vfio-pci-core can include a helper for cfg space emulation
on emulated BARs to be used by all variant drivers in that category?

btw intel vgpu also includes an emulation of BAR sizing. same for
future SIOV devices. so there sounds like a general requirement but
of course sharing it between vfio-pci and mdev/siov would be more
difficult.