Re: [patch RFC 00/38] x86, PCI, XEN, genirq ...: Prepare for device MSI

From: Jürgen Groß
Date: Sat Aug 22 2020 - 10:19:22 EST


On 21.08.20 02:24, Thomas Gleixner wrote:
First of all, sorry for the horrible long Cc list, which was
unfortunately unavoidable as this touches the world and some more.

This patch series aims to provide a base to support device MSI (non
PCI based) in a halfways architecture independent way.

It's a mixed bag of bug fixes, cleanups and general improvements which
are worthwhile independent of the device MSI stuff. Unfortunately this
also comes with an evil abuse of the irqdomain system to coerce XEN on
x86 into compliance without rewriting XEN from scratch.

As discussed in length in this mail thread:

https://lore.kernel.org/r/87h7tcgbs2.fsf@xxxxxxxxxxxxxxxxxxxxxxx

the initial attempt of piggypacking device MSI support on platform MSI
is doomed for various reasons, but creating independent interrupt
domains for these upcoming magic PCI subdevices which are not PCI, but
might be exposed as PCI devices is not as trivial as it seems.

The initially suggested and evaluated approach of extending platform
MSI turned out to be the completely wrong direction and in fact
platform MSI should be rewritten on top of device MSI or completely
replaced by it.

One of the main issues is that x86 does not support the concept of irq
domains associations stored in device::msi_domain and still relies on
the arch_*_msi_irqs() fallback implementations which has it's own set
of problems as outlined in

https://lore.kernel.org/r/87bljg7u4f.fsf@xxxxxxxxxxxxxxxxxxxxxxx/

in the very same thread.

The main obstacle of storing that pointer is XEN which has it's own
historical notiion of handling PCI MSI interupts.

This series tries to address these issues in several steps:

1) Accidental bug fixes
iommu/amd: Prevent NULL pointer dereference

2) Janitoring
x86/init: Remove unused init ops

3) Simplification of the x86 specific interrupt allocation mechanism

x86/irq: Rename X86_IRQ_ALLOC_TYPE_MSI* to reflect PCI dependency
x86/irq: Add allocation type for parent domain retrieval
iommu/vt-d: Consolidate irq domain getter
iommu/amd: Consolidate irq domain getter
iommu/irq_remapping: Consolidate irq domain lookup

4) Consolidation of the X86 specific interrupt allocation mechanism to be as close
as possible to the generic MSI allocation mechanism which allows to get rid
of quite a bunch of x86'isms which are pointless

x86/irq: Prepare consolidation of irq_alloc_info
x86/msi: Consolidate HPET allocation
x86/ioapic: Consolidate IOAPIC allocation
x86/irq: Consolidate DMAR irq allocation
x86/irq: Consolidate UV domain allocation
PCI: MSI: Rework pci_msi_domain_calc_hwirq()
x86/msi: Consolidate MSI allocation
x86/msi: Use generic MSI domain ops

5) x86 specific cleanups to remove the dependency on arch_*_msi_irqs()

x86/irq: Move apic_post_init() invocation to one place
z86/pci: Reducde #ifdeffery in PCI init code
x86/irq: Initialize PCI/MSI domain at PCI init time
irqdomain/msi: Provide DOMAIN_BUS_VMD_MSI
PCI: vmd: Mark VMD irqdomain with DOMAIN_BUS_VMD_MSI
PCI: MSI: Provide pci_dev_has_special_msi_domain() helper
x86/xen: Make xen_msi_init() static and rename it to xen_hvm_msi_init()
x86/xen: Rework MSI teardown
x86/xen: Consolidate XEN-MSI init
irqdomain/msi: Allow to override msi_domain_alloc/free_irqs()
x86/xen: Wrap XEN MSI management into irqdomain
iommm/vt-d: Store irq domain in struct device
iommm/amd: Store irq domain in struct device
x86/pci: Set default irq domain in pcibios_add_device()
PCI/MSI: Allow to disable arch fallbacks
x86/irq: Cleanup the arch_*_msi_irqs() leftovers
x86/irq: Make most MSI ops XEN private

This one is paving the way to device MSI support, but it comes
with an ugly and evil hack. The ability of overriding the default
allocation/free functions of an MSI irq domain is useful in general as
(hopefully) demonstrated with the device MSI POC, but the abuse
in context of XEN is evil. OTOH without enough XENology and without
rewriting XEN from scratch wrapping XEN MSI handling into a pseudo
irq domain is a reasonable step forward for mere mortals with severly
limited XENology. One day the XEN folks might make it a real irq domain.
Perhaps when they have to support the same mess on other architectures.
Hope dies last...

At least the mechanism to override alloc/free turned out to be useful
for implementing the base infrastructure for device MSI. So it's not a
completely lost case.

6) X86 specific preparation for device MSI

x86/irq: Add DEV_MSI allocation type
x86/msi: Let pci_msi_prepare() handle non-PCI MSI

7) Generic device MSI infrastructure

platform-msi: Provide default irq_chip:ack
platform-msi: Add device MSI infrastructure

8) Infrastructure for and a POC of an IMS (Interrupt Message
Storm) irq domain and irqchip implementation

irqdomain/msi: Provide msi_alloc/free_store() callbacks
irqchip: Add IMS array driver - NOT FOR MERGING

The whole lot is also available from git:

git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git device-msi

This has been tested on Intel/AMD/KVM but lacks testing on:

- HYPERV (-ENODEV)
- VMD enabled systems (-ENODEV)
- XEN (-ENOCLUE)

Tested to work in Xen dom0. Network is running fine with eth0 MSI
interrupts being routed through Xen.

You can add my:

Tested-by: Juergen Gross <jgross@xxxxxxxx>


Juergen