Re: [PATCH v2] PCI/MSI: Don't touch MSI bits when the PCI device is disconnected

From: Bjorn Helgaas
Date: Thu Nov 15 2018 - 01:24:26 EST


On Wed, Nov 14, 2018 at 07:22:04PM +0000, Alex_Gagniuc@xxxxxxxxxxxx wrote:
> On 11/14/2018 12:00 AM, Bjorn Helgaas wrote:
> > On Tue, Nov 13, 2018 at 10:39:15PM +0000, Alex_Gagniuc@xxxxxxxxxxxx wrote:
> >> On 11/12/2018 11:02 PM, Bjorn Helgaas wrote:
> >> ...
> >>> Do you think Linux observes the rule about not touching AER bits on
> >>> FFS? I'm not sure it does. I'm not even sure what section of the
> >>> spec is relevant.
> >>
> >> I haven't found any place where linux breaks this rule. I'm very
> >> confident that, unless otherwise instructed, we follow this rule.
> >
> > Just to make sure we're on the same page, can you point me to this
> > rule? I do see that OSPM must request control of AER using _OSC
> > before it touches the AER registers. What I don't see is the
> > connection between firmware-first and the AER registers.
>
> ACPI 6.2 - 6.2.11.3, Table 6-197:
>
> PCI Express Advanced Error Reporting control:
> * The firmware sets this bit to 1 to grant control over PCI Express
> Advanced Error Reporting. If firmware allows the OS control of this
> feature, then in the context of the _OSC method it must ensure that
> error messages are routed to device interrupts as described in the PCI
> Express Base Specification[...]

The PCIe Base Spec is pretty big, so I wish this reference were a
little more explicit. I *guess* maybe it's referring to PCIe r4.0,
figure 6-3 in sec 6.2.6, where PCIe ERR_* Messages can be routed to
"INTx or MSI Error Interrupts" and/or "platform-specific System Error"
interrupts.

"Device interrupts" seems like it refers to the "INTx or MSI"
interrupts, not the platform-specific System Errors, so I would read
that as saying "if firmware grants OS control of AER via _OSC,
firmware must set the AER Reporting Enables in the AER Root Error
Command register." But that seems a little silly because the OS now
*owns* the AER capability and it can set the AER Root Error Command
register itself if it wants to.

And I still don't see the connection here with Firmware-First. I'm
pretty sure firmware could not be notified via INTx or MSI interrupts
because those are totally managed by OSPM.

> > The closest I can find is the "Enabled" field in the HEST PCIe
> > AER structures (ACPI v6.2, sec 18.3.2.4, .5, .6), where it says:
> >
> > If the field value is 1, indicates this error source is
> > to be enabled.
> >
> > If the field value is 0, indicates that the error source
> > is not to be enabled.
> >
> > If FIRMWARE_FIRST is set in the flags field, the Enabled
> > field is ignored by the OSPM.
> >
> > AFAICT, Linux completely ignores the Enabled field in these
> > structures.
>
> I don't think ignoring the field is a problem:
> * With FFS, OS should ignore it.
> * Without FFS, we have control, and we get to make the decisions anyway.
> In the latter case we decide whether to use AER, independent of the crap
> in ACPI. I'm not even sure why "Enabled" matters in native AER handling.

It seems like these HEST structures are "here's how firmware thinks
you should set up AER on this device". But I agree, I have no idea
how to interpret "Enabled". The rest of the HEST fields cover all the
useful AER registers, including the Reporting Enables in the AER Root
Error Command register *and* the Error Reporting Enables in the Device
Control register. So I don't know what the "Enabled" field adds to
all that. What a mess.

> > For firmware-first to work, firmware has to get control. How does
> > it get control? How does OSPM know to either set up that
> > mechanism or keep its mitts off something firmware set up before
> > handoff?
>
> My understanding is that, if FW keeps control of AER in _OSC, then
> it will have set things up to get notified instead of the OS. OSPM
> not touching AER bits is to make sure it doesn't mess up FW's setup.
> I think there are some proprietary bits in the root port to route
> interrupts to SMIs instead of the AER vectors.

It makes good sense that if OSPM doesn't have AER control, firmware
does all AER handling, including any setup for firmware-first
notification. If we can assume that firmware-first notification is
done in some way the OS doesn't know about and can't mess up, that
would be awesome.

But I think the VMD model really has nothing to do with the APEI
firmware-first model. With VMD, it sounds like OSPM owns the AER
capability and doesn't know firmware exists *except* that it has to be
careful not to step on firmware's interrupt. So maybe we can handle it
separately.

Bjorn