Re: [PATCH v7 2/2] PCI: keystone: Fix pci_ops for AM654x SoC

[Siddharth Vadapalli]

On Mon, May 13, 2024 at 04:53:50PM -0500, Bjorn Helgaas wrote:
> On Thu, Mar 28, 2024 at 02:20:41PM +0530, Siddharth Vadapalli wrote:
> > In the process of converting .scan_bus() callbacks to .add_bus(), the
> > ks_pcie_v3_65_scan_bus() function was changed to ks_pcie_v3_65_add_bus().
> > The .scan_bus() method belonged to ks_pcie_host_ops which was specific
> > to controller version 3.65a, while the .add_bus() method had been added
> > to ks_pcie_ops which is shared between the controller versions 3.65a and
> > 4.90a. Neither the older ks_pcie_v3_65_scan_bus() method, nor the newer
> > ks_pcie_v3_65_add_bus() method is applicable to the controller version
> > 4.90a which is present in AM654x SoCs.
> > 
> > Thus, as a fix, remove "ks_pcie_v3_65_add_bus()" and move its contents
> > to the .msi_init callback "ks_pcie_msi_host_init()" which is specific to
> > the 3.65a controller.
> > 
> > Fixes: 6ab15b5e7057 ("PCI: dwc: keystone: Convert .scan_bus() callback to use add_bus")
> > Suggested-by: Serge Semin <fancer.lancer@xxxxxxxxx>
> > Suggested-by: Bjorn Helgaas <helgaas@xxxxxxxxxx>
> > Suggested-by: Niklas Cassel <cassel@xxxxxxxxxx>
> > Reviewed-by: Niklas Cassel <cassel@xxxxxxxxxx>
> > Signed-off-by: Siddharth Vadapalli <s-vadapalli@xxxxxx>
> 
> Thanks for splitting this into two patches.  Krzysztof has applied
> both to pci/controller/keystone and we hope to merge them for v6.10.
> 
> I *would* like the commit log to be at a little higher level if
> possible.  Right now it's a detailed description at the level of the
> code edits, but it doesn't say *why* we want this change.
> 
> I think the first cut at this was
> https://lore.kernel.org/linux-pci/20231011123451.34827-1-s-vadapalli@xxxxxx/t/#u,
> which mentioned Completion Timeouts during MSI-X configuration and 45
> second delays during boot.
> 
> IIUC, prior to 6ab15b5e7057, ks_pcie_v3_65_scan_bus() initialized BAR
> 0 and was only used for v3.65a devices.  6ab15b5e7057 renamed it to
> ks_pcie_v3_65_add_bus() and called it for both v3.65a and v4.90a.
> 
> I think the problem is that in the current code, the
> ks_pcie_ops.add_bus() method (ks_pcie_v3_65_add_bus()) is used for all
> devices (both v3.65a and v4.90a).  So I guess doing the BAR 0 setup on
> v4.90a broke something there?

BAR0 was set to a different value on AM654x SoC which has the v4.90a
controller, which is identical to what is set even for the v3.65a
controller. The difference is that BAR0 is programmed to a different
value for enabling inbound MSI writes on top of the common configuration
performed for BAR0.

Common configuration for BAR0:
ks_pcie_probe
  dw_pcie_host_init
    dw_pcie_setup_rc
    ...
     /* Setup RC BARs */
     dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x00000004);
     dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x00000000);
     ...
     dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0);
    ...

MSI specific configuration of BAR0 performed after the common
configuration via the ks_pcie_v3_65_scan_bus() callback:
	/* Configure and set up BAR0 */
	ks_pcie_set_dbi_mode(ks_pcie);

	/* Enable BAR0 */
	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 1);
	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, SZ_4K - 1);

	ks_pcie_clear_dbi_mode(ks_pcie);

	 /*
	  * For BAR0, just setting bus address for inbound writes (MSI) should
	  * be sufficient.  Use physical address to avoid any conflicts.
	  */
	dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, ks_pcie->app.start);

The above configuration of BAR0 shouldn't be performed for AM654x SoC.
While I am not certain, the timeouts are probably a result of the BAR
being programmed to a wrong value which results in a "no match" outcome.

> 
> I'm not quite clear on the mechanism, but it would be helpful to at
> least know what's wrong and on what platform.  E.g., currently v4.90
> suffers Completion Timeouts and 45 second boot delays?  And this patch
> fixes that?

Yes, the Completion Timeouts cause the 45 second boot delays and this
patch fixes that.

Regards,
Siddharth.