Re: [PATCH] arm64: Add architecture support for PCI

From: Rob Herring
Date: Tue Feb 04 2014 - 08:57:56 EST

On Tue, Feb 4, 2014 at 3:44 AM, Arnd Bergmann <arnd@xxxxxxxx> wrote:
> On Monday 03 February 2014 16:31:37 Jason Gunthorpe wrote:
>> Specifying 'use EHCI, AHCI, etc' - which are all PCI based standards
>> without clearly specifying exactly how PCI is suppose to work is
>> completely bonkers.
>> What is needed is a spec that says:
>> 1) Here is how you generate config TLPs. A MMIO region that
>> conforms to the already specified x86 ECAM would
>> be perfect
>> 2) Here is a dword by dword break down of the entire config space in
>> a SOC. Here is where a on-board AHCI controller must show up in
>> config space. Here is how an external PCI-E port must show
>> up. Etc. Most of this is already specified, but it clearly needs
>> to be layed out explicitly for ARM SOCs to actually follow it.
>> 3) Here is how you specify the aperture(s) associated with PCI BAR's
>> and bridge windows in config space. And yes: The CONFIG SPACE
>> 4) Here is how MSI works, these are the values you put in the
>> address/data and here is how you collect the interrupt.
>> 5) Here is how Legacy INTx must be mapped into the GIC.
>> This is what x86 does, and they have been doing it well for 10
>> years. If you want to play in the server game you have to properly
>> implement PCI.
> I'm pretty sure the authors of the SBSA actually thought that was
> what they wrote, by referring to external specifications (pci-3.0,
> ehci, ahci, ...). However, it seems they were either foolish enough
> to believe that hardware designers would follow these specs, or
> they were intentionally misled and got talked into putting ambiguous
> terminology in because there were already hardware designs that
> are not exactly in line with the spirit of the SBSA but can be
> argued to be conforming to the text for a lax interpretation.
> I think EHCI is a much better example than PCI here, because the
> spec has exactly one line to say about it, where it spends a whole
> chapter on PCI.
> Here is how a sane person would read SBSA to create a compliant
> implementation:


> I have to use EHCI version 1.1 to provide USB-2.0 support. EHCI
> is a PCI device, so I'll put it behind a PCIe port that complies
> to the PCIe section of the SBSA. Since EHCI by itself only provides
> high-speed USB, and USB-2.0 mandates I provide low-speed and
> full-speed as well, I have to add a USB hub device. It would have
> been easier to just use OHCI for these, but SBSA says I can't.
> Now I want to integrate the EHCI into my SoC and not waste one
> of my precious PCIe root ports, so I have to create another PCI
> domain with its own ECAM compliant config space to put it into.
> Fortunately SBSA lets me add an arbitrary number of PCI domains,
> as long as they are all strictly compliant. To software it will
> look exactly as if it was on an external card, I just have to
> ensure the boot loader correctly sets up the clocks and the phy
> before an SBSA compliant OS gets loaded, all runtime power
> management will get handled through the EHCI-1.1 energy-efficiency
> extensions.
> Here is how a crazy person would read the same sentence in the SBSA:


> I have an IP block that implements the EHCI register set, that
> should be good enough. It's not a fast device, so I can put it
> on a non-coherent bus. Since the SoC will be used for networking,
> I'll put the registers into big-endian configuration to make it
> easier for the OS to access. I'm not allowed to have USB-1.1
> according to SBSA, so I can get away without a hub or an extra
> OHCI. I can't support MSI because it's not a PCI device, and
> the GIC is pretty full, so I'll just connect the IRQ line to
> the GPIO controller. In order to do better power management,
> I'll design a fancy PHY that the device driver will manage
> for implementing autosuspend. I should also give the OS
> fine-grained control over the clocks, but it will have to share
> the clock domain with the other devices on the same edge of the
> chip. The EHCI device is not part of PCI, which measn I don't
> have to use the standard SMMU. However, my EHCI implementation
> can only do 32-bit DMA, and I'll have to design my own IOMMU
> to let it access the entire memory range. USB-OTG is a great
> feature and we already paid for having this in our EHCI
> implementation, better make sure it comes up in endpoint mode
> after waking up from power saving.

My money is on the latter. I think non-PCI implementations of xHCI
interfaces will be common. This was certainly the case at Calxeda in
what was believed to be a SBSA compliant SOC. However, I think PCI
device or not is the least of the issues and all the other examples
you list are the difficult ones to deal with.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at