Mapping more than 3GB PCIe BAR memory

[Anil Abraham]

Hi, I have a use case where there are two PCIe devices connected to
two CPU sockets. Each of the device is exposing 1GB device memory over
their one of the BARs(BAR5/6). Since other two BARs(BAR 1/2/3/4) also
expose some amount of memory, the total memory requirement on the
upstream PCI-bridge goes upto 1.5GB(some round-offs happen at the
bridge) per socket. With this the linux kernel fails to allocate
region in the physical address space. I see the following errors in
the kernel bootup:

[    6.589964] pci 0000:84:06.0: BAR 14: can't assign mem (size 0x60000000)
[    6.611992] pci 0000:84:00.0: BAR 14: can't assign mem (size 0x100000)
[    6.633446] pci 0000:84:01.0: BAR 14: can't assign mem (size 0x100000)
[    6.654898] pci 0000:84:02.0: BAR 14: can't assign mem (size 0x100000)
[    6.676351] pci 0000:84:03.0: BAR 14: can't assign mem (size 0x100000)
[    6.697806] pci 0000:84:04.0: BAR 14: can't assign mem (size 0x100000)
[    6.719261] pci 0000:84:05.0: BAR 14: can't assign mem (size 0x100000)
[    6.740716] pci 0000:85:00.0: BAR 0: can't assign mem (size 0x10000)
...
[    7.087688] pci 0000:8b:00.0: BAR 4: can't assign mem (size 0x40000000)
[    7.109427] pci 0000:8b:00.0: BAR 0: can't assign mem (size 0x100000)
[    7.130599] pci 0000:8b:00.0: BAR 2: can't assign mem (size 0x2000)

I looked at the kernel code specially the files
drivers/pci/setup-res.c and pci_bus_alloc_resource() functions,
further down to e820.c. There I found this macro

#define MAX_GAP_END 0x100000000ull

Does this mean that PCI devices can be allocated addresses in 32-bit
space only even though the kernel is 64 bit and the device supports
64-bit registers. I can see that one of the reason could be to be
backward compatible with 32-bit devices. Is there a way we can
overcome this if at all, it is a limitation.

kernel version: 3.10.62

Kernel boot logs attached

Thanks in Advance for any responses.

-Anil

Attachment: kernel-pci-boot.rtf
Description: RTF file