RE: [RFC PATCH Xilinx Alveo 0/6] Xilinx PCIe accelerator driver
From: Sonal Santan
Date: Wed Mar 27 2019 - 08:50:30 EST
> -----Original Message-----
> From: Daniel Vetter [mailto:daniel@xxxxxxxx]
> Sent: Wednesday, March 27, 2019 1:23 AM
> To: Sonal Santan <sonals@xxxxxxxxxx>
> Cc: dri-devel@xxxxxxxxxxxxxxxxxxxxx; gregkh@xxxxxxxxxxxxxxxxxxx; Cyril
> Chemparathy <cyrilc@xxxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx; Lizhi Hou
> <lizhih@xxxxxxxxxx>; Michal Simek <michals@xxxxxxxxxx>; airlied@xxxxxxxxxx
> Subject: Re: [RFC PATCH Xilinx Alveo 0/6] Xilinx PCIe accelerator driver
>
> On Wed, Mar 27, 2019 at 12:30 AM Sonal Santan <sonals@xxxxxxxxxx> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Daniel Vetter [mailto:daniel.vetter@xxxxxxxx] On Behalf Of
> > > Daniel Vetter
> > > Sent: Monday, March 25, 2019 1:28 PM
> > > To: Sonal Santan <sonals@xxxxxxxxxx>
> > > Cc: dri-devel@xxxxxxxxxxxxxxxxxxxxx; gregkh@xxxxxxxxxxxxxxxxxxx;
> > > Cyril Chemparathy <cyrilc@xxxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx;
> > > Lizhi Hou <lizhih@xxxxxxxxxx>; Michal Simek <michals@xxxxxxxxxx>;
> > > airlied@xxxxxxxxxx
> > > Subject: Re: [RFC PATCH Xilinx Alveo 0/6] Xilinx PCIe accelerator
> > > driver
> > >
> > > On Tue, Mar 19, 2019 at 02:53:55PM -0700, sonal.santan@xxxxxxxxxx
> wrote:
> > > > From: Sonal Santan <sonal.santan@xxxxxxxxxx>
> > > >
> > > > Hello,
> > > >
> > > > This patch series adds drivers for Xilinx Alveo PCIe accelerator cards.
> > > > These drivers are part of Xilinx Runtime (XRT) open source stack
> > > > and have been deployed by leading FaaS vendors and many enterprise
> > > customers.
> > >
> > > Cool, first fpga driver submitted to drm! And from a high level I
> > > think this makes a lot of sense.
> > >
> > > > PLATFORM ARCHITECTURE
> > > >
> > > > Alveo PCIe platforms have a static shell and a reconfigurable
> > > > (dynamic) region. The shell is automatically loaded from PROM when
> > > > host is booted and PCIe is enumerated by BIOS. Shell cannot be
> > > > changed till next cold reboot. The shell exposes two physical functions:
> > > > management physical function and user physical function.
> > > >
> > > > Users compile their high level design in C/C++/OpenCL or RTL into
> > > > FPGA image using SDx compiler. The FPGA image packaged as xclbin
> > > > file can be loaded onto reconfigurable region. The image may
> > > > contain one or more compute unit. Users can dynamically swap the
> > > > full image running on the reconfigurable region in order to switch
> > > > between different
> > > workloads.
> > > >
> > > > XRT DRIVERS
> > > >
> > > > XRT Linux kernel driver xmgmt binds to mgmt pf. The driver is
> > > > modular and organized into several platform drivers which
> > > > primarily handle the following functionality:
> > > > 1. ICAP programming (FPGA bitstream download with FPGA Mgr
> > > > integration) 2. Clock scaling 3. Loading firmware container also
> > > > called dsabin (embedded Microblaze
> > > > firmware for ERT and XMC, optional clearing bitstream) 4.
> > > > In-band
> > > > sensors: temp, voltage, power, etc.
> > > > 5. AXI Firewall management
> > > > 6. Device reset and rescan
> > > > 7. Hardware mailbox for communication between two physical
> > > > functions
> > > >
> > > > XRT Linux kernel driver xocl binds to user pf. Like its peer, this
> > > > driver is also modular and organized into several platform drivers
> > > > which handle the following functionality:
> > > > 1. Device memory topology discovery and memory management 2.
> > > > Buffer object abstraction and management for client process 3.
> > > > XDMA MM PCIe DMA engine programming 4. Multi-process aware
> context management 5.
> > > > Compute unit execution management (optionally with help of ERT) for
> > > > client processes
> > > > 6. Hardware mailbox for communication between two physical
> > > > functions
> > > >
> > > > The drivers export ioctls and sysfs nodes for various services.
> > > > xocl driver makes heavy use of DRM GEM features for device memory
> > > > management, reference counting, mmap support and export/import.
> > > > xocl also includes a simple scheduler called KDS which schedules
> > > > compute units and interacts with hardware scheduler running ERT
> > > > firmware. The scheduler understands custom opcodes packaged into
> > > > command objects
> > > and
> > > > provides an asynchronous command done notification via POSIX poll.
> > > >
> > > > More details on architecture, software APIs, ioctl definitions,
> > > > execution model, etc. is available as Sphinx documentation--
> > > >
> > > > https://xilinx.github.io/XRT/2018.3/html/index.html
> > > >
> > > > The complete runtime software stack (XRT) which includes out of
> > > > tree kernel drivers, user space libraries, board utilities and
> > > > firmware for the hardware scheduler is open source and available
> > > > at https://github.com/Xilinx/XRT
> > >
> > > Before digging into the implementation side more I looked into the
> > > userspace here. I admit I got lost a bit, since there's lots of
> > > indirections and abstractions going on, but it seems like this is
> > > just a fancy ioctl wrapper/driver backend abstractions. Not really
> something applications would use.
> > Sonal Santan <sonals@xxxxxxxxxx>
> >
> > 4:20 PM (1 minute ago)
> >
> > to me
> >
> >
> > > -----Original Message-----
> > > From: Daniel Vetter [mailto:daniel.vetter@xxxxxxxx] On Behalf Of
> > > Daniel Vetter
> > > Sent: Monday, March 25, 2019 1:28 PM
> > > To: Sonal Santan <sonals@xxxxxxxxxx>
> > > Cc: dri-devel@xxxxxxxxxxxxxxxxxxxxx; gregkh@xxxxxxxxxxxxxxxxxxx;
> > > Cyril Chemparathy <cyrilc@xxxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx;
> > > Lizhi Hou <lizhih@xxxxxxxxxx>; Michal Simek <michals@xxxxxxxxxx>;
> > > airlied@xxxxxxxxxx
> > > Subject: Re: [RFC PATCH Xilinx Alveo 0/6] Xilinx PCIe accelerator
> > > driver
> > >
> > > On Tue, Mar 19, 2019 at 02:53:55PM -0700, sonal.santan@xxxxxxxxxx
> wrote:
> > > > From: Sonal Santan <sonal.santan@xxxxxxxxxx>
> > > >
> > > > Hello,
> > > >
> > > > This patch series adds drivers for Xilinx Alveo PCIe accelerator cards.
> > > > These drivers are part of Xilinx Runtime (XRT) open source stack
> > > > and have been deployed by leading FaaS vendors and many enterprise
> > > customers.
> > >
> > > Cool, first fpga driver submitted to drm! And from a high level I
> > > think this makes a lot of sense.
> > >
> > > > PLATFORM ARCHITECTURE
> > > >
> > > > Alveo PCIe platforms have a static shell and a reconfigurable
> > > > (dynamic) region. The shell is automatically loaded from PROM when
> > > > host is booted and PCIe is enumerated by BIOS. Shell cannot be
> > > > changed till next cold reboot. The shell exposes two physical functions:
> > > > management physical function and user physical function.
> > > >
> > > > Users compile their high level design in C/C++/OpenCL or RTL into
> > > > FPGA image using SDx compiler. The FPGA image packaged as xclbin
> > > > file can be loaded onto reconfigurable region. The image may
> > > > contain one or more compute unit. Users can dynamically swap the
> > > > full image running on the reconfigurable region in order to switch
> > > > between different
> > > workloads.
> > > >
> > > > XRT DRIVERS
> > > >
> > > > XRT Linux kernel driver xmgmt binds to mgmt pf. The driver is
> > > > modular and organized into several platform drivers which
> > > > primarily handle the following functionality:
> > > > 1. ICAP programming (FPGA bitstream download with FPGA Mgr
> > > > integration) 2. Clock scaling 3. Loading firmware container also
> > > > called dsabin (embedded Microblaze
> > > > firmware for ERT and XMC, optional clearing bitstream) 4.
> > > > In-band
> > > > sensors: temp, voltage, power, etc.
> > > > 5. AXI Firewall management
> > > > 6. Device reset and rescan
> > > > 7. Hardware mailbox for communication between two physical
> > > > functions
> > > >
> > > > XRT Linux kernel driver xocl binds to user pf. Like its peer, this
> > > > driver is also modular and organized into several platform drivers
> > > > which handle the following functionality:
> > > > 1. Device memory topology discovery and memory management 2.
> > > > Buffer object abstraction and management for client process 3.
> > > > XDMA MM PCIe DMA engine programming 4. Multi-process aware
> context management 5.
> > > > Compute unit execution management (optionally with help of ERT) for
> > > > client processes
> > > > 6. Hardware mailbox for communication between two physical
> > > > functions
> > > >
> > > > The drivers export ioctls and sysfs nodes for various services.
> > > > xocl driver makes heavy use of DRM GEM features for device memory
> > > > management, reference counting, mmap support and export/import.
> > > > xocl also includes a simple scheduler called KDS which schedules
> > > > compute units and interacts with hardware scheduler running ERT
> > > > firmware. The scheduler understands custom opcodes packaged into
> > > > command objects
> > > and
> > > > provides an asynchronous command done notification via POSIX poll.
> > > >
> > > > More details on architecture, software APIs, ioctl definitions,
> > > > execution model, etc. is available as Sphinx documentation--
> > > >
> > > > https://xilinx.github.io/XRT/2018.3/html/index.html
> > > >
> > > > The complete runtime software stack (XRT) which includes out of
> > > > tree kernel drivers, user space libraries, board utilities and
> > > > firmware for the hardware scheduler is open source and available
> > > > at https://github.com/Xilinx/XRT
> > >
> > > Before digging into the implementation side more I looked into the
> > > userspace here. I admit I got lost a bit, since there's lots of
> > > indirections and abstractions going on, but it seems like this is
> > > just a fancy ioctl wrapper/driver backend abstractions. Not really
> something applications would use.
> > >
> >
> > Appreciate your feedback.
> >
> > The userspace libraries define a common abstraction but have different
> > implementations for Zynq Ultrascale+ embedded platform, PCIe based
> > Alveo (and Faas) and emulation flows. The latter lets you run your
> application without physical hardware.
> >
> > >
> > > From the pretty picture on github it looks like there's some
> > > opencl/ml/other fancy stuff sitting on top that applications would use. Is
> that also available?
> >
> > The full OpenCL runtime is available in the same repository. Xilinx ML
> > Suite is also based on XRT and its source can be found at
> https://github.com/Xilinx/ml-suite.
>
> Hm, I did a few git grep for the usual opencl entry points, but didn't find
> anything. Do I need to run some build scripts first (which downloads
> additional sourcecode)? Or is there some symbol mangling going on and that's
> why I don't find anything? Pointers very much appreciated.
The bulk of the OCL runtime code can be found inside
https://github.com/Xilinx/XRT/tree/master/src/runtime_src/xocl.
The OCL runtime also includes https://github.com/Xilinx/XRT/tree/master/src/runtime_src/xrt.
The OCL runtime library called libxilinxopencl.so in turn then uses XRT APIs to talk to the drivers.
For PCIe these XRT APIs are implemented in the library libxrt_core.so the source for which is
https://github.com/Xilinx/XRT/tree/master/src/runtime_src/driver/xclng/xrt.
You can build a fully functioning runtime stack by following very simple build instructions--
https://xilinx.github.io/XRT/master/html/build.html
We do have a few dependencies on standard Linux packages including a few OpenCL packages
bundled by Linux distros: ocl-icd, ocl-icd-devel and opencl-headers
Thanks,
-Sonal
>
> > Typically end users use OpenCL APIs which are part of XRT stack. One
> > can write an application to directly call XRT APIs defined at
> > https://xilinx.github.io/XRT/2018.3/html/xclhal2.main.html
>
> I have no clue about DNN/ML unfortunately, I think I'll try to look into the ocl
> side a bit more first.
>
> Thanks, Daniel
>
> >
> > Thanks,
> > -Sonal
> > >
> > > Thanks, Daniel
> > >
> > > >
> > > > Thanks,
> > > > -Sonal
> > > >
> > > > Sonal Santan (6):
> > > > Add skeleton code: ioctl definitions and build hooks
> > > > Global data structures shared between xocl and xmgmt drivers
> > > > Add platform drivers for various IPs and frameworks
> > > > Add core of XDMA driver
> > > > Add management driver
> > > > Add user physical function driver
> > > >
> > > > drivers/gpu/drm/Kconfig | 2 +
> > > > drivers/gpu/drm/Makefile | 1 +
> > > > drivers/gpu/drm/xocl/Kconfig | 22 +
> > > > drivers/gpu/drm/xocl/Makefile | 3 +
> > > > drivers/gpu/drm/xocl/devices.h | 954 +++++
> > > > drivers/gpu/drm/xocl/ert.h | 385 ++
> > > > drivers/gpu/drm/xocl/lib/Makefile.in | 16 +
> > > > drivers/gpu/drm/xocl/lib/cdev_sgdma.h | 63 +
> > > > drivers/gpu/drm/xocl/lib/libxdma.c | 4368 ++++++++++++++++++++
> > > > drivers/gpu/drm/xocl/lib/libxdma.h | 596 +++
> > > > drivers/gpu/drm/xocl/lib/libxdma_api.h | 127 +
> > > > drivers/gpu/drm/xocl/mgmtpf/Makefile | 29 +
> > > > drivers/gpu/drm/xocl/mgmtpf/mgmt-core.c | 960 +++++
> > > > drivers/gpu/drm/xocl/mgmtpf/mgmt-core.h | 147 +
> > > > drivers/gpu/drm/xocl/mgmtpf/mgmt-cw.c | 30 +
> > > > drivers/gpu/drm/xocl/mgmtpf/mgmt-ioctl.c | 148 +
> > > > drivers/gpu/drm/xocl/mgmtpf/mgmt-reg.h | 244 ++
> > > > drivers/gpu/drm/xocl/mgmtpf/mgmt-sysfs.c | 318 ++
> > > > drivers/gpu/drm/xocl/mgmtpf/mgmt-utils.c | 399 ++
> > > > drivers/gpu/drm/xocl/subdev/dna.c | 356 ++
> > > > drivers/gpu/drm/xocl/subdev/feature_rom.c | 412 ++
> > > > drivers/gpu/drm/xocl/subdev/firewall.c | 389 ++
> > > > drivers/gpu/drm/xocl/subdev/fmgr.c | 198 +
> > > > drivers/gpu/drm/xocl/subdev/icap.c | 2859 +++++++++++++
> > > > drivers/gpu/drm/xocl/subdev/mailbox.c | 1868 +++++++++
> > > > drivers/gpu/drm/xocl/subdev/mb_scheduler.c | 3059 ++++++++++++++
> > > > drivers/gpu/drm/xocl/subdev/microblaze.c | 722 ++++
> > > > drivers/gpu/drm/xocl/subdev/mig.c | 256 ++
> > > > drivers/gpu/drm/xocl/subdev/sysmon.c | 385 ++
> > > > drivers/gpu/drm/xocl/subdev/xdma.c | 510 +++
> > > > drivers/gpu/drm/xocl/subdev/xmc.c | 1480 +++++++
> > > > drivers/gpu/drm/xocl/subdev/xvc.c | 461 +++
> > > > drivers/gpu/drm/xocl/userpf/Makefile | 27 +
> > > > drivers/gpu/drm/xocl/userpf/common.h | 157 +
> > > > drivers/gpu/drm/xocl/userpf/xocl_bo.c | 1255 ++++++
> > > > drivers/gpu/drm/xocl/userpf/xocl_bo.h | 119 +
> > > > drivers/gpu/drm/xocl/userpf/xocl_drm.c | 640 +++
> > > > drivers/gpu/drm/xocl/userpf/xocl_drv.c | 743 ++++
> > > > drivers/gpu/drm/xocl/userpf/xocl_ioctl.c | 396 ++
> > > > drivers/gpu/drm/xocl/userpf/xocl_sysfs.c | 344 ++
> > > > drivers/gpu/drm/xocl/version.h | 22 +
> > > > drivers/gpu/drm/xocl/xclbin.h | 314 ++
> > > > drivers/gpu/drm/xocl/xclfeatures.h | 107 +
> > > > drivers/gpu/drm/xocl/xocl_ctx.c | 196 +
> > > > drivers/gpu/drm/xocl/xocl_drm.h | 91 +
> > > > drivers/gpu/drm/xocl/xocl_drv.h | 783 ++++
> > > > drivers/gpu/drm/xocl/xocl_subdev.c | 540 +++
> > > > drivers/gpu/drm/xocl/xocl_thread.c | 64 +
> > > > include/uapi/drm/xmgmt_drm.h | 204 +
> > > > include/uapi/drm/xocl_drm.h | 483 +++
> > > > 50 files changed, 28252 insertions(+) create mode 100644
> > > > drivers/gpu/drm/xocl/Kconfig create mode 100644
> > > > drivers/gpu/drm/xocl/Makefile create mode 100644
> > > > drivers/gpu/drm/xocl/devices.h create mode 100644
> > > > drivers/gpu/drm/xocl/ert.h create mode 100644
> > > > drivers/gpu/drm/xocl/lib/Makefile.in
> > > > create mode 100644 drivers/gpu/drm/xocl/lib/cdev_sgdma.h
> > > > create mode 100644 drivers/gpu/drm/xocl/lib/libxdma.c
> > > > create mode 100644 drivers/gpu/drm/xocl/lib/libxdma.h
> > > > create mode 100644 drivers/gpu/drm/xocl/lib/libxdma_api.h
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/Makefile
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-core.c
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-core.h
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-cw.c
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-ioctl.c
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-reg.h
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-sysfs.c
> > > > create mode 100644 drivers/gpu/drm/xocl/mgmtpf/mgmt-utils.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/dna.c create mode
> > > > 100644 drivers/gpu/drm/xocl/subdev/feature_rom.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/firewall.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/fmgr.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/icap.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/mailbox.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/mb_scheduler.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/microblaze.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/mig.c create mode
> > > > 100644 drivers/gpu/drm/xocl/subdev/sysmon.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/xdma.c
> > > > create mode 100644 drivers/gpu/drm/xocl/subdev/xmc.c create mode
> > > > 100644 drivers/gpu/drm/xocl/subdev/xvc.c create mode 100644
> > > > drivers/gpu/drm/xocl/userpf/Makefile
> > > > create mode 100644 drivers/gpu/drm/xocl/userpf/common.h
> > > > create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_bo.c
> > > > create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_bo.h
> > > > create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_drm.c
> > > > create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_drv.c
> > > > create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_ioctl.c
> > > > create mode 100644 drivers/gpu/drm/xocl/userpf/xocl_sysfs.c
> > > > create mode 100644 drivers/gpu/drm/xocl/version.h create mode
> > > > 100644 drivers/gpu/drm/xocl/xclbin.h create mode 100644
> > > > drivers/gpu/drm/xocl/xclfeatures.h
> > > > create mode 100644 drivers/gpu/drm/xocl/xocl_ctx.c create mode
> > > > 100644 drivers/gpu/drm/xocl/xocl_drm.h create mode 100644
> > > > drivers/gpu/drm/xocl/xocl_drv.h create mode 100644
> > > > drivers/gpu/drm/xocl/xocl_subdev.c
> > > > create mode 100644 drivers/gpu/drm/xocl/xocl_thread.c
> > > > create mode 100644 include/uapi/drm/xmgmt_drm.h create mode
> > > > 100644 include/uapi/drm/xocl_drm.h
> > > >
> > > > --
> > > > 2.17.0
> > > > _______________________________________________
> > > > dri-devel mailing list
> > > > dri-devel@xxxxxxxxxxxxxxxxxxxxx
> > > > https://lists.freedesktop.org/mailman/listinfo/dri-devel
> > >
> > > --
> > > Daniel Vetter
> > > Software Engineer, Intel Corporation http://blog.ffwll.ch
>
>
>
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch