Re: [PATCH v2 0/3] media: pci: intel: ivsc: Add driver of Intel Visual Sensing Controller(IVSC)

From: Bingbu Cao
Date: Wed Feb 15 2023 - 07:13:57 EST


Hi, Wentong,

On 2/15/23 5:43 PM, Laurent Pinchart wrote:
> Hello Wentong,
>
> On Mon, Feb 13, 2023 at 10:23:44AM +0800, Wentong Wu wrote:
>> Intel Visual Sensing Controller (IVSC), codenamed "Clover Falls", is a
>> companion chip designed to provide secure and low power vision capability
>> to IA platforms. IVSC is available in existing commercial platforms from
>> multiple OEMs.
>>
>> The primary use case of IVSC is to bring in context awareness. IVSC
>> interfaces directly with the platform main camera sensor via a CSI-2 link
>> and processes the image data with the embedded AI engine. The detected
>> events are sent over I2C to ISH (Intel Sensor Hub) for additional data
>> fusion from multiple sensors. The fusion results are used to implement
>> advanced use cases like:
>> - Face detection to unlock screen
>> - Detect user presence to manage backlight setting or waking up system
>
> Do you have plan to support these features in the ivsc driver in the
> future ?
>
>> Since the Image Processing Unit(IPU) used on the host processor needs to
>> configure the CSI-2 link in normal camera usages, the CSI-2 link and
>> camera sensor can only be used in mutually-exclusive ways by host IPU and
>> IVSC. By default the IVSC owns the CSI-2 link and camera sensor. The IPU
>> driver can take ownership of the CSI-2 link and camera sensor using
>> interfaces provided by this IVSC driver.
>>
>> Switching ownership requires an interface with two different hardware
>> modules inside IVSC. The software interface to these modules is via Intel
>> MEI (The Intel Management Engine) commands. These two hardware modules
>> have two different MEI UUIDs to enumerate. These hardware modules are:
>> - ACE (Algorithm Context Engine): This module is for algorithm computing
>> when IVSC owns camera sensor. Also ACE module controls camera sensor's
>> ownership. This hardware module is used to set ownership of camera sensor.
>> - CSI (Camera Serial Interface): This module is used to route camera
>> sensor data either to IVSC or to host for IPU driver and application.
>>
>> IVSC also provides a privacy mode. When privacy mode is turned on,
>> camera sensor can't be used. This means that both ACE and host IPU can't
>> get image data. And when this mode is turned on, host IPU driver is
>> informed via a registered callback, so that user can be notified.
>
> How does the privacy mode work, and how can the user trust that the
> closed-source IVSC and IME firmwares will honour the privacy settings ?
>

Continue with question from Laurent,

How IVSC handle the privacy request from user? Is there some notification
mechanism to user-space? I'd have concern if IVSC driver need private callback
to request back-end(e.g. ISP driver) to handle stream cutting.

>> In summary, to acquire ownership of camera by IPU driver, first ACE
>> module needs to be informed of ownership and then to setup MIPI CSI-2
>> link for the camera sensor and IPU.
>>
>> Implementation:
>> There are two different drivers to handle ACE and CSI hardware modules
>> inside IVSC.
>> - mei_csi: MEI client driver to send commands and receive notifications
>> from CSI module.
>> - mei_ace: MEI client driver to send commands and get status from ACE
>> module.
>> Interface is exposed via ivsc.h to acquire and release camera sensor and
>> CSI-2 link.
>
> Do I understand correctly, from your diagram below, that the
> communication between the IME and IVSC goes through SPI ?
>
>> Below diagram shows connections of IVSC/ISH/IPU/Camera sensor.
>> -----------------------------------------------------------------------------
>> | Host Processor |
>> | |
>> | ----------------- ----------------- --------------- |
>> | | | | | | | I2C |
>> | | IPU | | ISH | |camera driver|--| |
>> | | | | | | | | |
>> | ----------------- ----------------- --------------- | |
>> | | | | | |
>> | | | --------------- | |
>> | | | | | | |
>> | | | | IVSC driver | | |
>> | | | | | | |
>> | | | --------------- | |
>> | | | | | |
>> ----------------|-----------------------|----------------------|---------|---
>> | CSI | I2C |SPI |
>> | | | |
>> ----------------|-----------------------|---------------- | |
>> | IVSC | | | |
>> | | | | |
>> | ----------------- ----------------- | | |
>> | | | | | | | |
>> | | CSI | | ACE | |------| |
>> | | | | | | |
>> | ----------------- ----------------- | |
>> | | | I2C | |
>> ----------------|-----------------------|---------------- |
>> | CSI | |
>> | | |
>> -------------------------------- |
>> | | I2C |
>> | camera sensor |-----------------------------|
>> | |
>> --------------------------------
>>
>> Wentong Wu (3):
>> media: pci: intel: ivsc: Add CSI submodule
>> media: pci: intel: ivsc: Add ACE submodule
>> media: pci: intel: ivsc: Add acquire/release API for ivsc
>>
>> drivers/media/pci/Kconfig | 1 +
>> drivers/media/pci/intel/Makefile | 2 +
>> drivers/media/pci/intel/ivsc/Kconfig | 12 +
>> drivers/media/pci/intel/ivsc/Makefile | 7 +
>> drivers/media/pci/intel/ivsc/ivsc.c | 84 +++++
>> drivers/media/pci/intel/ivsc/mei_ace.c | 472 +++++++++++++++++++++++++
>> drivers/media/pci/intel/ivsc/mei_ace.h | 36 ++
>> drivers/media/pci/intel/ivsc/mei_csi.c | 342 ++++++++++++++++++
>> drivers/media/pci/intel/ivsc/mei_csi.h | 60 ++++
>> include/linux/ivsc.h | 74 ++++
>> 10 files changed, 1090 insertions(+)
>> create mode 100644 drivers/media/pci/intel/ivsc/Kconfig
>> create mode 100644 drivers/media/pci/intel/ivsc/Makefile
>> create mode 100644 drivers/media/pci/intel/ivsc/ivsc.c
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.c
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_ace.h
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.c
>> create mode 100644 drivers/media/pci/intel/ivsc/mei_csi.h
>> create mode 100644 include/linux/ivsc.h
>

--
Best regards,
Bingbu Cao