Re: Re: [RFC 0/2] Add standard exposure and gain controls for multiple captures

[Mirela Rabulea]

Hi Laurent and all,

On 7/23/25 17:00, Laurent Pinchart wrote:

> On Tue, Jul 22, 2025 at 10:46:16AM +0200, Julien Vuillaumier wrote:
> > On 16/07/2025 02:12, Laurent Pinchart wrote:
> > > On Wed, Jul 16, 2025 at 02:59:54AM +0300, Laurent Pinchart wrote:
> > > > On Fri, Jul 11, 2025 at 01:05:42AM +0300, Mirela Rabulea wrote:
> > > > > Add new standard controls as U32 arrays, for sensors with multiple
> > > > > captures: V4L2_CID_EXPOSURE_MULTI, V4L2_CID_AGAIN_MULTI and
> > > > > V4L2_CID_DGAIN_MULTI. These will be particularly useful for sensors
> > > > > that have multiple captures, but the HDR merge is done inside the sensor,
> > > > > in the end exposing a single stream, but still requiring AEC control
> > > > > for all captures.
> > > >
> > > > It's also useful for sensors supporting DOL or DCG with HDR merge being
> > > > performed outside of the sensor.
> > >
> > > Regarless of where HDR merge is implemented, we will also need controls
> > > to select the HDR mode. We have V4L2_CID_HDR_SENSOR_MODE, which doesn't
> > > standardize the values, and that's not good enough. At least for DOL and
> > > DCG with HDR merge implemented outside of the sensor, we need to
> > > standardize the modes.
> >
> > For the HDR-capable sensors with the HDR merge implemented outside, the
> > short capture(s) are likely implemented as separate streams, in order to
> > match the raw camera sensor model.
>
> Yes, that's my expectation. They should use a different data type or a
> different virtual channel (I expect most sensors to support both
> options).
>
> > In that case, the SDR/HDR mode switch, when supported, can be done by
> > configuring the sensor device internal route for the short capture stream.
>
> That's an option too, but it won't allow us to select between different
> HDR modes. For instance, the AR0830 supports both DOL (2 exposures) and
> DCG (2 gains). We would need a way to select between those two modes.
>
> > You mentioned the need to be able to select the HDR mode in a standard
> > way. Could you elaborate on the foreseen usage: would it be to select
> > SDR/HDR operation, to select between different HDR sub-modes, to inform
> > user space about HDR capability... ?
>
> Both. From a libcamera perspective, I want standardized controls for
> this, to avoid sensor-specific code as much as possible.

This sounds complicated to achieve, at least with the one existing 
V4L2_CID_HDR_SENSOR_MODE control. There are a multitude of modes and 
technologies used around HDR sensors.

A few things we can handle with existing API's:
- number of exposures (the ones that have a separated stream)
- bitdepth (by mbus code, can give a hint on compression type)

There are other aspects we need to offer user-space some controls on, in 
order to be able to select the desired mode:
-number of exposures, even when not exposed as a separate stream
-number of captures (which in some cases may be different than the 
number of exposures)
-type of gain, or number of gains (DCG, LCG, HCG)
-type: sequential/ staggered / interleaved
-type of companding (none, PWL, other types?)
-LFM indication
-LFM replacement enable/disabled
-SPD data present (Split-Pixel Detection, as LFM enhancement)

For ox03c10, the sensor hdr mode can be determined by these factors: 
number of exposures (dual/triple), staggered/unstaggered, companded (pwl 
20/16/14/12)/uncompanded, LFM/LFM+SPD/none, with all combinations 
possible. HDR data is on VC0, LFM/SPD if enabled on VC1.

For os08a20 there is staggered hdr mode or sequential hdr mode (via 
context switching, up to 4 set each group having different exposure/gain 
sets). For staggered HDR mode there are 2 possible output modes: on 
separate 2 virtual channels for long/short exposure, or on single VC 
with dummy lines.

For ov2775, besides no hdr(either HCG or LCG), there is single exposure 
hdr (DCG) or dual exposure hdr (DCG + VS). The DCG data may be combined 
or not (HCG+LCG). Compression may apply.

I'm sure there are a lot of other fancy HDR related technologies around.
Do you think this can be standardized? Up to what level of detail? I 
think most sensor drivers will only support a limited number of hdr 
modes, out of the multitude supported by the hardware. Parameters that 
may be relevant for one sensor may have no relevance for others.
What exactly is disturbing with the current approach, where each driver 
defines the hdr modes it supports, and what do you expect to have for 
libcamera?

Is this standardization talk something you would like to conclude in the 
context of multi-controls, or can it go as a separate topic? I propose 
divide-et-impera, conquer one by one ;)

Thanks,
Mirela

> > > Can you tell which sensor(s) you're working with ?
> > >
> > > > > All controls are in the same class, so they could all be set
> > > > > atomically via VIDIOC_S_EXT_CTRLS, this could turn out to be
> > > > > useful in case of sensors with context switching.
> > > >
> > > > Agreed, we should be able to set them all. Are we still unable to set
> > > > controls from multiple classes atomatically ? I thought that limitation
> > > > has been lifted.
> > > >
> > > > > Each element of the array will hold an u32 value (exposure or gain)
> > > > > for one capture. The size of the array is up to the sensor driver which
> > > > > will implement the controls and initialize them via v4l2_ctrl_new_custom().
> > > > > With this approach, the user-space will have to set valid values
> > > > > for all the captures represented in the array.
> > > >
> > > > I'll comment on the controls themselves in patch 2/2.
> > > >
> > > > > The v4l2-core only supports one scalar min/max/step value for the
> > > > > entire array, and each element is validated and adjusted to be within
> > > > > these bounds in v4l2_ctrl_type_op_validate(). The significance for the
> > > > > maximum value for the exposure control could be "the max value for the
> > > > > long exposure" or "the max value for the sum of all exposures". If none
> > > > > of these is ok, the sensor driver can adjust the values as supported and
> > > > > the user space can use the TRY operation to query the sensor for the
> > > > > minimum or maximum values.
> > > >
> > > > Hmmmm... I wonder if we would need the ability to report different
> > > > limits for different array elements. There may be over-engineering
> > > > though, my experience with libcamera is that userspace really needs
> > > > detailed information about those controls, and attempting to convey the
> > > > precise information through the kernel-userspace API is bound to fail.
> > > > That's why we implement a sensor database in libcamera, with information
> > > > about how to convert control values to real gain and exposure time.
> > > > Exposing (close to) raw register values and letting userspace handle the
> > > > rest may be better.
> > > >
> > > > > Mirela Rabulea (2):
> > > > >     LF-15161-6: media: Add exposure and gain controls for multiple
> > > > >       captures
> > > > >     LF-15161-7: Documentation: media: Describe exposure and gain controls
> > > > >       for multiple captures
> > > >
> > > > Did you forget to remove the LF-* identifiers ? :-)
> > > >
> > > > >    .../media/v4l/ext-ctrls-image-source.rst             | 12 ++++++++++++
> > > > >    drivers/media/v4l2-core/v4l2-ctrls-defs.c            |  8 ++++++++
> > > > >    include/uapi/linux/v4l2-controls.h                   |  3 +++
> > > > >    3 files changed, 23 insertions(+)
>
> --
> Regards,
>
> Laurent Pinchart