Re: [RFC PATCH v3] media: docs-rst: Document m2m stateless video decoder interface

From: Alexandre Courbot
Date: Mon Oct 22 2018 - 02:05:18 EST


On Fri, Oct 19, 2018 at 5:44 PM Hans Verkuil <hverkuil@xxxxxxxxx> wrote:
>
> On 10/19/18 10:09, Alexandre Courbot wrote:
> > Thanks everyone for the feedback on v2! I have not replied to all the
> > individual emails but hope this v3 will address some of the problems
> > raised and become a continuation point for the topics still in
> > discussion (probably during the ELCE Media Summit).
> >
> > This patch documents the protocol that user-space should follow when
> > communicating with stateless video decoders. It is based on the
> > following references:
> >
> > * The current protocol used by Chromium (converted from config store to
> > request API)
> >
> > * The submitted Cedrus VPU driver
> >
> > As such, some things may not be entirely consistent with the current
> > state of drivers, so it would be great if all stakeholders could point
> > out these inconsistencies. :)
> >
> > This patch is supposed to be applied on top of the Request API V18 as
> > well as the memory-to-memory video decoder interface series by Tomasz
> > Figa.
> >
> > Changes since v2:
> >
> > * Specify that the frame header controls should be set prior to
> > enumerating the CAPTURE queue, instead of the profile which as Paul
> > and Tomasz pointed out is not enough to know which raw formats will be
> > usable.
> > * Change V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAM to
> > V4L2_CID_MPEG_VIDEO_H264_SLICE_PARAMS.
> > * Various rewording and rephrasing
> >
> > Two points being currently discussed have not been changed in this
> > revision due to lack of better idea. Of course this is open to change:
> >
> > * The restriction of having to send full frames for each input buffer is
> > kept as-is. As Hans pointed, we currently have a hard limit of 32
> > buffers per queue, and it may be non-trivial to lift. Also some codecs
> > (at least Venus AFAIK) do have this restriction in hardware, so unless
> > we want to do some buffer-rearranging in-kernel, it is probably better
> > to keep the default behavior as-is. Finally, relaxing the rule should
> > be easy enough if we add one extra control to query whether the
> > hardware can work with slice units, as opposed to frame units.
>
> Makes sense, as long as the restriction can be lifted in the future.

Lifting this limitation once we support more than 32 buffers should
not be an issue. Just add a new capability control and process things
in slice units. Right now we have hardware that can only work with
whole frames (venus) but I suspect that some slice-only hardware must
exist, so it may actually become a necessity at some point (lest
drivers do some splitting themselves).

>
> > * The other hot topic is the use of capture buffer indexes in order to
> > reference frames. I understand the concerns, but I doesn't seem like
> > we have come with a better proposal so far - and since capture buffers
> > are essentially well, frames, using their buffer index to directly
> > reference them doesn't sound too inappropriate to me. There is also
> > the restriction that drivers must return capture buffers in queue
> > order. Do we have any concrete example where this scenario would not
> > work?
>
> I'll start a separate discussion thread for this to avoid polluting the
> review of this documentation.

Thanks! Following up there.