RE: [EXT] Re: [PATCH] media: videobuf2: add V4L2_BUF_FLAG_CODECCONFIG flag

[Ming Qian]

>>From: Ming Qian
>>Sent: 2022年7月5日 9:52
>>To: Nicolas Dufresne <nicolas@xxxxxxxxxxxx>; mchehab@xxxxxxxxxx;
>>hverkuil-cisco@xxxxxxxxx
>>Cc: shawnguo@xxxxxxxxxx; robh+dt@xxxxxxxxxx; s.hauer@xxxxxxxxxxxxxx;
>>kernel@xxxxxxxxxxxxxx; festevam@xxxxxxxxx; dl-linux-imx
>><linux-imx@xxxxxxx>; linux-media@xxxxxxxxxxxxxxx;
>>linux-kernel@xxxxxxxxxxxxxxx; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
>>Subject: RE: [EXT] Re: [PATCH] media: videobuf2: add
>>V4L2_BUF_FLAG_CODECCONFIG flag
>>
>>> From: Nicolas Dufresne <nicolas@xxxxxxxxxxxx>
>>> Sent: 2022年7月4日 23:53
>>> To: Ming Qian <ming.qian@xxxxxxx>; mchehab@xxxxxxxxxx;
>>> hverkuil-cisco@xxxxxxxxx
>>> Cc: shawnguo@xxxxxxxxxx; robh+dt@xxxxxxxxxx; s.hauer@xxxxxxxxxxxxxx;
>>> kernel@xxxxxxxxxxxxxx; festevam@xxxxxxxxx; dl-linux-imx
>>> <linux-imx@xxxxxxx>; linux-media@xxxxxxxxxxxxxxx;
>>> linux-kernel@xxxxxxxxxxxxxxx; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
>>> Subject: [EXT] Re: [PATCH] media: videobuf2: add
>>> V4L2_BUF_FLAG_CODECCONFIG flag
>>>
>>> Caution: EXT Email
>>>
>>> Le mardi 28 juin 2022 à 10:19 +0800, Ming Qian a écrit :
>>> > By setting the V4L2_BUF_FLAG_CODECCONFIG flag, user-space should be
>>> > able to hint decoder the vb2 only contains codec config header, but
>>> > does not contain any frame data.
>>> > It's only used for parsing header, and can't be decoded.
>>>
>>> This is copied from OMX specification. I think we we import this, we
>>> should at least refer to the original.
>>>
>>
>>Hi Nicolas,
>>    Do you mean OMX_BUFFERFLAG_CODECCONFIG?
>>    I'm sorry that I didn't notice it before.
>>    Currently we only encounter this requirement on Android, I'm not sure if
>>it has a reference to omx.
>>    And thank you very much for pointing out it.
>>
>
> Android media stack has been based on OMX for the last decade. They are slowly moving to CODEC2 which more or less is a similar abstraction with similar ideas. Let's research prior art, so we don't screw compatibility.
>

I got it, I'll try to study the android codec2, 
and do you agree that we should add V4L2_BUF_FLAG_CODECCONFIG flag, just like OMX_BUFFERFLAG_CODECCONFIG?
Or is there any other solution that can handle this case?

>>Ming
>>
>>> >
>>> > Current, it's usually used by android.
>>> >
>>> > Signed-off-by: Ming Qian <ming.qian@xxxxxxx>
>>> > ---
>>> >  Documentation/userspace-api/media/v4l/buffer.rst | 9 +++++++++
>>> >  include/uapi/linux/videodev2.h                   | 2 ++
>>> >  2 files changed, 11 insertions(+)
>>> >
>>> > diff --git a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > index 4638ec64db00..acdc4556f4f4 100644
>>> > --- a/Documentation/userspace-api/media/v4l/buffer.rst
>>> > +++ b/Documentation/userspace-api/media/v4l/buffer.rst
>>> > @@ -607,6 +607,15 @@ Buffer Flags
>>> >       the format. Any subsequent call to the
>>> >       :ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
>>> >       but return an ``EPIPE`` error code.
>>> > +    * .. _`V4L2-BUF-FLAG-CODECCONFIG`:
>>> > +
>>> > +      - ``V4L2_BUF_FLAG_CODECCONFIG``
>>> > +      - 0x00200000
>>> > +      - This flag may be set when the buffer only contains codec config
>>> > +    header, but does not contain any frame data. Usually the codec
>>config
>>> > +    header is merged to the next idr frame, with the flag
>>> > +    ``V4L2_BUF_FLAG_KEYFRAME``, but there is still some scenes that
>>will
>>> > +    split the header and queue it separately.
>>>
>>> I think the documentation is clear. Now, if a driver uses this, will
>>> existing userland (perhaps good to check GStreamer, FFMPEG and
>>> Chromium ?) will break ?
>>> So we need existing driver to do this when flagged to, and just
>>> copy/append when the userland didn't opt-in that feature ?
>>>
>>> >      * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>>> >
>>> >        - ``V4L2_BUF_FLAG_REQUEST_FD`` diff --git
>>> > a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>>> > index 5311ac4fde35..8708ef257710
>>> > 100644
>>> > --- a/include/uapi/linux/videodev2.h
>>> > +++ b/include/uapi/linux/videodev2.h
>>> > @@ -1131,6 +1131,8 @@ static inline __u64 v4l2_timeval_to_ns(const
>>> struct timeval *tv)
>>> >  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE         0x00010000
>>> >  /* mem2mem encoder/decoder */
>>> >  #define V4L2_BUF_FLAG_LAST                   0x00100000
>>> > +/* Buffer only contains codec header */
>>> > +#define V4L2_BUF_FLAG_CODECCONFIG            0x00200000
>>> >  /* request_fd is valid */
>>> >  #define V4L2_BUF_FLAG_REQUEST_FD             0x00800000
>>> >