Re: [PATCH v4] media: docs-rst: Document m2m stateless video decoder interface

[Hans Verkuil]

On 4/29/19 10:48 AM, Paul Kocialkowski wrote:
> Hi,
> 
> On Mon, 2019-04-29 at 10:41 +0200, Hans Verkuil wrote:
>> On 4/27/19 2:06 PM, Nicolas Dufresne wrote:
>>> Le vendredi 26 avril 2019 Ã 16:18 +0200, Hans Verkuil a Ãcrit :
>>>> On 4/16/19 9:22 AM, Alexandre Courbot wrote:
>>>>
>>>> <snip>
>>>>
>>>>> Thanks for this great discussion. Let me try to summarize the status
>>>>> of this thread + the IRC discussion and add my own thoughts:
>>>>>
>>>>> Proper support for multiple decoding units (e.g. H.264 slices) per
>>>>> frame should not be an afterthought ; compliance to encoded formats
>>>>> depend on it, and the benefit of lower latency is a significant
>>>>> consideration for vendors.
>>>>>
>>>>> m2m, which we use for all stateless codecs, has a strong assumption
>>>>> that one OUTPUT buffer consumed results in one CAPTURE buffer being
>>>>> produced. This assumption can however be overruled: at least the venus
>>>>> driver does it to implement the stateful specification.
>>>>>
>>>>> So we need a way to specify frame boundaries when submitting encoded
>>>>> content to the driver. One request should contain a single OUTPUT
>>>>> buffer, containing a single decoding unit, but we need a way to
>>>>> specify whether the driver should directly produce a CAPTURE buffer
>>>>> from this request, or keep using the same CAPTURE buffer with
>>>>> subsequent requests.
>>>>>
>>>>> I can think of 2 ways this can be expressed:
>>>>> 1) We keep the current m2m behavior as the default (a CAPTURE buffer
>>>>> is produced), and add a flag to ask the driver to change that behavior
>>>>> and hold on the CAPTURE buffer and reuse it with the next request(s) ;
>>>>> 2) We specify that no CAPTURE buffer is produced by default, unless a
>>>>> flag asking so is specified.
>>>>>
>>>>> The flag could be specified in one of two ways:
>>>>> a) As a new v4l2_buffer.flag for the OUTPUT buffer ;
>>>>> b) As a dedicated control, either format-specific or more common to all codecs.
>>>>>
>>>>> I tend to favor 2) and b) for this, for the reason that with H.264 at
>>>>> least, user-space does not know whether a slice is the last slice of a
>>>>> frame until it starts parsing the next one, and we don't know when we
>>>>> will receive it. If we use a control to ask that a CAPTURE buffer be
>>>>> produced, we can always submit another request with only that control
>>>>> set once it is clear that the frame is complete (and not delay
>>>>> decoding meanwhile). In practice I am not that familiar with
>>>>> latency-sensitive streaming ; maybe a smart streamer would just append
>>>>> an AUD NAL unit at the end of every frame and we can thus submit the
>>>>> flag it with the last slice without further delay?
>>>>>
>>>>> An extra constraint to enforce would be that each decoding unit
>>>>> belonging to the same frame must be submitted with the same timestamp,
>>>>> otherwise the request submission would fail. We really need a
>>>>> framework to enforce all this at a higher level than individual
>>>>> drivers, once we reach an agreement I will start working on this.
>>>>>
>>>>> Formats that do not support multiple decoding units per frame would
>>>>> reject any request that does not carry the end-of-frame information.
>>>>>
>>>>> Anything missing / any further comment?
>>>>>
>>>>
>>>> After reading through this thread and a further irc discussion I now
>>>> understand the problem. I think there are several ways this can be
>>>> solved, but I think this is the easiest:
>>>>
>>>> Introduce a new V4L2_BUF_FLAG_HOLD_CAPTURE_BUFFER flag.
>>>>
>>>> If set in the OUTPUT buffer, then don't mark the CAPTURE buffer as
>>>> done after processing the OUTPUT buffer.
>>>>
>>>> If an OUTPUT buffer was queued with a different timestamp than was
>>>> used for the currently held CAPTURE buffer, then mark that CAPTURE
>>>> buffer as done before starting processing this OUTPUT buffer.
>>>
>>> Just a curiosity, can you extend on how this would be handled. If there
>>> is a number of capture buffer, these should have "no-timestamp". So I
>>> suspect we need the condition to differentiate no-timestamp from
>>> previous timestamp. What I'm unclear is to what does it mean "no-
>>> timestamp". We already stated the timestamp 0 cannot be reserved as
>>> being an unset timestamp.
>>
>> For OUTPUT buffers there is no such thing as 'no timestamp'. They always
>> have a timestamp (which may be 0). The currently active CAPTURE buffer
>> also always has a timestamp as that was copied from the first OUTPUT buffer
>> for that CAPTURE buffer.
>>
>>>> In other words, for slicing you can just always set this flag and
>>>> group the slices by the OUTPUT timestamp. If you know that you
>>>> reached the last slice of a frame, then you can optionally clear the
>>>> flag to ensure the CAPTURE buffer is marked done without having to wait
>>>> for the first slice of the next frame to arrive.
>>>>
>>>> Potential disadvantage of this approach is that this relies on the
>>>> OUTPUT timestamp to be the same for all slices of the same frame.
>>>>
>>>> Which sounds reasonable to me.
>>>>
>>>> In addition add a V4L2_BUF_CAP_SUPPORTS_HOLD_CAPTURE_BUFFER
>>>> capability to signal support for this flag.
>>>>
>>>> I think this can be fairly easily implemented in v4l2-mem2mem.c.
>>>>
>>>> In addition, this approach is not specific to codecs, it can be
>>>> used elsewhere as well (composing multiple output buffers into one
>>>> capture buffer is one use-case that comes to mind).
>>>>
>>>> Comments? Other ideas?
>>>
>>> Sounds reasonable to me. I'll read through Paul's comment now and
>>> comment if needed.
>>
>> Paul's OK with it as well. The only thing I am not 100% happy with is
>> the name of the flag. It's a very low-level name: i.e. it does what it
>> says, but it doesn't say for what purpose.
>>
>> Does anyone have any better suggestions?
> 
> Good naming is always so hard to find... I don't have anything better
> to suggest off the top of my head, but will definitely keep thinking
> about it.
> 
>> Also, who will implement this in v4l2-mem2mem? Paul, where you planning to do that?
> 
> Well, I no longer have time chunks allocated to the VPU topic at work,
> so that means I'll have to do it on spare time and it may take me a
> while to get there.
> 
> So if either one of you would like to pick it up to get it over with
> faster, feel free to do that!

OK, then I'll try to come up with something this week or next week.

Regards,

	Hans