[RFC v5 00/11] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@xxxxxxxxxxxxx>

Hi,

After the comments received in the last patchset[1] and 
during the media summit [2] here is the new and improved version
of the patchset. The implementation is simpler, smaller and cover
a lot more cases.

If you look to the last patchset I got rid of a few things, the main
one is the OUT_FENCE event, one thing that we decided in Prague was
that, when using fences, we would keep ordering of all buffers queued
to vb2. That means they would be queued to the drivers in the same order
that the QBUF calls happen, just like it already happens when not using
fences. Fences can signal in whatever order, so we need this guarantee
here. Drivers can, however, not keep ordering when processing the
buffers.

But there is one conclusion of that that we didn't reached at the
summit, maybe because of the order we discussed things, and that is: we do
not need the OUT_FENCE event anymore, because now at the QBUF call time
we *always* know the order in which the buffers will be queued to the
v4l2 driver. So the out-fence fd is now returned using the fence_fd
field as a return argument, thus the event is not necessary anymore.

The fence_fd field is now used to comunicate both in-fences and
out-fences, just like we do for GPU drivers. We pass in-fences as input
arguments and get out-fences as return arguments on the QBUF call.
The approach is documented.

I also added a capability flag, V4L2_CAP_ORDERED, to tell userspace if
the v4l2 drivers keep the buffers ordered or not. 

We still have the 'ordered_in_driver' property for queues, but its
meaning has changed. When set videobuf2 will know that the driver can
keep the order of the buffers, thus videobuf2 can use the same fence
context for all out-fences. Fences inside the same context should signal
in order, so 'ordered_in_driver' is a optimization for that case.
When not set, a context for each out-fence is created.

So now explicit synchronization also works for drivers that do not keep
the ordering of buffers.

Another thing is that we do not allow videobuf2 to requeue buffers
internally when using fences, they have a fence associated to it and
we need to finish the job on them, i.e., signal the fence, even if an
error happened.

The rest of the changes are documented in each patch separated.

There a test app at:

https://gitlab.collabora.com/padovan/v4l2-fences-test

Among my next steps is to create a v4l2->drm test app using fences as a
PoC, and also look into how to support it in ChromeOS.

Open Questions
--------------

* Do drivers reorder buffers internally? How to handle that with fences?

* How to handle audio/video syncronization? Fences aren't enough, we  need 
  to know things like the start of capture timestamp.

Regards,

Gustavo
--

[1] https://www.mail-archive.com/linux-kernel@xxxxxxxxxxxxxxx/msg1518928.html
[2] http://muistio.tieke.fi/p/media-summit-2017

Gustavo Padovan (10):
  [media] v4l: add V4L2_CAP_ORDERED to the uapi
  [media] vivid: add the V4L2_CAP_ORDERED capability
  [media] vb2: add 'ordered_in_driver' property to queues
  [media] vivid: mark vivid queues as ordered_in_driver
  [media] vb2: check earlier if stream can be started
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add infrastructure to support out-fences
  [media] vb2: add out-fence support to QBUF
  [media] v4l: Document explicit synchronization behavior

Javier Martinez Canillas (1):
  [media] vb2: add videobuf2 dma-buf fence helpers

 Documentation/media/uapi/v4l/buffer.rst          |  15 ++
 Documentation/media/uapi/v4l/vidioc-qbuf.rst     |  42 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst |   9 +-
 Documentation/media/uapi/v4l/vidioc-querycap.rst |   3 +
 drivers/media/platform/vivid/vivid-core.c        |  24 +-
 drivers/media/usb/cpia2/cpia2_v4l.c              |   2 +-
 drivers/media/v4l2-core/Kconfig                  |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c    |   4 +-
 drivers/media/v4l2-core/videobuf2-core.c         | 274 +++++++++++++++++++++--
 drivers/media/v4l2-core/videobuf2-v4l2.c         |  48 +++-
 include/media/videobuf2-core.h                   |  44 +++-
 include/media/videobuf2-fence.h                  |  48 ++++
 include/uapi/linux/videodev2.h                   |   8 +-
 13 files changed, 485 insertions(+), 37 deletions(-)
 create mode 100644 include/media/videobuf2-fence.h

-- 
2.13.6