[Bug 1921474] Re: Support client usage notification via V4l2 Event API

You-Sheng Yang 1921474 at bugs.launchpad.net
Mon Jan 10 10:52:07 UTC 2022 

Hi,

On Wed, Jan 5, 2022 at 8:35 PM Robie Basak <1921474 at bugs.launchpad.net> wrote:
>
> > On a MIPI camera through Intel IPU6 platform that its raw V4L2
> loopback interface is preserved for Intel Camera HAL libraries, a relay
> daemon + v4l2loopback is used to allow the usage of legacy V4L2 based
> apps.
>
> Why is this camera's raw V4L2 loopback interface being "preserved for
> Intel Camera HAL libraries"?

The Intel IPU camera provides only raw image access through kernel
V4L2 interface, and one will need the corresponding userspace
middleware to manipulate the image/video stream. For IPU6 that is
being used on a certain oem platforms, following components are
needed:

* https://github.com/intel/ipu6-camera-bins: prebuilt proprietary
binaries and firmware blobs,
* https://github.com/intel/ipu6-camera-hal: camera hardware
abstraction layer library,
* https://github.com/intel/icamerasrc: gstreamer element plugin.

So these components will allow a camera app, with the knowledge of
icamerasrc, to retrieve normal image/video streams. For all existing,
legacy, v4l2-only camera apps, they will not be able to find
/dev/video* as they don't exist at all. And even there are, they will
still need the above components to manipulate the image/video for
correct presentation.

In order to provide best backward compatibility, a v4l2-relayd is used
to open hardware camera via gstreamer icamerasrc element and redirect
the stream back into kernel v4l2loopback device, then a general,
innocent userspace camera app may directly use the video device
provided by v4l2loopback.

All the components can be found in
https://launchpad.net/~oem-solutions-group/+archive/ubuntu/intel-ipu6,
and are being proposed for inclusion to Ubuntu archive at. the same
time

> Is this a requirement for hardware
> enablement of this device (and if so, why?), or merely a userspace
> software rearrangement?

Mostly in the userspace, but we'll need v4l2loopback as a way to
redirect processed video/image streams.

> What functionality won't work if Ubuntu users on
> Focal just use the V4L2 interface without using the "Intel Camera HAL
> libraries"?

This SRU is only for adding V4L2 Event APIs and a few coverity fixes,
so it doesn't really cause functionality changes.

However, it's true that v4l2-relayd now installs a
/etc/modprobe.d/v4l2-relayd.conf file that specifies module parameters
for v4l2loopback. So for Ubuntu users without v4l2-relayd installed,
there is no visible difference.

> What package provides the "Intel Camera HAL libraries"?

Listed above.

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1921474

Title:
  Support client usage notification via V4l2 Event API

Status in HWE Next:
  In Progress
Status in OEM Priority Project:
  In Progress
Status in v4l2loopback package in Ubuntu:
  Fix Released
Status in v4l2loopback source package in Focal:
  In Progress
Status in v4l2loopback source package in Hirsute:
  Fix Released

Bug description:
  [SRU Justification]

  [Impact]

  On a MIPI camera through Intel IPU6 platform that its raw V4L2
  loopback interface is preserved for Intel Camera HAL libraries, a
  relay daemon + v4l2loopback is used to allow the usage of legacy V4L2
  based apps.

  By design, the relayd will open v4l2sink to v4l2loopback OUTPUT
  deivce, and it will only open libcamhal based GStreamer source
  element, and therefore underlying camera hardware, until received new
  client notifications via V4L2 Event API that is introduced in this
  SRU.

  Besides, frame sizes/intervals enumeration is also fixed to meet
  better compliance with user apps.

  [Test Plan]

  v4l2loopback doesn't support V4L2 Event API until recently, so
  requests for the usage will always fail:

    struct v4l2_event_subscription sub;
    int fd;

    memset (&sub, 0, sizeof (sub));
    sub.type = ...;
    if (ioctl (fd, VIDIOC_SUBSCRIBE_EVENT, &sub) == 0) // fail

  With this fix, it shall support Event API operations, e.g.
  VIDIOC_SUBSCRIBE_EVENT, VIDIOC_UNSUBSCRIBE_EVENT, VIDIOC_DQEVENT.

  [Where problems could occur]

  While a custom type of V4L2 event is now registered for OEM projects,
  software expected same ID (numerically equivalent to
  V4L2_EVENT_PRIVATE_START) may get confused. While this falls in the
  private usage section, it's not supposed to be used in general, and
  when it does, it should be under some presumptions, e.g. selected
  hardware, so it's unlikely to happen on OEM projects.

  For generic Ubuntu, programs may begin to take advantage of this new
  capability to update its UI, or to take other actions after receiving
  desired events. There might be behavior changes, but should be under
  the original design if was done carefully.

  [Other Info]

  For Focal backports, 0.12.5-1 is equivalent to 0.12.3-1ubuntu0.3 plus
  micro version updates, so there should be little risk backport a new
  release, and we can drop additional patches carried. This implies bug
  1905613.

  This focal backport includes changes for bug 1921474, bug 1930208, and
  bug 1936250, so it will then become synced with Impish and newer
  again. The 0007-compliance-stop-
  declaring-V4L2_CAP_VIDEO_M2M-capabil.patch carried in bug 1930208 is
  deliberately skipped to avoid bug 1946660.

  ========= original bug report ==========

  In addition to kernel/firmware proposed in bug 1921345, several user
  space daemons/libraries, as well as fixes/features in v4l2loopback-
  dkms, are also required to have seamless support for legacy/existing
  Linux kernel V4L2 API based applications to adopt libcamera or Intel
  libcamhal provided camera interfaces.

  The idea is to add a v4l2 streaming relay daemon (currently developed
  in [1] and packaged in [2]) that helps redirecting V4L2 buffer streams
  into v4l2loopback output device, and then legacy apps open
  v4l2loopback capture device in the ordinary way.

    hw -> libcamera/libcamhal -> v4l2-relayd -> v4l2loopback ->
  v4l2-based apps

  To achieve this, we'd like v4l2-relayd to only open GStreamer
  icamerasrc pipeline (provided by [3],[4]) when there is actual usage
  request from v4l2-based apps for privacy and power consumption's
  concerns. However, current (0.12.5 and therefore Ubuntu/Debian
  0.12.5-1) doesn't have any available mechanism for this, therefore a
  V4L2 Event API based proposal has been sent and accepted by
  upstream[5][6].

  This is needed for Ubuntu OEM projects with MIPI cameras through Intel
  IPU6 (Imaging Processing Unit version 6).

  [1]: https://gitlab.com/vicamo/v4l2-relayd
  [2]: https://code.launchpad.net/~oem-solutions-engineers/v4l2-relayd/+git/packaging
  [3]: https://github.com/intel/ipu6-camera-hal
  [4]: https://github.com/intel/icamerasrc
  [5]: https://github.com/umlaeute/v4l2loopback/pull/345
  [6]: https://github.com/umlaeute/v4l2loopback/pull/352

To manage notifications about this bug go to:
https://bugs.launchpad.net/hwe-next/+bug/1921474/+subscriptions

More information about the Ubuntu-sponsors mailing list