Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jacopo Mondi - Complex cameras are complex

46 views

Published on

The ‘complex camera’ user-space library is a fairly new and hot topic in the Linux kernel video community. The issue is debated and targets a real technical debt of Linux systems.

The Linux kernel video input subsystem and its APIs changed greatly in the last years to keep up with the increasing complexity and processing power embedded in modern SoCs. Namely, the biggest game-changing feature introduced already 5 years ago is the media controller subsystem and its pad oriented APIs, that allows composing pipelines of processing blocks to model the acquisition and re-processing of video and images.

While most of the media subsystem drivers in mainline Linux have been ported to fully support media-controller, the real missing component is now user space support to automate setting up and controlling the image processing pipelines.

The Video4Linux community is now tackling the issue by implementing support for “complex camera” systems by providing a user space library that aims to support the most recent use cases represented by mobile consumer devices and high end laptops.

This talks provides an overview of the in-kernel APIs of modern media-controller capable video device drivers, their userspace APIs and the challenges the currently in-development library has to face.

It also aims to provide to both driver and application developers an overview of the most modern implementation of the Linux video input stack architecture, that will likely be found in most system in next years.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Jacopo Mondi - Complex cameras are complex

  1. 1. Complex Cameras are complex Jacopo Mondi Linux Lab 2018 Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (1/ 65)
  2. 2. Hello, I’m Jacopo Embedded Linux software engineer I mostly work on integrating cameras and multimedia devices I (mostly) work with Renesas Electronics mainline kernel team jacopo@jmondi.org irc: jmondi freenode.net Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (1/ 65)
  3. 3. Motivations Rapidly increasing complexity of image capture process is moving from the embedded and mobile space to the more canonical personal computing space (say, laptops). Are Linux systems ready for this? Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (2/ 65)
  4. 4. Motivations Rapidly increasing complexity of image capture process is moving from the embedded and mobile space to the more canonical personal computing space (say... laptops). Are Linux systems ready for this? No they’re not (I wouldn’t be giving this presentation otherwise :p ) Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (3/ 65)
  5. 5. Complex Cameras are complex Image acquisition is complex Image capture Smart sensors Smart(er) ISPs The media controller The media graph Handling pipelines Intel’s IPU3 Libcamera Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (4/ 65)
  6. 6. Image Capture The image acquisition process is a pipeline of digital signals processing blocks, that spans from the application processor to external components connected through a data and a control bus Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (5/ 65)
  7. 7. Image Capture: Sensors Sensors grid of photo-detectors (pixels) measure intensity of photo-electric properties filters: each pixel transports a single color information Red Green Blue Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (6/ 65)
  8. 8. Image Capture: Raw Bayer Bayer Patterns disposition of color filters BGGR - RGBG - GRBG - RGGB 50% Green - 25% Red - 25% Blue Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (7/ 65)
  9. 9. Image Capture: Raw Bayer Bayer Patterns disposition of color filters BGGR - RGBG - GRBG - RGGB 50% Green - 25% Red - 25% Blues The full pixel color has to be reconstructed demosaicing or de-bayering Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (8/ 65)
  10. 10. Image Capture: Raw Bayer It’s a tough job! interpolation of colors from neighbour pixels computational intensive problems Figure: From: http://www.cambridgeincolour.com Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (9/ 65)
  11. 11. Image Capture: Color Encoding The RAW digital values have to be interpreted and encoded RAW usually converted to commonly used colorspaces as sRGB, Adobe RGB, CYMK from a known colorspace, pixels get color encoded to ease transportation media bus formats: RGB565, RGB666, YUYV, VYUY easier to transport, store and manipulate More computations! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (10/ 65)
  12. 12. Image Capture: Feedback loop What about 3A, black level calibration, enhancements etc ? feedback loop on captured images even more computationally expensive often based on user application inputs Application SoC Sensor Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (11/ 65)
  13. 13. Image Capture: Feedback loop What about 3A, black level calibration, enhancements etc ? feedback loop on captured images even more computationally expensive often based on user application inputs Application SoC Sensor Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (12/ 65)
  14. 14. Image Capture: Feedback loop What about 3A, black level calibration, enhancements etc ? feedback loop on captured images even more computationally expensive often based on user application inputs Application SoC Sensor Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (13/ 65)
  15. 15. Image Capture: Feedback loop What about 3A, black level calibration, enhancements etc ? feedback loop on captured images even more computationally expensive often based on user application inputs Application SoC Sensor Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (14/ 65)
  16. 16. Image Capture: Feedback loop What about 3A, black level calibration, enhancements etc ? feedback loop on captured images even more computationally expensive often based on user application inputs Application SoC Sensor Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (15/ 65)
  17. 17. Image Capture: Summary Digital imaging is a computational expensive process! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (16/ 65)
  18. 18. Image Capture: Summary Digital imaging is a computational expensive process! complexity in the image acquisition de-bayering, enhancing, pixel encoding Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (17/ 65)
  19. 19. Image Capture: Summary Digital imaging is a computational expensive process! complexity in the image acquisition de-bayering, enhancing, pixel encoding complexity in image quality tuning 3A parameters computation, tuning Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (18/ 65)
  20. 20. Image Capture: Summary Digital imaging is a computational expensive process! complexity in the image acquisition de-bayering, enhancing, pixel encoding complexity in image quality tuning 3A parameters computation, tuning strict correlation with application requests feedback loop Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (19/ 65)
  21. 21. Image Capture: Summary Digital imaging is a computational expensive process! complexity in the image acquisition de-bayering, enanching, pixel encoding complexity in image quality tuning 3A parameters computation, tuning strict correlation with application requests feedback loop don’t forget we’re operating on -big- buffers 1080p in YUYV format is 4MB of data Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (20/ 65)
  22. 22. Image Capture: Architecture SoC (fat) CPUs ISP Sensor RAM DMA I2c control image data CSI-2 Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (21/ 65)
  23. 23. Smart sensors In the pre-mobile computing era, embedded systems where: much less powerful Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (22/ 65)
  24. 24. Smart sensors In the pre-mobile computing era, embedded systems where: much less powerful much more proprietary Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (23/ 65)
  25. 25. Smart sensors In the pre-mobile computing era, embedded systems where: much less powerful much more proprietary much less standardized Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (24/ 65)
  26. 26. Smart sensors SoC (fat) CPUs ISP Sensor RAM DMA I2c control image data CSI-2 Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (25/ 65)
  27. 27. Smart sensors Let the sensor do the computations! complex sensor architectures -> complex drivers Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (26/ 65)
  28. 28. Smart sensors Let the sensor do the computations! complex sensor architectures -> complex drivers complex sensor architectures -> expensive sensors Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (27/ 65)
  29. 29. Smart sensors Let the sensor do the computations! complex sensor architectures -> complex drivers complex sensor architectures -> expensive sensors complex sensor architectures -> power hungry sensors Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (28/ 65)
  30. 30. Smart sensors Let the sensor do the computations! complex sensor architectures -> complex drivers complex sensor architectures -> expensive sensors complex sensor architectures -> power hungry sensors more computing blocks on the sensor = less space for picture quality Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (29/ 65)
  31. 31. Smart(er) ISPs Image Signal Processor system peripheral that operates on the digital images usually features configurable processing blocks each block performs a specific operation, including: Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (30/ 65)
  32. 32. Smart(er) ISPs Image Signal Processor system peripheral that operates on the digital images usually features configurable processing blocks each block perform a specific operation, including: memory transfers Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (31/ 65)
  33. 33. Smart(er) ISPs Image Signal Processor system peripheral that operates on the digital images usually features configurable processing blocks each block perform a specific operation, including: memory transfers image manipulation Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (32/ 65)
  34. 34. Smart(er) ISPs Image Signal Processor system peripheral that operates on the digital images usually features configurable processing blocks each block perform a specific operation, including: memory transfers image manipulation format conversions Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (33/ 65)
  35. 35. Smart(er) ISPs Image Signal Processor system peripheral that operates on the digital images usually features configurable processing blocks each block perform a specific operations, including: memory transfers image manipulation format conversions the design is very platform specific Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (34/ 65)
  36. 36. Smart(er) ISPs Image Signal Processor fast interconnections with very fast CPUs interfaces with (a lot of) system memory (possibly) interfaced with other system peripherals as GPUs, DRM, codecs etc. Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (35/ 65)
  37. 37. Smart(er) ISPs Image Signal Processor it comes with your SoC silicon vendors are competing on this Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (36/ 65)
  38. 38. Smart(er) ISPs Image Signal Processor it comes with your SoC silicon vendors are competing on this So, yes, we should really use ISPs as much as we could Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (37/ 65)
  39. 39. Smart(er) ISPs, and somewhat in the middle :/ Image Signal Processor, not on the SoC some silicon vendors de-couple SoC and ISP.. off load image manipulation to tiny co-processors and build stacks of crazy complex software components to deal with that Is this still a thing? Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (38/ 65)
  40. 40. Media Controller The V4L2 APIs are designed for ’simple’ device basically, DMA engines that transfers ’untouched’ images i2creceiver sensor HARDWARE pixel data i2c messages KERNEL receiver driver sensor driver V4L2 /dev/video0 system memory Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (39/ 65)
  41. 41. Media Controller Support for more complex devices, as ISPs the media controller exposes ISP processing blocks to userspace i2c receiver sensor HARDWAREpixel data i2c messages resizer converter formatter /dev/v4l-subdev-1 /dev/video0 /dev/v4l-subdev-2 /dev/v4l-subdev-4 /dev/v4l-subdev-5 system memory receiver driver isp driver pixel formatter KERNEL Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (40/ 65)
  42. 42. Media Controller Support for more complex devices, as ISPs the media graph is controlled through /dev/mediaX i2c receiver sensor HARDWAREpixel data i2c messages resizer converter formatter /dev/v4l-subdev-1 /dev/video0 /dev/v4l-subdev-2 /dev/v4l-subdev-4 /dev/v4l-subdev-5 /dev/media0 system memory receiver driver isp driver pixel formatter KERNEL Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (41/ 65)
  43. 43. Media Controller Nice, but.... subdevices have to be linked to form a pipeline MEDIA_IOC_SETUP_LINK ioctl media−ctl −d /dev/media0 −l ” ’ rcar_csi2 feaa0000 . csi2 ’ : 1 −> ’VIN2 output ’ : 0 [ 1 ] ” Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (42/ 65)
  44. 44. Media Controller Nice, but.... subdevices have to be configured properly VIDIOC_SUBDEV_S_FMT ioctl media−ctl −d /dev/media0 −V ” ’ rcar_csi2 feaa0000 . csi2 ’ : 1 [fmt : RGB888_1X24/640x480 field : none] ” Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (43/ 65)
  45. 45. Media Controller Nice, but.... the system boots in unusable state applications are requested to know a lot of details on the underlying hardware triggering capture operations requires precise setup of the whole pipeline Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (44/ 65)
  46. 46. Media Controller Nice, but.... the system boots in unusable state applications are requested to know a lot of details on the underlying hardware triggering capture operations requires setup of the whole pipeline This is (sort of) acceptable for embedded systems Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (45/ 65)
  47. 47. Media Controller Nice, but.... the system boots in unusable state applications are requested to know a lot of details on the underlying hardware triggering capture operations requires setup of the whole pipeline This is (sort of) acceptable for embedded systems Is it for consumer laptops? Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (46/ 65)
  48. 48. Media Controller - now on laptops! Until today, cameras in laptop have usually been USB cameras super-smart cameras: they talk USB! fit well in the single device node abstraction platform agnostic solution controlled through the UVC control protocol Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (47/ 65)
  49. 49. Media Controller - now on laptops! Intel IPU3 Intel is now shipping processors with powerful ISPs (some) KabyLake and (some) SkyLake platforms have IPU3 ISP patches on linux-media and on their way to mainline: yey! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (48/ 65)
  50. 50. Media Controller - now on laptops! (simplified) IPU3 media graph CSI-2 CSI-2 capture media0 media1 input params viewfinder output stat imgu Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (49/ 65)
  51. 51. Media Controller - now on laptops! (simplified) IPU3 media graph CSI-2 CSI-2 capture media0 media1 input params viewfinder output stat imgu RAW Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (50/ 65)
  52. 52. Media Controller - now on laptops! (simplified) IPU3 media graph CSI-2 CSI-2 capture media0 media1 input params viewfinder output stat imgu RAW RAW Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (51/ 65)
  53. 53. Media Controller - now on laptops! (simplified) IPU3 media graph CSI-2 CSI-2 capture media0 media1 input params viewfinder output stat imgu RAW RAW Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (52/ 65)
  54. 54. Media Controller - now on laptops! (simplified) IPU3 media graph CSI-2 CSI-2 capture media0 media1 input params viewfinder output stat imgu RAW RAW Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (53/ 65)
  55. 55. Media Controller - now on laptops! (simplified) IPU3 media graph CSI-2 CSI-2 capture media0 media1 input params viewfinder output stat imgu RAW RAW YUV YUV meta Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (54/ 65)
  56. 56. Media Controller - now on laptops! (simplified) IPU3 media graph CSI-2 CSI-2 capture media0 media1 input params viewfinder output stat imgu RAW 3A - tuning meta Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (55/ 65)
  57. 57. Media Controller - now on laptops! ”Re: Webcams not recognized on a Dell Latitude 5285 laptop” https://www.spinics.net/lists/linux-media/msg131388.html This laptop embeds one of these new ”complex” cameras from Intel. They requires IPU3 driver. Though, unlike traditional webcam, you need special userspace to use it (there is no embedded firmware to manage focus, whitebalance, etc, userspace code need to read the stats and manage that). As of now, there is no good plan on how to support this in userspace. Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (56/ 65)
  58. 58. Libcamera! V4L2 community joins forces with the industry to provide a full camera stack for Linux-based systems libcamera.org user-space camera stack designed to work with complex cameras aims to be compatible with V4L2 applications, Android and ChromeOs free software - open development - safe space for vendor IP Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (57/ 65)
  59. 59. Media Controller - now on laptops! We already have devices on the market using IPU3 for cameras Linux distributions won’t support cameras on those platforms :( vendors of Linux based systems (say, Android and ChromeOS) will keep using their in-house solutions Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (58/ 65)
  60. 60. Media Controller - now on laptops! We already have devices on the market using IPU3 for cameras Linux distributions won’t support cameras on that platforms :( vendors of Linux based systems (say, Android and ChromeOS) will keep using their in-house solutions Don’t despair though! We might have a solution! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (59/ 65)
  61. 61. Libcamera! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (60/ 65)
  62. 62. Libcamera! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (61/ 65)
  63. 63. Libcamera! Please join! read the doc: http://libcamera.org/docs.htm read patches: git://linuxtv.org/libcamera.git have a chat: #libcamera on freenode we welcome inputs from anyone working with cameras! please help! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (62/ 65)
  64. 64. Complex Cameras are debated ELC-E: Why Embedded Cameras are Difficult, and How to Make Them Easy - Laurent Pinchart, Ideas on Board ELC-E: Complex Cameras on Linux - Mauro Carvalho Chehab, Samsung Meeting notes on the Tokyo complex cameras workshop: https://linuxtv.org/news.php?entry=2018-07-13.mchehab Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (63/ 65)
  65. 65. Complex Cameras Thanks! Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (64/ 65)
  66. 66. Complex Cameras Thanks! Questions? Jacopo Mondi - Linux Lab 2018 Complex Cameras are complex (65/ 65)

×