Camera Architecture – Options for Success
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 2
Tizen-IVI offers the whole worl...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 3
As with any camera related proj...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 4
It is important to introduce po...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 5
At a low level, some of availab...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 6
• Analog video capture board wh...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 7
• USB cameras to support requir...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 8
• Mitigations for USB bandwidth...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 9
In context of Tizen-IVI, let’s ...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 10
General architecture and types...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 11
• WebKit in this view is a san...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 12
• Underlying Middleware ( Proc...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 13
• Could be a Linux Video devic...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 14
• Webkit: video tag. Does buff...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 15
• WRT plug-in: Combination wit...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 16
• GStreamer: Complex, but powe...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 17
• FFMPEG: Simpler than GStream...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 18
Let’s try to assemble combinat...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 19
• 360 degree stitched view to ...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 20
• In car camera to control tax...
Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 21
Questions?
Last slide 
Upcoming SlideShare
Loading in...5
×

Camera Architecture from Failure to Success

306

Published on

Camera Architecture from Failure to Success - Symphony Teleca

Published in: Technology, Art & Photos
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
306
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
5
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Camera Architecture from Failure to Success

  1. 1. Camera Architecture – Options for Success
  2. 2. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 2 Tizen-IVI offers the whole world of HTML5/CSS/JS solutions from a UI point of view, however many of the features require developers to step out of the WRT for native support. The Camera related use case is a good example where JS alone may not be enough to build whole SW stack. Considering available technologies, and the specifics of the use cases, there might be several ways to implement Camera support. Within STC we did some exploration of the options, and would like to present details of this investigation. We hope this knowledge sharing will be helpful to other developers. We will briefly go through possible HW configurations and related use cases before really stepping into the SW side. Then we will introduce SW stack details and the possible solutions as they relate to the configurations and use cases. Introduction
  3. 3. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 3 As with any camera related project it is important to pay attention to the following key requirements: • Desired FPS • Video Resolution • Video Lag • Distance of camera(s) from the processing unit • Number of cameras to connect • Number of cameras to display simultaneously • Time to switch camera view if user allowed to select camera from the list of inactive devices • Performance of CPU or assisting en(de)coding modules Considering the level of abstraction of Tizen-IVI applications with respect to Native solutions, it is extremely important to fix these requirements before the HW selection step. We will review some of the implications in the next slides. Typical requirements to consider
  4. 4. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 4 It is important to introduce possible types of HW first, as use cases may split in its requirements to particular HW combinations. We could recommend first to focus on these three groups of HW in selecting particular combinations of HW for the setup. • Camera HW which could be USB, Analog and IP ones. • Host system main board offering USB root hub (one or many, USB2.0 or USB3.0), a means of connecting an Analog video capture board, and one or more Network interfaces. • Analog video capture board offering single or multiple channel support, together with a video driver which may or may not perform channel multiplexing ( mapping of multiple channels onto a single Linux video device ). Types of HW
  5. 5. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 5 At a low level, some of available options will limit possible use cases. Typical points to consider: • IP cameras to support required quality of the video (frame size/fps) with MJPEG over TCP/IP and/or H.264 over RTSP/RTP depending on particular needs.  There is not too much HW related specific in this option if you have TCP connection to the camera working along with UDP packets going well and you satisfied with the quality versus performance of the network and host system. Why it is important to care about particular HW set up?
  6. 6. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 6 • Analog video capture board which supports non-multiplexed Linux video devices.  Many cheap and small miniPCI Express cards support many input channels, but their drivers expose only one Linux video device, and only one channel sends frames to the interface at one time. Hence multichannel support requires rapid switching between channels. To minimize problems on the SW side, it is preferable to use a capture board whose driver supports as many Linux video devices as required, ideally one per camera. Another option would be several video capture boards, if allowed by the number of MiniPCI Express ports on the mainboard.  Cards with higher total FPS should be preferred. If channel multiplexing is required, then higher FPS would help to minimize time lost ( about 3-4 frames ) due to tuning phase after each channel switch. It’s important to distinguish between per-channel FPS, and the total FPS supported by the video chip/card.  Card manufacturers have a habit of not keeping their drivers up to date. Check that you can build your card’s driver using the particular kernel used on the target system. Why it is important to care about particular HW set up? ( cont. )
  7. 7. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 7 • USB cameras to support required quality of the video with acceptable USB bandwidth considering other devices connected to same USB Root Hub at the host system mainboard.  Feasibility of multiple USB camera solutions is determined by single camera USB bandwidth allocation for acceptable quality video, the number of cameras needed, and the number of USB Root Hubs.  The main issue here is the difficulty in knowing how much USB bandwidth will be allocated by a Hub for a given camera – it is not a simple function of video bandwidth, but depends on options presented to the USB driver by the device.  Typical USB Bandwidth allocations to expect for different camera profiles are from 20% up to 60%+. 20% of bus bandwidth is reserved for non-isochronous devices. Hence this might be a limiting factor in multiple camera projects.  Another complication is that you would never find this USB bandwidth usage written on the box. Even more, it might depend to particular Firmware version. So, you would need to be in touch with manufacturer or perform hands-on experiments with sample cameras. Why it is important to care about particular HW set up? ( cont. )
  8. 8. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 8 • Mitigations for USB bandwidth issues  Consider requiring USB 3.0 support in Root Hub and Camera as this might relax some of the limits around USB bandwidth.  Consider H.264 support in the camera and availability of the SW options to deal with such interfaces. Why it is important to care about particular HW set up? ( cont. )
  9. 9. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 9 In context of Tizen-IVI, let’s assume we are always talking about Web Applications presenting a video stream in WebKit or Crosswalk browser. This is the ‘Tizen IVI way’. Typical use cases and the most important requirements to consider might be: • Slow motion camera - Intermediate quality of the video, no requirements regarding lag • Rear view on the display - Minimal lag, high resolution, flipped frames • 360 degree stitched view to present road around the car - Multiple cameras setup, correction of lens distortion, stitching of camera outputs into single panoramic video, minimal lag • Visualization of the driving details and some camera view merged on windshield - Merge of the dynamic/static data with actual camera view, minimal lag, medium FPS. • In car camera to view rear passenger area - Multiple cameras setup, medium or low FPS, medium lag. • One by one frame processing before display in various recognition cases such as faces, road signs or other objects recognition – low lag is preferred, but may not be critical ( depends on circumstance ), medium FPS as the recognition may take longer than single frame release time, no en(de)coding artifacts to achieve best recognition results Typical use cases
  10. 10. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 10 General architecture and types of SW Source Weston Underlying Middleware System/Kernel Services Kernel Card Drivers WebKit WRT Application WRT Plug-in JS API Wayland client D-BUS API Variety of Weston plug-ins, shell, compositor ProcessingPresentation A general architecture can be presented in the following way, where some of the layers can be excluded for particular cases.
  11. 11. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 11 • WebKit in this view is a sandbox where the app is running  WRT Application is a combination of HTML/CSS/JS technologies, hence it can deal with the video frames with help of JS or leave it to the functions available in WebKit core. So, it can be Video tag, Pure Canvas, WebGL or JS MPEG-4 decoder.  WRT Plug-in primary usage is control (over D-BUS or similar interface ) and secondary is bypass interface to the source of the video data (Underlying Middleware implementing access to particular System/Kernel services) With respect to webkit there are few more options to deal with particular camera options such as:  Combination of webkitGetUserMedia and createObjectURL to get blob object referencing the stream taken from V4l2, hence applicable to USB and Video Capture Board options  Make your own Blob object with help of JS and WRT Plug-in and feed it to video tag like the one from the above option • Weston plug-ins might be one of the options if you want to make overlay client where the WRT Application only controls the way and time video is presented. Video frame processing is then done outside WRT sandbox. Presentation layer
  12. 12. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 12 • Underlying Middleware ( Processing layer ) is a combination of several components:  Tizen components such as GStreamer (or FFMPEG) to unify interface to the various video streaming devices. NB This is distinct from GStreamer as used directly inside Webkit.  Custom SW components to control parameters of System components  Or Custom SW components to deal with the device interfaces directly, considering specific details of each video streaming device • What is the purpose of Processing layer group? Essentially it is an abstraction layer that hides details of the various types of processing required by different video sources before that source is exposed to the WRT. • GStreamer can perform most such processing tasks, even if some of the options required are really complicated. Where little or no processing is required, then of course you may avoid any GStreamer overhead by creating Custom SW to deal with V4l2 or IP camera interface directly. However, if it is simply that GStreamer does not provide the processing you need, you should probably consider creating a GStreamer plug-in, which would then make all the GStreamer facilities available too. Processing layer (could include WRT or Weston Plug-in)
  13. 13. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 13 • Could be a Linux Video device in combination with V4l2 interfaces. Most applicable to USB and Video Capture board types of HW. Functionality and parameters available thru ioctl interface should be also considered as part of this layer group • Could be custom interfaces made to support specific features such as video in h.264 • Network interface to IP cameras or non-standard interfaces to the devices where you would need to use some intermediate layers to get access to the camera even thru IP or other type of connection. Source layer
  14. 14. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 14 • Webkit: video tag. Does buffering of the video if used with network stream set directly to SRC. Lag depends to video type and properties, but typical values to expect are from 3 to 5 seconds. Another complication is list of supported video formats. Webkit used in Tizen-IVI supports OGG and H.264/MPEG-4 Part 10. On the plus side, it’s possible to run up to 6-8 streams at the same time with good FPS and quality of decoding. • Webkit: pure canvas. As fast as can be expected for single camera set up and can offer single camera view with FPS close to 25-30, but strictly depends on implementation in JS and below, which provides byte array with frame. Typical lag is about 1 second or less. • Webkit: WebGL could give you some fancy features with additional processing of the video frames, but still might be comparable to the pure canvas solution. • Webkit: JS MPEG-4 decoder is fast enough to get smooth play of the video with very minimal lag, but might not work out of the box on Tizen IVI. • WRT plug-in: D-BUS control interface. Can be Corba or any other interface to the system or custom service running to serve the webkit with the video streams and control the cameras over ioctl and V4L2 interfaces. Specific of particular selections of SW and HW
  15. 15. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 15 • WRT plug-in: Combination with underlying MW to control and run Gstreamer, or to read the frames directly from the Linux video interface such as /dev/videoX. Good for intermediate processing of the frames, but relatively slow because can work only with Canvas or other direct draw method. • Webkit: webkitGetUserMedia and createObjectURL. Doesn’t work out of the box with Tizen-IVI as Webkit is using GStreamer 1.0 library, but still sends the parameters in the GStreamer 0.10 format. Hence some modifications would be required to bypass this issue. Meanwhile, this way does work with Crosswalk and result is very smooth. • Webkit: own blob object in combination with WRT plug-in to serve video tag source. Could be comparable to webkitGetUserMedia and createObjectURL method, but performance would depend entirely on the WRT Plug-in implementation. • Weston: shell or compositor. Fast and flexible as you are managing merge of the layers and surfaces directly. Can be helpful in complex layer merge scenarios such as Media Player with data shown on top of the video, but constraints are in the high knowledge entrance level and complexity of the system. Additionally it may impact whole system performance and stability, hence shouldn’t be used without proper assessment of the needs and risks. Specific of particular selections of SW and HW (cont.)
  16. 16. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 16 • GStreamer: Complex, but powerful solution which can read multiple sources (from files to IP cameras) and give multiple types of outputs with different conversions in between input and output. Main points to consider are:  There will probably be more than one way to build a GStreamer pipe to get functionality you need.  Different versions of Tizen may include different versions of GStreamer and its plug-ins leading to different and sometimes failing behavior in between releases  Vaapi ( HW support ) is present, but it’s effectiveness depends on the Host system.  SW encoders may fail under heavy system load and produce corrupted streams.  Not everything is obvious . E.g. tcpserversink is good for streaming the data over the network, but you would need to create your own wrapper (proxy) to add HTTP headers if you try to use it with browser Video tag.  Many useful plug-ins are still in “bad” and “ugly” packages meaning you have very minimal support for most of functionality outside mainstream usage. Specific of particular selections of SW and HW (cont.)
  17. 17. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 17 • FFMPEG: Simpler than GStreamer, but also limited. Upstream versions are not expected to be available in the Tizen IVI repo and please be aware of modules which are not even built for Tizen. • Custom SW: Needed to deal with devices directly, or to control device status while streaming is ongoing thru other components. Highly dependent on circumstances, but might be required for:  Setting up Video Capture board parameters  Controlling status and availability of the devices  Matching devices versus some ID to identify exact cameras in multicamera setup. Specific of particular selections of SW and HW (cont.)
  18. 18. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 18 Let’s try to assemble combinations of SW for particular usecases defined before • Slow motion camera - Intermediate quality of the video, no requirements regarding lag In short this is the case to record/present status of object over the time rather than to display its immediate state. Simplest solution would be to use GStreamer and split the source into two identical outputs where the one saved to the file and second displayed to user. No lag requirement means variety of options to implement presentation layer • Rear view on the display - Minimal lag, high resolution, flipped frames For the USB and Analog cameras the simplest way would be a combination of webkitGetUserMedia and createObjectURL methods. A custom solution could be implemented, such as own blob object or Weston plug-in for overlay in combination with GStreamer. Last options are helpful especially if there is a need for camera multiplexing or IP camera usage. Flipping could be done by browser as layer property, hence no additional processing required. Solutions
  19. 19. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 19 • 360 degree stitched view to present road around the car - Multiple cameras setup, correction of lens distortion, stitching of camera outputs into single panoramic video, minimal lag Architecturally correct way would be custom GStreamer plug-in implementation and usage of OpenCV for different sources stitching, blending and lens corrections. Standard videomixer plug-in is good example for quick start with GStreamer plug-ins. Custom SW could implement the same functionality by reading the frames from Linux Video interfaces directly or using GStreamer as a library. For minimal lag Weston overlay or own blob object could be used to pass video. • Visualization of the driving details and some camera view merged on windshield - Merge of the dynamic/static data with actual camera view, minimal lag, medium FPS If data is primarily textual information, then browser layer capabilities could be used to merge it with video stream. Otherwise variety of GStreamer plug-ins could do it at the cost of some MW complication. With medium FPS browser Canvas is good option where the frames are read directly from device (or from GStreamer for IP cameras), processed in custom SW and then passed to Canvas thru WRT plug-in. Meanwhile webkitGetUserMedia and createObjectURL methods are still good if there is no need to deal with IP cameras. Solutions (cont.)
  20. 20. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 20 • In car camera to control taxi passenger area - Multiple cameras setup, medium or low FPS, medium lag. In general similar to “360 degree stitched view” case, but lower requirements for presentation part, hence Canvas or own blob object can be implemented to handle display. • One by one frame processing before display in various recognition cases such as faces, road signs or other objects recognition - Better no lag, but depends to the case, medium FPS as the recognition may take longer than single frame release time, no en(de)coding artifacts to achieve best recognition results. Custom SW to read frames directly from the device or GStreamer library and to process it further can be implemented. Meanwhile, depending on the processing required, it could be a Gstreamer plug-in doing analysis of the frames and sending the signals to upper layers. Canvas or blob object for Medium FPS and minimal lag. Weston overlay may overcomplicate the system, but technically possible to achieve minimal lag and best performance of the display. Solutions (cont.)
  21. 21. Copyright © 2013 Symphony Teleca Corp. All rights reserved. CONFIDENTIAL AND PROPRIETARY 21 Questions? Last slide 
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×