Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion Systems

1,911 views

Published on

The expanding demand for imaging- and vision-based systems in mobile, IoT and automotive products is making the need for multi camera and sensor fusion systems look for novel ways to gather and process multiple data streams while still fitting into the mobile interface. The CSI-2 protocol allows camera sensor and processed image data to be combined into a single data stream using interleaving, allowing the application processor to extract the image data using the virtual channel or data type information. In this presentation, Richard Sproul of Cadence Design Systems will highlight some of the key details and requirements for a system with image processing of multi camera/sensor systems.

Published in: Mobile

MIPI DevCon 2016: MIPI CSI-2 Application for Vision and Sensor Fusion Systems

  1. 1. MIPI CSI-2℠ CSI-2℠ Application for Vision and Sensor Fusion Systems Richard Sproul – Cadence Design Systems, IP Architect
  2. 2. Overview •  The expanding demand for imaging and vision based systems in mobile, IoT and automotive products is creating the need for multi-camera and sensor fusion systems to look for novel ways to gather and process multiple camera/sensor data streams whilst still fitting into the mobile interface. •  The presentation will highlight some of the key details and requirements for a system with image processing of a multi-camera/sensor system. 2
  3. 3. CSI-2 Application for Multi-Sensor Systems Multi-Camera Applications •  Imaging applications are everywhere •  Mobile Phone –  Selfie Picture in Picture –  Gesture Recognition •  Video Games –  Gesture Recognition •  Autonomous Driving –  Pedestrian Detection –  Signage Recognition –  Night Vision –  Parallel Parking! •  In-Car Control –  Gesture Recognition
  4. 4. 4 More than the naked eye… CSI-2 Application for Multi-Sensor Systems Machine Vision
  5. 5. 5 CSI-2 Application for Multi-Sensor Systems Camera Applications Op0mal pathway for mul0ple forward-looking advancements in imaging – Key Drivers: Health, Convenience, Security, Lifestyle, Efficiency – High-perf pixel conduit needs met with C/D-PHY advancements – Broad defini0ons and fuzzy range: (i.e. Wearable: Near Body, On Body, In Body) • Explore possibili0es of overlap between Imaging and low-speed sensor requirements and solu0ons Camera Controller Interface (CCI/CCS) advancement considera0ons: - Point-to-Point and Mul0-Drop configura0ons - Energy consumed / Gb transfer - Limit latency for VB & HB - Precision Timing & Sync - Independent Transport: Pixel Data & Control - Channel Integrity (Error Detec0on) - FW Upload (ISP, Neural)
  6. 6. •  Options for physical interface •  Pins, legacy, bandwidth 6 CSI-2 Application for Multi-Sensor Systems MIPI CSI-2 Interfaces
  7. 7. •  Evolution of the CSI-2 7 CSI-2 Application for Multi-Sensor Systems CSI-2 Generations
  8. 8. 8 CSI-2 Application for Multi-Sensor Systems CSI-2 Performance
  9. 9. •  CSI-2 packets V1.x to V2.0 9 LP LP LP LP Transi0on between packets to LP state for PHY data lane (100ns) Transi0on between packets by using filler pa_erns CSI-2 Application for Multi-Sensor Systems CSI-2 Packet Structure
  10. 10. •  Improve the effective bandwidth 10 0 20 40 60 80 100 120 1000 1250 1500 1800 2000 2250 2500 Frame Rate fps Bit Rate (Mbps) CSI-2 Frame Rate Improvement V1.x to V2.x 1920x1080 RAW12 FPS (V1.x) FPF (V2.x) CSI-2 Application for Multi-Sensor Systems CSI-2 Packet Transmission
  11. 11. •  Automotive application for driver assistance - External systems and for in-car control • Objects • High resolu0on • Night image and IR • In-car gesture • People detec0on • Medium resolu0on • Road signage • Medium resolu0on Parking assistance CSI2 Application for Multi-Sensor Systems Advanced Driver-Assistance System
  12. 12. CSI-2 Application for Multi-Sensor Systems •  Application in a Multi-Camera Platform 12 Automotive AV Reference Subsystem MIPI DPHY Audio DSP $I $D System Interconnect Image/Vision DSP DMA I-RAM D-RAM AXI2 AHB UART I2C 32b APB Timer I2S GPIO AHB2 APB 32b AHB QSP I Soun dWir e Audio USB 2/3 devic e Ethe rnet MAC On-Chip System SRAM 1300MT/s DDR3 Controller DDR-PHY 64b DDR3 SODIMM SD SDIO eMM C Displa y Inm. BR PHY USB PHY Pixel 2AXI Color Conver t Video Scalar HDMI PHY Image/Vision DSP DMA I-RAM D-RAM MIPI CSI-2 Rx MIPI CSI-2 RX MIPI CSI-2 Rx MIPI DPHY MIPI DPHY Sensor DSP $I
  13. 13. 13 CSI-2 Application for Multi-Sensor Systems Sensors Everywhere
  14. 14. •  The data does not have to be images… •  LiDAR •  The resolution is low (IR RAW data, typically 64 pixels high, though much more horizontally) •  Range is limited. Typical LiDARs see well to about 70 metres. •  Refresh rates tend to be slower, at around 10fps. •  RADAR •  Long range – cruise control, brake assist •  Ultrasonic •  Short-range parking assist •  Self parking ☺ •  Protocol support with user-defined data to transfer the bytes 14 CSI-2 Application for Multi-Sensor Systems CSI-2 for Sensors
  15. 15. CSI-2 Application for Multi-Sensor Systems CSI-2 Example Video Frame •  Bandwidth on CSI-2 V1.1 – 4 Lanes 6Gbps •  So with our 30fps, we have 200M bit to use •  3 HD camera RGB888 1920x1200x24=165.888M •  Also adding 100ns gaps (150 bit clocks) •  3 x(1920x24) +3x150 = 138240 •  Embedded data line with image processed data (clusters, edges)
  16. 16. 16 CSI-2 TX Controller Video Buffer Centralized ECU for Infotainment or ADAS Pixel Pixel Data Data Sensor DSP Sensor DSP PCIe Sensor DSP I2S I2S I2S SoundWire eMMC Apps USB DSI DDR DPHY PCIe PHY USB PHY Sensor Sensor Video EP Video EP PPI PPI DPHY DPHY Pixel Pixel PHY Data Data CSI-2 domain Video Buffer Data Buffer Data Buffer I2S SoundWire Sensor DSP CSI-2 RX CTRL Vision DSP CSI-2 Vision DSP CSI-2 CSI-2 Application for Multi-Sensor Systems CSI-2 Sensor Fusion Example •  Sensor Fusion ADAS System Topology •  Merge the data from image and other sensors •  Pre-processing the inline data for the application
  17. 17. •  Image Processing and the Application •  Application processing will need to perform the ADAS system and sensor analysis 17 CSI2 Application for Multi-Sensor Systems CSI-2 Sensor Fusion Example
  18. 18. 18 CSI-2 Application for Multi-Sensor Systems Filling the Channels D PHY (MCNN) D-PHY (MFEN) Pixel Processor/ Application DP DP DN DN D PHY (SCNN) D-PHY (SFEN) Pixel Stream 0 PPI PPI D-PHY (MFEN) DP DN D-PHY (SFEN) PPI Pixel Stream 1 Pixel Stream 2 Pixel Stream 3 Sensor Processor/ Application Pixel Stream 4 Pixel Stream 5 Pixel Stream 6 Pixel Stream 7 CSI-2 Host Controller PPI PPI PPI D-PHY (MFEN) D-PHY (SFEN) DP DN CSI-2 Slave Controller PPI PPI Sensor Control Sensor Control Sensor Sensor Sensor Sensor Sensor DATA DATA DATA DATA DATA DATA DATA CSI-2 Slave Controller CSI-2 Host Controller
  19. 19. 19 CSI-2 Application for Multi-Sensor Systems Physical Interface
  20. 20. CSI-2 Application for Multi-Sensor Systems CSI-2 Example Video Frame •  Bandwidth on CSI-2 V1.1 – 4 Lanes 6Gbps –  Using LS/LE to keep synchronisation and sequence –  Use the virtual channel to identify the sensor –  Use the data types (RAW, RGB, YUV and user defined) –  Use the short packet sync events
  21. 21. •  Functional safety considerations •  DPHY BER, RX error detection •  Packet header ECC •  Payload CRC •  SP sync sequences, counting values 21 CSI2 Application for Multi-Sensor Systems Functional Safety In CSI2 ADAS
  22. 22. •  Pitfalls of interleaved streams 22 CSI-2 Application for Multi-Sensor Systems CSI-2 Interleaving Data
  23. 23. 23 CSI-2 Application for Multi-Sensor Systems CSI-2 Interleaving
  24. 24. CSI-2 Beyond Mobile 24
  25. 25. CSI-2 Beyond Mobile •  System architecture considerations for CNN applications: •  Assist •  Co-pilot •  Automated •  Optimal platform arch for the CNN engines •  Central processing (+SW dev, lacks scalability / modularity, cost may not align w/ entry-level cars) •  Distributed processing (plug-and-play, scalable, each camera unit enhances capability, complex system) •  AlgoEngine: CPU / GPU / DSP / FPGA •  Overall risks and uncertainty: •  Market, product, execution, timing, regulators, infrastructure 25
  26. 26. CSI-2 Beyond Mobile •  What can technology do for us? •  Imaging: digital photography vs. vision •  Scene capture, object capture & track, modeling & measurement •  Perception and decision-making using real-time streaming image data: •  Camera, RADAR, LiDAR, sonar (varying detection capabilities vs. cost) •  Performance vs robustness – consequence of error •  Camera position, lighting, environmental factors, required accuracy for object detection 26

×