Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Multiple MIPI
CSI-2℠ Cameras
Leveraging FPGAs
Ted Marena
Director of SoC FPGA
Marketing, Microsemi
Agenda
•  History & adoption of MIPI CSI-2 image sensors
•  MIPI CSI-2 interface primer
•  FPGA usage models
•  Applicatio...
Evolution of Image Sensor Interfaces
Parallel	CMOS	interface	 subLVDS	/	HiSPi	interface	/	etc.	
In	90s	&	2000s,	parallel	i...
As mobile platforms exploded, so did CSI-2
4
Because of mobile popularity, everyone
drifted to CSI-2
•  Just like a decade ago when PC components were used
broadly, as...
Title & Date
“LP	Mode”	–	1.2V	CMOS	
“HS	Mode”	–	SLVS200;		Vcm	=	200mV	&	Vdiff	=	+/-	100mV	
MIPI CSI-2 D-PHY Overview
•  Waveforms
Low	
Power	
Mode:		
TransiUon	from	
Low	Power	to	
High	Speed	
High	Speed	Mode	
TransiUon	from	
High	Speed	to	...
MIPI Byte Packet Signaling
8’hB8	
8’hB8	
4-lane	example	
byte_data[7:0]	 {VC,DT}	 Byte(0)	 Byte(4)	 CS	 ECC	 Byte(n)	…	
…	...
MIPI D-PHY, CSI-2 Overview
•  Universal D-PHY
Title & Date
1.2V	CMOS	Transminer	
1.2V	CMOS	Receiver	
1.2V	CMOS		
ContenUon...
D-PHY vs. C-PHY
•  The majority of MIPI CSI-2 cameras use the D-PHY.
C-PHY,	3-wire,	embedded	clock,	3-phase	coding	
D-PHY,...
Emulating D-PHY in FPGA
•  Normally FPGAs do not have a D-PHY hard I/O block
•  They usually interface via translation res...
Key Blocks Used in FPGAs for Imaging
12
AP/uP/uC	
Math	Blocks/DSP	 Memory	Blocks	
I/O	Gearing	 Processor/Micro
FPGAs in Imaging/Video Applications
Bridging is the simplest design.
Acceleration requires more performance & capabilities...
FPGAs Also Do Processing & Aggregating
Processing could be in an embedded processor or with FPGA fabric,
memory and math b...
Why Use an FPGA?
•  Not enough MIPI CSI-2 or other inputs on an AP/ISP
•  Limited numbers of ISP engines on an AP/ISP
•  F...
Multi MIPI CSI-2 Camera Applications
•  3D camera / virtual reality
•  Dual surveillance
•  Multiple-image sensor HDR
•  1...
3D Camera Example
Dual	Image	Sensors	
Bridge
CSI-2
CSI-2	
1-4	
Data	
Lanes	
Dual CSI-2 Bridge
CSI-2	
1-4	
Data	
Lanes	
Syn...
3D Camera FPGA Implementation
Dual	Image	Sensors	
Bridge
CSI-2
CSI-2	
1-4	
Data	
Lanes	
Dual CSI-2 Bridge
CSI-2	
1-4	
Data...
Dual Surveillance Camera
Dual	Image	Sensors	
Bridge
CSI-2
CSI-2	
1-4	
Data	
Lanes	
CSI-2	
1-4	
Data	
Lanes	
CSI-2	Tx	
Alth...
Dual Surveillance FPGA Function
This	soluUon	allows	an	ISP	to	process	both	images.	
•  Both	images	are	recovered	in	the	FP...
Image Sensor HDR Processing
CSI-2
May go to ISP/AP or storage
Each image sensor captures frames at exactly the
same time. ...
Image Sensor HDR Processing
CSI-2
Short-, medium- and long-exposure images. Processed using local and global
tone mapping,...
180-Degree Surveillance Camera
Bridge
CSI-2
CSI-2	
1-4	
Data	
Lanes	
CSI-2 Stitching
CSI-2	
1-4	
Data	
Lanes	
CSI-2	
The	i...
180-Degree Surveillance FPGA Function
•  MulUple	images	are	recovered	in	the	
FPGA	
•  The	frames	are	stored	likely	in	ext...
360-Degree Cameras
CSI-2
Each image sensor frames are captured and combined.
Image processing could be in the FPGA or AP/I...
360-Degree Cameras
•  The FPGA performs an analysis to determine
where to merge the images
•  The images are stitched toge...
Surround View Automotive
CSI-2
Frame	
Buffer	
memory	
FPGA	
FPD3	Link	to	
MIPI	CSI-2	
FPD3	Link	to	
MIPI	CSI-2
Surround View Application
•  The FPGA implements the stitching of the images
•  It formats the image for the ISP/AP
•  Or ...
Multi Camera for Depth Detection
CSI-2
The FPGA processes all the images and
provides the lowest possible latency for
high...
Multi Cameras for Depth-Based Analytics
Dual	Image	Sensors	
The	FPGA	or	AP	computes	3D	point	cloud	from	a	top-down	stereo	...
Depth Detection for Robotics
•  The FPGA synchronizes each camera pair and processes what each
camera pair sees
•  The rob...
Multi Camera Drone Application
CSI-2
These dual cameras allow a drone to “see.”
Image	Buffer	
Memory	
FPGA	syncs	
dual	came...
Multi Camera Drone Application
•  The FPGA can merge, mux, pre-process, and run
analytics pre-processing before the AP get...
Summary
•  Multiple camera applications continue to grow.
•  MIPI CSI-2 adoption will grow and be further adopted
outside ...
Upcoming SlideShare
Loading in …5
×

MIPI DevCon 2016: Multiple MIPI CSI-2 Cameras Leveraging FPGAs

4,689 views

Published on

The use of multiple cameras applications is exploding. We're not just talking about 2 cameras for 3D or depth sensing, but 3-12 cameras for applications like drones, robotics, automotive, etc. The increasing use of multiple cameras combined with the growing use of mobile components such as apps processors, image sensors, displays, etc., being used by the broad market requires logic to connect these devices. In this presentation, Ted Marena of Microsemi explains how FPGAs can be used to leverage mobile components to aggregate a large number of MIPI CSI-2 camera interfaces.

Published in: Mobile
  • Be the first to comment

MIPI DevCon 2016: Multiple MIPI CSI-2 Cameras Leveraging FPGAs

  1. 1. Multiple MIPI CSI-2℠ Cameras Leveraging FPGAs Ted Marena Director of SoC FPGA Marketing, Microsemi
  2. 2. Agenda •  History & adoption of MIPI CSI-2 image sensors •  MIPI CSI-2 interface primer •  FPGA usage models •  Applications for multiple MIPI CSI-2 image sensors with FPGAs •  Summary 2
  3. 3. Evolution of Image Sensor Interfaces Parallel CMOS interface subLVDS / HiSPi interface / etc. In 90s & 2000s, parallel interface was the norm In the next decade, various proprietary interfaces were introduced ~ 2-3MP
  4. 4. As mobile platforms exploded, so did CSI-2 4
  5. 5. Because of mobile popularity, everyone drifted to CSI-2 •  Just like a decade ago when PC components were used broadly, as mobile adoption exploded, so did the acceptance of MIPI CSI-2 5
  6. 6. Title & Date “LP Mode” – 1.2V CMOS “HS Mode” – SLVS200; Vcm = 200mV & Vdiff = +/- 100mV MIPI CSI-2 D-PHY Overview
  7. 7. •  Waveforms Low Power Mode: TransiUon from Low Power to High Speed High Speed Mode TransiUon from High Speed to Low Power LP11, LP01, LP00, HS0, HS-SYNC …. HS0, LP-11 References the P and N channel value in LP mode respecUvely References the value of the SLVS200 signal state …00011101 = ‘hB8 MIPI CSI-2 Transition
  8. 8. MIPI Byte Packet Signaling 8’hB8 8’hB8 4-lane example byte_data[7:0] {VC,DT} Byte(0) Byte(4) CS ECC Byte(n) … … byte_clk sync byte_data[15:8] WC Byte(1) Byte(5) {VC,DT} Byte(0) Byte(n-1) … 8’hB8 WC Byte(2) Byte(6) WC Byte(1) CS … 8’hB8 ECC Byte(3) CS WC Byte(2) CS … byte_data[23:16] byte_data[31:24] 8’b00011101 HS-SYNC Sequence Packet Header VC – Virtual Channel number DT – Data Type WC = 16 bit word length count ECC – Checksum on word count CS –Checksum on data
  9. 9. MIPI D-PHY, CSI-2 Overview •  Universal D-PHY Title & Date 1.2V CMOS Transminer 1.2V CMOS Receiver 1.2V CMOS ContenUon DetecUon Circuit SLVS200 Transminer SLVS200 Receiver Dynamic TerminaUon
  10. 10. D-PHY vs. C-PHY •  The majority of MIPI CSI-2 cameras use the D-PHY. C-PHY, 3-wire, embedded clock, 3-phase coding D-PHY, diff-pairs, source synchronous, no coding TransiUon at every symbol, used for clock recovery
  11. 11. Emulating D-PHY in FPGA •  Normally FPGAs do not have a D-PHY hard I/O block •  They usually interface via translation resistors 11 FPGA I/Os Image Sensor I/Os
  12. 12. Key Blocks Used in FPGAs for Imaging 12 AP/uP/uC Math Blocks/DSP Memory Blocks I/O Gearing Processor/Micro
  13. 13. FPGAs in Imaging/Video Applications Bridging is the simplest design. Acceleration requires more performance & capabilities. 13 FPGA A B FPGA HW ACC A A’ DRAM Op9onal Bridging AcceleraUon
  14. 14. FPGAs Also Do Processing & Aggregating Processing could be in an embedded processor or with FPGA fabric, memory and math blocks (DSP blocks). Aggregation leverages the large I/O capability of FPGAs and the fabric. 14 FPGA Aggrega9on A A’ DRAM Processing AggregaUon FPGA Processing A A’ B’ DRAM B C
  15. 15. Why Use an FPGA? •  Not enough MIPI CSI-2 or other inputs on an AP/ISP •  Limited numbers of ISP engines on an AP/ISP •  FPGA may perform some processing allowing for a lower cost AP/ISP •  Newer capabilities that are not available with an AP/ISP •  An FPGA can implement a complete ISP 15
  16. 16. Multi MIPI CSI-2 Camera Applications •  3D camera / virtual reality •  Dual surveillance •  Multiple-image sensor HDR •  180-degree surveillance •  360-degree panorama •  Surround view automotive •  Depth-detection applications •  Drone usage 16
  17. 17. 3D Camera Example Dual Image Sensors Bridge CSI-2 CSI-2 1-4 Data Lanes Dual CSI-2 Bridge CSI-2 1-4 Data Lanes Sync and format logic Although ISP devices oren have mulUple camera inputs, they oren benefit from an FPGA. Synchronizing the image sensors & arranging them
  18. 18. 3D Camera FPGA Implementation Dual Image Sensors Bridge CSI-2 CSI-2 1-4 Data Lanes Dual CSI-2 Bridge CSI-2 1-4 Data Lanes Sync and format logic The FPGA can arrange the image in a side-by-side or a top-bonom configuraUon. This makes it easier for the ISP or AP to process the image.
  19. 19. Dual Surveillance Camera Dual Image Sensors Bridge CSI-2 CSI-2 1-4 Data Lanes CSI-2 1-4 Data Lanes CSI-2 Tx Although ISP devices oren have mulUple camera inputs, they oren benefit from an FPGA that can arrange the image. ISP Image Buffer Memory Display spliter
  20. 20. Dual Surveillance FPGA Function This soluUon allows an ISP to process both images. •  Both images are recovered in the FPGA •  The FPGA combines the two images into one for the ISP – oren a top-bonom configuraUon •  This allows the ISP to process the two images as one, but the output can be split into two images
  21. 21. Image Sensor HDR Processing CSI-2 May go to ISP/AP or storage Each image sensor captures frames at exactly the same time. A short, medium and long exposure is used for each. Image Buffer Memory FPGA with HDR engine
  22. 22. Image Sensor HDR Processing CSI-2 Short-, medium- and long-exposure images. Processed using local and global tone mapping, motion artifact correction, etc. Image Buffer Memory FPGA with HDR engine
  23. 23. 180-Degree Surveillance Camera Bridge CSI-2 CSI-2 1-4 Data Lanes CSI-2 Stitching CSI-2 1-4 Data Lanes CSI-2 The image sUtching funcUon is more easily done in an FPGA. In this design, the enUre ISP could also be in the FPGA. ISP Image Buffer Memory . . . 3 or more
  24. 24. 180-Degree Surveillance FPGA Function •  MulUple images are recovered in the FPGA •  The frames are stored likely in external memory •  The FPGA performs an analysis to determine where to merge the image •  The images are sUtched together •  Also likely a smoothing technique is used •  The image output is then processed in the FPGA or formaned and passed onto the ISP or AP
  25. 25. 360-Degree Cameras CSI-2 Each image sensor frames are captured and combined. Image processing could be in the FPGA or AP/ISP. Image Buffer Memory FPGA with sUtching, fisheye correcUon, etc . . . ISP or AP
  26. 26. 360-Degree Cameras •  The FPGA performs an analysis to determine where to merge the images •  The images are stitched together •  Depending on the output format, fisheye correction may be implemented •  The image output is then processed in the FPGA or formatted and passed onto the ISP or AP 26
  27. 27. Surround View Automotive CSI-2 Frame Buffer memory FPGA FPD3 Link to MIPI CSI-2 FPD3 Link to MIPI CSI-2
  28. 28. Surround View Application •  The FPGA implements the stitching of the images •  It formats the image for the ISP/AP •  Or FPGA processes the image and drives the display •  FPGA could add overlay such as directional lines
  29. 29. Multi Camera for Depth Detection CSI-2 The FPGA processes all the images and provides the lowest possible latency for highest accuracy & response. Image Buffer Memory FPGA syncs dual camera pairs & does the processing . . . CSI-2 Could be one or more pairs
  30. 30. Multi Cameras for Depth-Based Analytics Dual Image Sensors The FPGA or AP computes 3D point cloud from a top-down stereo pair. Depth-based analyUcs can disUnguish adults, kids, people from shopping carts for accurate people counUng. Top-Down Depth Image FlowMetrics™ by PercepTonic
  31. 31. Depth Detection for Robotics •  The FPGA synchronizes each camera pair and processes what each camera pair sees •  The robot can preciously determine the distance for each axis that has a camera pair •  Parallel processing of the FPGA gives the robot the quickest response & accuracy
  32. 32. Multi Camera Drone Application CSI-2 These dual cameras allow a drone to “see.” Image Buffer Memory FPGA syncs dual camera pairs, muxes, encrypUon, etc. . . . CSI-2 GigE/10GE Encrypted video (wifi or other RF) PCIe
  33. 33. Multi Camera Drone Application •  The FPGA can merge, mux, pre-process, and run analytics pre-processing before the AP gets involved for 3D point cloud. Adds situational awareness so that drones do not crash into anything.
  34. 34. Summary •  Multiple camera applications continue to grow. •  MIPI CSI-2 adoption will grow and be further adopted outside of mobile applications. •  An increasing number of FPGAs will support MIPI CSI-2 image sensors. •  The parallel architecture of the FPGA is ideal for multiple camera applications. 34

×