Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Selecting the Right Imager for Your Embedded Vision Application," a Presentation from Capable Robot Components


Published on

For the full video of this presentation, please visit:

For more information about embedded vision, please visit:

Chris Osterwood, Founder and CEO of Capable Robot Components, presents the "Selecting the Right Imager for Your Embedded Vision Application" tutorial at the May 2019 Embedded Vision Summit.

The performance of your embedded vision product is inexorably linked to the imager and lens it uses. Selecting these critical components is sometimes overwhelming due to the breadth of imager metrics to consider and their interactions with lens characteristics.

In this presentation, you’ll learn how to analyze imagers for your application and see how some attributes compete and conflict with each other. A walk-through of selecting an imager and associated lens for a robotic surround-view application shows the real-world impact of some of these choices. Understanding the terms, the trade-off space and application impact will guide you to the right components for your design.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

"Selecting the Right Imager for Your Embedded Vision Application," a Presentation from Capable Robot Components

  1. 1. © 2019 Your Company Name Selecting the Right Imager and Lens for Your Embedded Vision Application Chris Osterwood Capable Robot Components May 2019
  2. 2. © 2019 Capable Robot Components Capable Robot Components •Focused on making better building blocks for mobile robots and autonomous systems. Product areas: •Cameras: Sealed cameras and downstream data & sync interfaces. •Communications: USB Hub & Embeddable GigE Switch •Sensor & Computing 2 GMSL to CSI-2 Interface with Power over Coax Injection GMSL Camera TX2 CSI-2 Interface (6x) with FSYNC Programmable & Instrumented USB Hub
  3. 3. © 2019 Capable Robot Components Presentation Outline Camera System Overview: Data Types, Pixel Encoding, Physical Interconnect Imagers: Performance & Non-Performance Criteria Matching / Interfacing to Processor Lenses: Performance & Non-Performance Criteria Matching to Imagers 3
  4. 4. © 2019 Your Company Name Camera System Overview
  5. 5. © 2019 Capable Robot Components A vision system - at the highest level 5
  6. 6. © 2019 Capable Robot Components What are we sending / creating? 6
  7. 7. © 2019 Capable Robot Components Pixel Encoding Options 7 Software Imager
  8. 8. © 2019 Capable Robot Components 8 Physical Layer Options : From Imagers
  9. 9. © 2019 Capable Robot Components 9 Physical Layer Options : From Cameras
  10. 10. © 2019 Capable Robot Components Imager : Data Interface & Pixel Encoding Choice of processing system can limit or increase the implementation cost of some data interfaces & pixel encoding. For example: •Intel CPUs: No low-level way to connect a camera. Options are Ethernet, PCIe based frame grabber, or USB. •NVIDIA Jetson: CSI-2 input supported, but most imagers are not CSI-2. Bridge chips (Lattice, TI) or GMSL / FPD-Link required to translate. •MCUs: Some are parallel input only, some are CSI-2 input only. Hardware encoding may only work with YUV input data. Some have no camera interface at all. •subLVDS / HiSPi: FPGA or custom ASIC needed to receive these imager busses and convert to CSI-2 or parallel. •USB UVC: Spec only allows for YUV, MJPEG, H264 encoding. Cannot transmit RAW data. Many imagers will need a companion ISP before the USB PHY. 10
  11. 11. © 2019 Your Company Name Imagers
  12. 12. © 2019 Capable Robot Components Imager : Fundamentals •Imagers count photons. •Photons hitting a pixel increase the voltage on that pixel. •Micro-filters over pixels allow some spectral information to be measured. •In CMOS imagers, much surface area is dedicated to support electronics, not the photodiode itself. Micro-lenses help to direct more light to the sensitive region. •Process: •Reset = Drain charge in pixel and bias the photodiode. •Expose / Integrate = Photons hitting photodiode produce electrons, stored beneath the diode in a “potential well.” •Readout = Well voltage read by amplifier (which has a controllable gain). Generally have to complete this before the next frame can be exposed. 12
  13. 13. © 2019 Capable Robot Components Imager : Performance Criteria •Horizontal & Vertical Pixels. Lots of different aspect ratios & sizes out there. •Pixel Size. Distance from pixel to pixel. Note, NOT the size of the active part of the pixel. •Shutter Type. Global Shutter (GS) or Electronic Rolling Shutter (ERS) •Frame Rate. Specifications here can be misleading. Can change with operating resolution and interface used. •Color Filter Array (CFA). None, Bayer (RGGB), RCCC, RCCB, RGBC / RGB-IR. •Quantum Efficiency & Spectral Response. Percent of photons that add to pixel voltage. •Dynamic Range. Ratio between full well voltage and noise floor. •Signal to Noise (SNR). Ratio between full well voltage and RMS noise of full well signal. •Power Draw. CMOS imagers have higher noise at higher temperatures. Imagers that draw excessive power have self-heating and exhibit this sooner. 13
  14. 14. © 2019 Capable Robot Components Imager : Non-Performance Criteria •Pixel Data Interface. Parallel, LVDS, and CSI-2 most common. Vendor specific busses: subLVDS, HiSPi •Pixel Encoding. Generally RAW (8, 10, or 12 bits). Some imagers have an ISP on-chip which can expose YUV or RGB data. •Control Interface. I2C most common (sometimes called something else). Occasionally SPI. •Trigger / Flash Signaling. Generally imagers have a trigger / sync control input. ERS imagers sometimes have a flash / strobe output. •Additional IO / Sensors / Data. GPIO? On-die temperature sensor? Histogram? •Additional Features. WOI/ROI? AEC? AGC? •Price •Lead Time / Availability. Ranges from stock to 20+ weeks. Depends on the vendor and the SKU. •Die Packaging. BGA & LGA most common. PLCC, QFP, PGA, Bare Die less common. •Documentation Availability. NDAs generally required to get full data-sheet & register interfaces. Difficulty ranges from easy to impossible between vendors (unless you’re very large and known company). 14
  15. 15. © 2019 Capable Robot Components Imager : Shutter Types 15
  16. 16. © 2019 Capable Robot Components Imager : Shutter Types 16
  17. 17. © 2019 Capable Robot Components Imager : Distortion caused by ERS 17 At same pixel rate, skew increase as imager resolution increases
  18. 18. © 2019 Capable Robot Components Imager : Filter Patterns 18 No Filter RGGB / Bayer RCCB RCCC • High Sensitivity. • No chroma information. • High Sensitivity . • Common in automotive, allows red-lights to be identified. • Most common pattern • Extra green pixels due to human sensitive in that spectrum. • Full color information can be extracted, with extra processing step: G = (C+C)/2 - (R+B) • Higher sensitivity / better SNR (3 dB) than RGGB. RGB C/IR • Full color information can be extracted. • Very few shipping imagers have this pattern.
  19. 19. © 2019 Capable Robot Components Imager : Filter Patterns & Sensitivity •These response graphs are from public CMOSIS / AMS CMV2000 data- sheet, but are typical for all CFAs. •Cannot get 100% QE due to: losses in non-photodiode region, reflections, incident angle to the micro-lens, and absorption w/o charge generation (prevalent with higher wavelengths). •Peak QE drops from ~65% to ~45% due to CFA. •Notice CFA pass-behavior from 700 nm upwards — this will cause color distortion without an additional IR-cut filter in the optical path. 19 CFA + Lenses
  20. 20. © 2019 Capable Robot Components Imager : Pixel Size : Noise vs Spatial Resolution Larger pixels have greater well capacity (e.g. more electrons before saturation), therefore greater SNR. Smaller pixels have greater spatial resolution. 20 SNR of 2.0 thru 5.2 um pixels Spatial Resolution of 2.0 thru 5.2 um pixels
  21. 21. © 2019 Capable Robot Components Imager : Effect of Temperature • As imager temperature increases, dark current draw increases. • This causes the pixel noise floor to increase, which reduces the dynamic range of the imager. • Table here shows images & associated histograms of a particular imager at different ambient temperatures. • This imager starts to loose dynamic range between 20C and 40C, with significant range compression occurring between 60C and 80C. 21 0C 20C 40C 60C 80C 100C
  22. 22. © 2019 Your Company Name Lenses
  23. 23. © 2019 Capable Robot Components •Optical Size: Trade size, does not equal the actual size of the image circle •Focal Length: (assuming no distortion affects). •Aperture: Ability to gather light. •Lower numeric f-stop increases number of photons hitting the imager. •Higher numeric f-stop increases depth of field (what is in focus). •Resolving Power / Sharpness: Scored via Modulation Transfer Function (MTF) •Distortion: No recognized standard here. •Temperature & Vibration Rating: No recognized standard here. Lens : Performance Criteria 23
  24. 24. © 2019 Capable Robot Components Lens : Non-Performance Criteria •Mounting •Sealing •Length & Diameter •Weight •Coatings •Adjustable Aperture •Adjustable Focus •Adjustable Zoom •Price 24
  25. 25. © 2019 Capable Robot Components Lens & Imager Matching : Optical Size 25 Note: This is physical imager size vs lens optical format. No bearing on system’s optical resolving power (dependent on imager resolution and sharpness of lens).
  26. 26. © 2019 Capable Robot Components Lens & Imager Matching : Focal Length / Imager Size 26 Focal Length 12mm 5mm 3.6 mm Optical Format 4/3” 2/3” 1/3” FOV 75° x 60° 82° x 65° 78° x 49° Size ⌀57mm x 85mm 94x ⌀42mm x 38mm 23x ⌀14mm x 15mm Weight 270g 67x 85g 21x 4g Similar
  27. 27. © 2019 Capable Robot Components 360° Robot Teleoperation 27 Number of Cameras Minimum H FOV Lens FL Lens H FOV Lens V FOV PX / degree Total Mpx 2 180° 1.58mm fisheye 185° 138° 10 @ center 4.2 3 120° 2.25mm fisheye 136° 102° 14 @ center 6.3 4 90° 2.5mm 98° 66° 19 8.4 5 72° 3.6mm 78° 49° 24 10.5 6 60° 4.4mm 67° 41° 28 12.6 •2 Mpx imager for camera. Automotive grade. •1/3” optical format allows for compact & low cost M12 lenses. •GMSL interconnect to central processing NVIDIA SOC Non-linearity of most fisheye lenses causes space compression at image edges. Limited Vertical FOV Limited Resolution
  28. 28. © 2019 Your Company Name Conclusion
  29. 29. © 2019 Capable Robot Components Conclusions •Imager and lens choices should be driven by your application. Consider: •Resolving power (px/deg or px/dim). •Required field of view. •Size, weight, cost •Imager and lens are inexorably linked. Match optical size and resolving power, otherwise waste in size, weight, & cost •Interconnect between imager and processors has large impact to system design and overall topology. 29 Optical Format & Imager Resolution & Lens Focal Length
  30. 30. © 2019 Capable Robot Components Additional Resources 30 Imagers s.html Lenses