Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"How to Choose a 3D Vision Sensor," a Presentation from Capable Robot Components


Published on

For the full video of this presentation, please visit:

For more information about embedded vision, please visit:

Chris Osterwood, Founder and CEO of Capable Robot Components, presents the "How to Choose a 3D Vision Sensor" tutorial at the May 2019 Embedded Vision Summit.

Designers of autonomous vehicles, robots and many other systems are faced with a critical challenge: Which 3D vision sensor technology to use? There are a wide variety of sensors on the market, employing modalities including passive stereo, active stereo, time of flight, 2D and 3D lasers and monocular approaches. This talk provides an overview of 3D vision sensor technologies and their capabilities and limitations, based on Osterwood's experience selecting the right 3D technology and sensor for a diverse range of autonomous robot designs.

There is no perfect sensor technology and no perfect sensor, but there is always a sensor which best aligns with the requirements of your application—you just need to find it. Osterwood describes a quantitative and qualitative evaluation process for 3D vision sensors, including testing processes using both controlled environments and field testing, and some surprising characteristics and limitations he's uncovered through that testing.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

"How to Choose a 3D Vision Sensor," a Presentation from Capable Robot Components

  1. 1. Copyright © 2019 Carnegie Robotics How to Choose a 3D Vision Technology Chris Osterwood Capable Robot Components May 2019
  2. 2. Copyright © 2019 Carnegie Robotics Carnegie Robotics, LLC 2 Carnegie Robotics is a leading provider of rugged sensors, autonomy software and platforms for defense, agriculture, mining, infrastructure, and energy applications. The company’s 5-acre campus collocates engineering, product testing and a 9,000 square foot production facility. Certified to ISO 9001-2015, the company prides itself in integrated systems of record for engineering, production, finance and quality.
  3. 3. Copyright © 2019 Carnegie Robotics •Criteria & Definitions •Overview of 3D Sensor Modalities: LIDAR, ToF Camera, Stereo •How do they work? •What are their general advantages and disadvantages? •Sample Data •3D Sensor Testing & Edge Cases •Not covered: •Laser Triangulation, Structured Light, & Computed Tomography Outline 3
  4. 4. Copyright © 2019 Carnegie Robotics • Field of View (FOV): Defines angular area of perceivable field of sensor • Density: • Angular step size between sample points • Can be different horizontally and vertically • Resolution: Generally FOV x Density • Depth Accuracy: Difference between measured range and actual range Depth Resolution: Step size between possible measurement ranges (along measurement axis) • Minimum and Maximum Range: Defines distances that are perceivable to the sensor. May vary by object material, brightness, reflectivity, etc. • Rate: Specified in “frames” or “points” per second, depending on modality Evaluation Criteria – Performance 4
  5. 5. Copyright © 2019 Carnegie Robotics • Size, Weight, & Power • Cost • Sealing: Dust proof, Water splash, Water streams, Immersion • Shock & Vibration Rating • Communication Interface: Ethernet, USB, Serial, CANBus, Firewire • Synchronization: None, Hardware, Software (broadcast trigger or network time sync) • Software Interface: • Published wire protocol? • Drivers. Platform? Language? License? • Example usage software? • Other: Trigger, GPIO, Lighting, IMU, Temperature & Health Reporting Evaluation Criteria – Non-Performance 5
  6. 6. Copyright © 2019 Carnegie Robotics How does end-use change criteria weighting? 6 UAV UGV Automotive Mapping Mapping Safety Mapping Safety HFOV VFOV Density Accuracy Mix range Max range Rate Swap Cost Sealing Shock/Vibe Sync Moderate weighting High weighting Weightings are affected by the platform and end-use of the sensor There are many other uses for 3D sensor The table here is an example of how there are unique drivers: • Mapping: Max Range • Safety: Rate • UAV: Weight • UGV: Vertical FOV / Min Range • Automotive: Cost
  7. 7. Copyright © 2019 Carnegie Robotics 7 3D Sensor Technologies
  8. 8. Copyright © 2019 Carnegie Robotics 3D LIDAR: How do they work? 8 Transmitter Phase measurement Transmitted Beam Reflected Beam Building block is single-beam system shown in the diagram here. 3D LIDARS can have: • 1 beam steered mechanically in 2 DOF • Multiple beams steered mechanically in 1 DOF Steering can be done with rotating mirror, galvanometer, or directly upon the emitter & detector
  9. 9. Copyright © 2019 Carnegie Robotics Downsides Advantages 3D LIDAR 9 • Constant error model — 1 sigma can be +/- 1 cm • Long range. 10 m to 2 km possible depending on laser power and pulse duration • General wide horizontal FOV due to rotating mirror or rotating emitters and detectors • Can have large vertical FOV. Dependent on laser and mechanism details • Cost. Trends are encouraging here, but low cost systems are not yet available. • Scan time — no instantaneous capture of data. System has to compensate for platform (or mechanism) motion during scan • Limited angular density & asymmetric density • Moving parts & non-planar windows are difficult to seal • Affected by obscurants in the environment
  10. 10. Copyright © 2019 Carnegie Robotics ToF Cameras: How do they work? 10 • Like LIDAR, but every pixel is a measurement point. • Results in limited range or more transmission power. • Accuracy increases with modulation frequency. • Maximum range decreases with modulation frequency. • Systems generally use multiple frequencies to allow for long range and non-ambiguous range.
  11. 11. Copyright © 2019 Carnegie Robotics • Dense data regardless of scene texture • Instantaneous horizontal & vertical FOV • Linear error with respect to distance • Narrower shadow behind positive obstacles • Solid state — no moving parts • Fewer “camera” parameters ToF Cameras 11 • Low resolution • Long integration time (to increase SNR and resolve range ambiguity) causes motion blur if platform or objects are moving • Susceptible to multi-echo returns distorting data • Affected by obscurant in the environment • Limited FOV — generally both horizontal and vertical • May not have an “image” output Downsides Advantages
  12. 12. Copyright © 2019 Carnegie Robotics Stereo Camera: How do they work? 12 Feature matches are found in left and right cameras. Difference in lateral position (disparity) is inversely proportional to distance. Matching process (correspondence problem) is critical to data accuracy and density. “Disparity search range” = number of horizontal pixels in right camera searched before moving onto the next left pixel. A larger search range allows seeing objects closer to the camera, but generally reduces 3D throughput.
  13. 13. Copyright © 2019 Carnegie Robotics Active & Passive Stereo 13 • Active Stereo: Correspondence problem aided through projection of arbitrary texture into the scene. • Advantages: • Very high resolution and density • Instantaneous horizontal & vertical FOV • Flexible FOV options (from 130° to 10°) • Monochrome or color images are co-registered with 3D data • Many “camera” parameters • Passive solution is immune to obscurant in the environment • Solid state — no moving parts • Downsides: • Data density dependent on scene texture or pattern projection • Non-linear error model with respect to distance • Double shadow behind positive obstacles • Computationally expensive process increases BOM cost
  14. 14. Copyright © 2019 Carnegie Robotics Sensor Error Models 14 Understanding the stereo error model is key to effective use. It is VERY different than every other sensor. Stereo accuracy here assumes matches with 1/2 pixel of accuracy. That is common in real world environments, but not guaranteed. Accuracy is affected by scene texture.
  15. 15. Copyright © 2019 Carnegie Robotics LIDAR & Stereo for Mobile Footstep Planning 15 Rotating LIDAR Stereo Videos courtesy of MIT’s DARPA Robotics Challenge Team. 3D data is being used to plan foot placement and robot movement over the steps. Both videos are sped up from real-time.
  16. 16. Copyright © 2019 Carnegie Robotics 3D LIDAR Sample Data 16 1: 8 seconds of persisted data 2: 1 second of persisted data, high rotation rate Notice: • Wide FOV • Difficulty in seeing/ identifying person • Density differences horizontally & vertically • Mixed pixel returns
  17. 17. Copyright © 2019 Carnegie Robotics ToF Camera Sample Data: Low Cost 17 1: Axis Colored Scene 2: Intensity Colored Scene Notice: • Low resolution / Low Rate • Narrow FOV • Invisible / bent cart handle. • “Flashlight effect” • Mid-air returns • 1/2 of floor missing
  18. 18. Copyright © 2019 Carnegie Robotics ToF Camera Sample Data: Industrial 18 1 & 2: Axis colored scene Notice: • High resolution / High rate • Distortion with near-field object • Distortion to rear wall with foreground object • Multi-path distortion in floor • Mixed-pixel returns
  19. 19. Copyright © 2019 Carnegie Robotics Stereo Sample Data 19 1: Overview of scene Notice: • High density — can see pieces, can see cart handle • High Rate & Wider FOV • Correlated scene color • Data below ground caused by specular reflections • False “spike” in rear wall caused by cart handle 2: Close-up of chess board • Post-matching cost threshold starts high (all data), is lowered (only high confidence data), and is raised again
  20. 20. Copyright © 2019 Carnegie Robotics Stereo Sample Data 20 IR Pattern Projector turned on and off throughout scene. Scene has RGB colormap of monochrome intensity. Areas of low-confidence have higher-confidence with projector on. This results in lower-noise and more accurate data.
  21. 21. Copyright © 2019 Carnegie Robotics Stereo from Passive Thermal Thermal Stereo Optical Stereo
  22. 22. Copyright © 2019 Carnegie Robotics 22 General Sweet Spots LIDAR ToF Stereo HFOV VFOV Density Range accuracy Min range Max range Data rate Obscurant Cost Sealing Shock/Vibe LIDAR • High speed vehicles — well served by their long range and high accuracy at range • Vehicles with high turning rates — require wide horizontal FOV ToF Camera • Indoor environments • Short range object scanning & gesture recognition Stereo Camera • Outdoor applications with high levels of ambient obscurants • Multi-view applications (overlapping FOVs) • Applications which require a non-emitting solution
  23. 23. Copyright © 2019 Carnegie Robotics Testing 3D Sensors
  24. 24. Copyright © 2019 Carnegie Robotics ToF & Stereo: Outdoors Significant horizontal and vertical field of view (FOV) advantage Much denser data allows smaller objects to be detected No mixed-pixel returns Stereo ToF Both views are nearly top- down RGB Image of Scene
  25. 25. Copyright © 2019 Carnegie Robotics 25 ToF & Stereo: Indoors Stereo allows near and far objects to be detected Wider vertical and horizontal field of view (FOV) allows more understanding of the scene ToF Camera has dense returns on visually featureless drywall No far-range data returned Both 3D views are from same perspective and are above sensor, angled down slightly ToFStereo
  26. 26. Copyright © 2019 Carnegie Robotics 26 ToF Camera: Multi-echo Distortions Reflective surfaces distort ToF camera 3D data. IR energy reflects and scattered returns biases range reading longer than actual geometry Causes ruts in floors and bows in vertical walls Polarization can help mitigate but severely limits maximum range. RGB Image of Scene
  27. 27. Copyright © 2019 Carnegie Robotics 27 Stereo & LIDAR in Obscurant Stereo & Laser Point Clouds Viewed from above Colorized Stereo Point Cloud 50 m Tunnel viewed from near camera Colorized Stereo Point Cloud 3D Laser
  28. 28. Copyright © 2019 Carnegie Robotics 28 3D LIDAR: Lighting: Vendor A 300 lux: Fluorescent 5000 lux: Halogen Lexan Mirror Mesh Black Tile Acrylic 8 3 6 9 7 5 8 3 6 9 7 5 White areas shifted 2.5 cm closer to sensor Std. dev. of first row (mm)
  29. 29. Copyright © 2019 Carnegie Robotics Mirror Lexan Polished Brass Mesh Acrylic Black Tile 29 3D LIDAR: Lighting: Vendor B Lexan Mirror Mesh Black Tile Acrylic 10 7 7 9 10 8 11 9 10 12 13 12 High IR content of Halogen increases noise additional 40% Std. dev. of first row (mm) 50% more noise than Vendor A 300 lux: Fluorescent 5000 lux: Halogen
  30. 30. Copyright © 2019 Carnegie Robotics 30 ToF Camera: Lighting Lexan Mirror Mesh Black Tile Acrylic High IR content of Halogen increases noise by 2x to 5x Higher range noise on dark objects Mirror Lexan Polished Brass Mesh Acrylic Black Tile 300 lux: Fluorescent 5000 lux: Halogen
  31. 31. Copyright © 2019 Carnegie Robotics 31 Stereo Camera: 7cm Baseline Range Accuracy 4 4 4 4 4 4 9 9 12 10 8 10 Higher range noise on low-texture objects Std. dev. of first row (mm) Mirror Lexan Black Tile Polished Brass Mesh Acrylic 0.4 meter 1.2 meter
  32. 32. Copyright © 2019 Carnegie Robotics 32 3D LIDAR: Cost vs Performance Price X Price 3X • Test is static scan of flat surface, 5 cm to 140 cm in 5 cm increments • Datasets are from the same vendor, different models of LIDAR • Very different scan density — can be known from datasheet • Range noise, distortions, & localized errors cannot be known from datasheet Error to Color Mapping 0.00 cm: Blue 0.50 cm: Green 1.00 cm: Yellow 1.25 cm: Orange 1.50+ cm: Red
  33. 33. Copyright © 2019 Carnegie Robotics 33 3D LIDAR: Error Changes Over Time • Every laser (even same SKU) is different • Lasers change over time • Error can vary by emission angle & range Unit A Unit A Unit BJanuary July Error to Color Mapping 0.00 cm: Blue 0.50 cm: Green 1.00 cm: Yellow 1.25 cm: Orange 1.50+ cm: Red
  34. 34. Copyright © 2019 Carnegie Robotics 34 Stereo: Lens Flare • Lens flare can cause loss of 3D data due to over-exposure of that region of the scene. • It is possible for flare to be similar enough between left and right cameras to result in 3D matches. More likely with Block Matching and with lower resolution stereo system. • Can also affect ToF camera systems.
  35. 35. Copyright © 2019 Carnegie Robotics 35 3D Resolution Testing In some applications XYZ resolution matters as much as depth accuracy. How do you measure 3D resolution? Take cues from 2D camera testing. • Raised Relief 3D Resolution Target • Derived from USAF 1951 resolution target • THE standard image quality test for 60 years
  36. 36. Copyright © 2019 Carnegie Robotics 36 3D Resolution Testing Process Capture Image Identify and Isolate Target Outputs: • Colorized Depth Image • Colorized Depth Error Image • RMS error on features and between features for each grouping size
  37. 37. Copyright © 2019 Carnegie Robotics 37 3D Resolution: Stereo Error Image Range Image
  38. 38. Copyright © 2019 Carnegie Robotics 38 3D Resolution: Industrial ToF Camera Error Image Range Image
  39. 39. Copyright © 2019 Carnegie Robotics 39 3D Resolution: Low Cost ToF Camera Error Image Range Image
  40. 40. Copyright © 2019 Carnegie Robotics 40 Take Aways There is no perfect sensor. They all have limitations and edge cases. Look for published test data. If none is available: • Ask the sensor manufacturer, • Collect it yourself, or • Turn to those who have past experience in integration and downstream software. It is critical to understand the performance & non-performance criteria that matter for your application. • Those weights must drive your sensor selection process. • You may have to sacrifice one or more lesser characteristics to meet the ones which really matter to you.
  41. 41. Copyright © 2019 Carnegie Robotics 41 Resources Companies & Products Carnegie Robotics MultiSense IFM O3D SICK 3vistor-T ToF Camera pmd pico ToF Camera ZED Camera Orbbec Persee Structure Sensor SICK TiM / LMS LIDARs Velodyne 3D LIDARs Hokuyo 2D & 3D LIDAR Carnegie Robotics Intel RealSense nerian Karmin2 Stereo Camera iDS Ensenso Roboception rc_visard SICK PLB500 Stereo Camera LIDAR ToF Camera