Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

"Computer-vision-based 360-degree Video Systems: Architectures, Algorithms and Trade-offs," a Presentation from videantis


Published on

For the full video of this presentation, please visit:

For more information about embedded vision, please visit:

Marco Jacobs, VP of Marketing at videantis, presents the "Computer-vision-based 360-degree Video Systems: Architectures, Algorithms and Trade-offs" tutorial at the May 2017 Embedded Vision Summit.

360-degree video systems use multiple cameras to capture a complete view of their surroundings. These systems are being adopted in cars, drones, virtual reality, and online streaming systems. At first glance, these systems wouldn’t seem require computer vision since they’re simply presenting images that the cameras capture. But even relatively simple 360-degree video systems require computer vision techniques to geometrically align the cameras – both in the factory and while in use. Additionally, differences in illumination between the cameras cause color and brightness mismatches, which must be addressed when combining images from different cameras.

Computer vision also comes into play when rendering the captured 360-degree video. For example, some simple automotive systems simply provide a top-down view, but more sophisticated systems enable the driver to select the desired viewpoint. In this talk, Jacobs explores the challenges, trade-offs and lessons learned while developing 360-degree video systems, with a focus on the crucial role that computer vision plays in these systems.

Published in: Technology
  • Be the first to comment

"Computer-vision-based 360-degree Video Systems: Architectures, Algorithms and Trade-offs," a Presentation from videantis

  1. 1. Copyright © 2017 videantis GmbH 1 Marco Jacobs May 2017 Computer-vision-based 360-degree Video Systems: Architectures, Algorithms and Trade-offs
  2. 2. Copyright © 2017 videantis GmbH 2 About videantis #1 vision processor 100% vision company automotive since 2008 Algorithm partner
  3. 3. Copyright © 2017 videantis GmbH 3 360-degree video systems Automotive Consumer Capture Vision Render Calibration to align images Blindness detection 3D from 2D Obstacle detection
  4. 4. Copyright © 2017 videantis GmbH 4 Extend visibility, warn driver and autonomy Problem: Limited visibility Solution: Surround view & automation
  5. 5. Copyright © 2017 videantis GmbH 5 360-video processing pipeline Display path Vision path Capture Render and Act Display Calibration Render Camera parameters Color & intensity Warp Color correct Stitch +Graphics Vision 2-4 Cameras Wide angle lens Image sensor ISP Video codec Analytics Blindness detect Crossing traffic alert Pedestrian detection Parking assist Blind spot assist Lane recognition ETC
  6. 6. Copyright © 2017 videantis GmbH 6 Manufacturing tolerances Mechanical shifts Vehicle load Tire pressure Calibration – camera parameters change Accurate camera parameters needed to generate surround view
  7. 7. Copyright © 2017 videantis GmbH 7 Calibration – required for aligning the images Camera manufacturing Assembly line Vehicle power on While driving Intrinsics Extrinsics Photometric alignment Markers Markerless focal length sensor resolution principal point lens distortion camera's location in the world: translation (x, y, z) rotation (pitch, yaw, roll) use patterns that can easily be found 1. Matching image features 2. Using vanishing point Match color and intensity Changes based on sensor gain and scene illumination Different calibration techniques for production and in field
  8. 8. Copyright © 2017 videantis GmbH 8 Input information • Static: initial extrinsic parameters • Dynamic: CAN data (speed and steering angle) Computing the vanishing point • Sparse optical flow to find motion vectors • Vanishing point where flow vectors intersect Benefits • Single camera, not multiple • Less compute compared to feature matching • Estimates ground plane location Results • ~1 minute at 5 – 120 km/h • <1 degree accuracy Calibration – Extrinsics using vanishing point dy (Pitch) dx (Yaw) dr (Roll)
  9. 9. Copyright © 2017 videantis GmbH 9 Rendering stage uses output from calibration +3D model Calibration Mesh geometry Color correction data Alpha blending data 2D / 3D model params Render Warp Color correct Blend & stitch 2D / 3D Graphics
  10. 10. Copyright © 2017 videantis GmbH 10 Projection onto the ground plane Projection onto ground plane flattens & stretches objects Image formed in cameraLight from surface Projected pixel
  11. 11. Copyright © 2017 videantis GmbH 11 Projection onto a bowl shape Image formed in cameraLight from surface Projected pixel Projection onto bowl gives more accurate images
  12. 12. Copyright © 2017 videantis GmbH 12 • Warping for surround view systems: • Projections onto surface implemented with generic image warp • Uses camera calibration results • Wide angle lens correction single camera view: • Accounts for camera (radial) lens distortion, translation, scale, and yaw, pitch, roll • Single camera view often shown in addition • Generic mesh-based implementation: • Mesh of points defines geometric transformation • Similar to OpenCV remap() function Warping algorithm (1/2)
  13. 13. Copyright © 2017 videantis GmbH 13 Output from intrinsic & extrinsic calibration: • Surround view mesh per camera • Alpha blending per node for camera overlap • Color correction per mesh node to account for illumination Parameters: • 8-bit or 16-bit color (for wide dynamic range) • Bilinear or bicubic interpolation Special cases: • Aliasing when scaling down – low pass filter • Blurring when scaling up – use higher order interpolation Warping algorithm (2/2)
  14. 14. Copyright © 2017 videantis GmbH 14 Trend to incorporate more ADAS functions: • Pedestrian detection for backover protection • Autonomous parking (free space detection, parking marker detection, path planning) • Lane detection for lane departure warning, blind spot assist We will highlight a few: • Blindness detection • Crossing traffic alert • Obstacle detection using structure from motion ADAS functions in 360-video systems
  15. 15. Copyright © 2017 videantis GmbH 15 Algorithm: • Separate image into tiles • Classify each tile using image quality metrics • Sharpness, Laplacian, gradients, Local Binary Patterns • Support vector machine to find most important features • Temporally filter grid to avoid state changes due to noise Challenges: • Homogeneous areas like sky and ground: • Smooth gradient values, while dirt usually does not • Low light performance. Detection may be turned off in dark Blindness detection Blindness detect camera clean?
  16. 16. Copyright © 2017 videantis GmbH 16 Structure from motion: • 3D point cloud from monocular camera on moving vehicle, static objects Point cloud analysis in order to: • Warn for objects close to car • Automated parking • Adjust surround view generation Enables removal of ultrasound sensors Structure from motion (1/2) Structure from motion 3D point cloud car speed & angle Point cloud analysis
  17. 17. Copyright © 2017 videantis GmbH 17 Algorithm: • Vehicle speed received from CAN data • Uses sparse optical flow, clustering, and 3D reconstruction • Operates on fisheye images Challenges: • Need accurate intrinsic and extrinsic camera calibration. Small errors affect height and distance estimations greatly • Areas in images that don’t have features (sky, walls, road) • Moving obstacles Structure from motion (2/2) (R,T) c0 c1 x1 x2 P P Find camera motion Estimate 3D location Detect feature points Sparse optical flow Estimate 3D location
  18. 18. Copyright © 2017 videantis GmbH 18 Crossing traffic alert (1/2) Exiting parking space Exiting driveway Crossing traffic alert time to collisioncar speed
  19. 19. Copyright © 2017 videantis GmbH 19 Overview of method: • Single monocular camera • Vehicle speed received from CAN data • Uses sparse optical flow, clustering, and structure from motion • After clustering, only track features inside region of interest Working conditions: • No need to know type of vehicle, based on movement of obstacles • Low vehicle speed, up to 5 km/hr Crossing traffic alert (2/2) Sparse optical flow (LK) Subtract vehicle egomotion Cluster motion vectors Estimate direction & velocity Time to collision < threshold Driver warning or apply brakes Camera images Vehicle speed Feature detection
  20. 20. Copyright © 2017 videantis GmbH 20 System architecture options DisplayHead unit Camera Lens Sensor ISP Camera Lens SensorCamera Lens SensorCamera Lens Sensor ISP ISP ISP 360 view gen Vision Central head unit does it all
  21. 21. Copyright © 2017 videantis GmbH 21 System architecture options DisplaySurround ECU Camera Lens Sensor ISP Camera Lens SensorCamera Lens SensorCamera Lens Sensor ISP ISP ISP 360 view gen Head unit Vision Surround view ECU sends rendered images to head unit
  22. 22. Copyright © 2017 videantis GmbH 22 System architecture options DisplaySurround ECU 360 view gen Head unit Camera Lens Sensor ISP Camera Lens Sensor ISP Camera Lens Sensor ISP Camera Lens Sensor ISP Vision Vision Computer vision in cameras and ECU
  23. 23. Copyright © 2017 videantis GmbH 23 • Even simple 360-video requires computer vision for calibration • During manufacturing and in the field • Automotive 360-video rapidly adding more computer vision features • Cameras are there, relatively inexpensive to add embedded vision • Cameras are replacing other sensors like ultrasound • Integration other vision systems in the car (e.g., front camera) • System architectures still evolving Summary Huge opportunity for 360-vision/video, both in automotive and consumer applications
  24. 24. Copyright © 2017 videantis GmbH 24 • Feature-based alignment chapter in Computer Vision: Algorithms and Applications by Richard Szeliski • • Structure from Motion tutorial from EV Summit 2015: • • • Resources
  25. 25. Copyright © 2017 videantis GmbH 25 • Demos at booth 804 • Thank You!