For the full video of this presentation, please visit: http://www.embedded-vision.com/platinum-members/qualcomm/embedded-vision-training/videos/pages/may-2016-embedded-vision-summit-mangen For more information about embedded vision, please visit: http://www.embedded-vision.com Michael Mangen, Product Manager for Camera and Computer Vision at Qualcomm, presents the "High-resolution 3D Reconstruction on a Mobile Processor" tutorial at the May 2016 Embedded Vision Summit. Computer vision has come a long way. Use cases that were previously not possible in mass-market devices are now more accessible thanks to advances in depth sensors and mobile processors. In this presentation, Mangen provides an overview of how we are able to implement high-resolution 3D reconstruction – a capability typically requiring cloud/server processing – on a mobile processor. This is an exciting example of how new sensor technology and advanced mobile processors are bringing computer vision capabilities to broader markets.

1. 1
High-resolution 3D Reconstruction
on a Mobile Processor
Michael Mangan
Senior Product Manager
Qualcomm Technologies, Inc.
May 3, 2016

2. 2
30
years of driving
the evolution
of wireless
#1
in 3G/4G
LTE modem
#1
in RF
Source: Qualcomm Incorporated data. Currently, Qualcomm semiconductors are products of Qualcomm Technologies, Inc. or its subsidiaries
IHS, Jan. ’16 (RF); Strategy Analytics, Dec. ’15 (modem, AP)

3. 3
Qualcomm® Snapdragon™ Chipsets
drive new experiences
Context aware
computing
Machine learning
Computing
performance
VR / AR - beyond small screen
360 degree camera
3D and low-light photography
Security
Biometric sensor
Virtual SIM/Multiple devices
Ultra HD VoLTE / audio quality
4G+
Wi-FiSuperior converged connectivity
Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.
Gaming

4. 4
What is Active Depth Capture?
Depth provides z-dimension to a scene; a photograph provides only x-y information.
Two ways to capture depth information from a scene or object:
Passive Depth Capture:
(No IR Transmitter)
• Stereo RGB cameras can passively
generate a depth map of a scene.
• Baseline separation between the cameras
causes parallax between the two received
images.
• Parallax can be used to infer a disparity
estimate, which in turn is used to
generate a depth map.
Active Depth Capture:
(IR Transmitter)
• IR laser transmits, various
techniques are used to infer depth
from the reflected laser.
» Time of Flight
» Active Stereo
» Structured Light

5. 5
Depth from Structured Light—
Technology Overview
Depth information is generated
using a structured light sensor
• Coded pattern is projected onto the scene
using near infrared (NIR) light
• NIR camera receives the reflected,
distorted pattern
• Codes in the received image are matched
against known codes in the transmitted
pattern
• Depth at each code location estimated
from the disparity between original and
received code positions, leading to
a dense depth map
NIR image
Depth map
coded pattern
transmitter receiver

6. 6
Scanner Flow in Action

7. 7
Scanner Block Diagram
Scan
Starts
Color + Depth
(Structure Light Depth
Based Generation)
Live 3D
Renderer/Viewer
USER MOVES USER STOPS
Scan
Finishes
USE CASE:
3D Printing, Social
Networking, Gaming
Avatars, etc.
Computer Vision Based
Initial Pose Estimation
Inertial Motion
Sensor Fusion
Bundle
Adjustment
HD Texture
Generation
3D Mesh
Generation
Color
Correction
TRACKING/ALIGNMENT

8. 8
Scanner System Architecture
3D Scanner Application
RGBD Image Grabber
Camera 2 APIDepth JNI 3D Scanner JNI
Depth Engine
(DSP/HVX)
RGB
Grabber
NIR
Grabber
3D Scanner Engine
(CPU/GPU)
SysFS Camera HAL Camera HAL
Raw
RGB Data
Raw
NIR Data
Driver
Laser NIR Camera
RGB
Camera
Active Sensing Module
Note: Arrows indicate
dependency, not dataflow
Apps(Java)Middleware(C++)Drivers(C)Hardware

9. 9
3DR Workload Summary—
Running on Snapdragon 820
3D Reconstruction requires running
several computational demanding
processes simultaneously:
1. Camera Pose Tracking
2. Sensor Fusion
3. Bundle Adjustment
4. Rendering
5. Mesh Generation
6. Texture Mapping
7. Structured Light Sensor Decoding
Thanks to the heterogeneous computational
framework of the Snapdragon 820, we are able
to do all of this at 15 FPS:
Cryo—CPU/Neon:
• Pose Tracking
• Bundle Adjustment
• Sensor Fusion
• Mesh Generation
Adreno—GPU:
• Rendering
• Texture Mapping
Hexagon—DSP/HVX:
• Depth from Structured
Light
3DR powered by
Snapdragon 820
Spectra ISP:
• RGB sensor processing
• Depth sensor interface

10. 10
Highest quality 3DR requires
great HW & SW. Efficient CV
SW algorithms, operating with
accurate depth sensors, &
power efficient processors,
bring commercial grade 3DR
to mobile platforms.
Lessons Learned
Running 3DR on mobile
requires tuning algorithms for
power as well as performance.
Power efficient heterogeneous
processors are mandatory for
3DR to run within mobile power
and thermal envelopes.
The heterogeneous
processing cores on
Snapdragon 820, enable a
high-quality, 3DR experience
on mobile platforms.

11. 11
3DR Algorithmic Details

12. 12
Scanner Block Diagram
Scan
Starts
Color + Depth
(Structure Light Depth
Based Generation)
Live 3D
Renderer/Viewer
USER MOVES USER STOPS
Scan
Finishes
USE CASE:
3D Printing, Social
Networking, Gaming
Avatars, etc.
Computer Vision Based
Initial Pose Estimation
Inertial Motion
Sensor Fusion
Bundle
Adjustment
HD Texture
Generation
3D Mesh
Generation
Color
Correction
TRACKING/ALIGNMENT

13. 13
Based on the Iterative Closest Point (ICP) Concept, minimize the sum of pixel
intensity differences (errors) and the sum of depth errors to align Images
𝑐𝑜𝑠𝑡 = 𝑃𝑖𝑥𝑒𝑙 𝐼𝑛𝑡𝑒𝑛𝑠𝑖𝑡𝑦 𝐸𝑟𝑟𝑜𝑟 2
+ 𝜆 𝑃𝑖𝑥𝑒𝑙 𝐷𝑒𝑝𝑡ℎ 𝐸𝑟𝑟𝑜𝑟 2
Pixel Intensity Error Depth Error
• F. Steinbruecker,et al., “Real-Time Visual Odometry from Dense RGB-D Images”, ICCV 2011
• C. Kerl et al., “Dense Continuous-Time Tracking and Mapping with Rolling Shutter RGB-D Cameras”, ICCV 2015
Computer Vision Based
Pose Estimation (6-DOF)

14. 14
Flow
Reference Image
Current Image
Warp
subtract
Repeat to
Minimize Error
– =
Warped Image Error Image
Computer Vision Based
Pose Estimation (6-DOF)

15. 15
Example
Computer Vision Based
Pose Estimation (6-DOF)

16. 16
The Vision Pose will likely contain some errors.
• One example is lack of geometrical and textural structures
This can be overcome by fusing the vision pose with the Inertial Motion Unit (IMU) of the tablet
Using The Extended Kalman Filter (EKF) concept, one can predict poses from the IMU.
These are then fused in the update step of EKF to obtain the fused pose estimate
Motion Sensor Fusion
• M. Li et al., “3-D motion estimation and online temporal calibration for camera-IMU systems”, ICRA 2013
• S. Weiss et al., “Real-Time Metric State Estimation for Modular Vision-Inertial Systems. in IEEE International Conference on Robotics and Automation ”, ICRA 2011
Extended
Kalman Filter
(Predict)
Vision Based
Pose
Estimation
Extended
Kalman Filter
(Update)
Gyro
Accelerometer

17. 17
Fused Poses need to be refined in order
to reduce the visual errors.
• Reason: Poses are being computed locally,
“between consecutive frames”
We use bundle adjustment to find optimal
global or semi-global poses
• Construct links (red lines) between captured frames
(blue nodes). Links are established if the re-projection
between captured images is above a certain threshold
• Jointly optimize the connected nodes
Bundle Adjustment
• V. Indelman et al., “Incremental Light Bundle Adjustment for Robotics Navigation”, IROS 2013
• R. Newcombe et al., “KinectFusion: Real-Time Dense Surface Mapping and Tracking”, IEEE ISMAR 2011
• K. Konolige et al., “FrameSLAM: from Bundle Adjustment to Realtime Visual Mappping”. IEEE Transactions on Robotics 2008
-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
-0.2
0
0.2
0.4
0.6
0.8
1
1.2

18. 18
Having computed the 3D points, we need to generate the 3D surface mesh that best describes the
scene while reducing the noise
Many methods are available in the literature for surface reconstruction: Moving Least Squares
(MLS), TSDF & Poisson. Any can be used in theory. TSDF is the least computational demanding,
MLS and Poisson are more demanding
These are then followed by the marching cubes concept to generate the mesh
Surface Reconstruction / Mesh Generation
• S. Fleischmann et al., “Robust Moving Least-squares Fitting with Sharp Features”, ACM SIGGRAPH 2005
• M. Kazdan et al., “Poisson Surface Reconstruction”, Symposium on Geometry Processing 2006
• R. Newcombe et al., “KinectFusion: Real-Time Dense Surface Mapping and Tracking”, IEEE ISMAR 2011

19. 19
Captured color images can suffer from casting due to many reasons like different lighting
sources. We need to correct that so that the overall color of the 3D model is in harmony
Solution: Estimate Color Casts & Remove them
• Gray points provide best estimate about color
• Estimate gray pixels & shift the appropriate channel gain to bring them to neutral gray
• Repeat until convergence
Color Correction
• J. Huo et al., ‘”Robust Automatic White Balance Algorithm Using Gray Color Points in Images”, IEEE Trans. Consumer Electronics, 2006
BEFORE
AFTER

20. 20
The captured images need to be joined in one or more images called Texture Maps
Texture mapping can be thought of as “3D stitching of the images on the 3D model”
Obtaining the Texture Map consists in general of two steps:
• Determine where to put the pixels on a 3D model (texture coordinates)
• Determine what is the color of the pixel given a sequence of input images
Texture Mapping
• P. Debevec et al., “Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping”, Eurographics Rendering Workshop 1998
• M. Waechter et al., “Let There Be Color! Large-Scale Texturing of 3D Reconstructions”, ECCV 2015
Input Camera Images Output Texture Map Colored 3D Model
Using the Texture Map

21. 21
Some 3DR Examples

22. 22
Using our system we can scan
a small toy, human face/body
or an object
All of this can happen easily
on the Snapdragon 820, thanks
to its powerful heterogeneous
computational framework
Some Results

23. Thank you
Follow us on:
For more information, visit us at:
www.qualcomm.com & www.qualcomm.com/blog
Nothing in these materials is an offer to sell any of the components or devices referenced herein.
©2016 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Qualcomm and Snapdragon are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Why Wait is a trademark of Qualcomm
Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners.
References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within
the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent
portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s
engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.
23