Computer vision and machine learning techniques can enhance user experiences in various industries like mobile, drones, IoT, cameras, VR/AR, and automotive. These techniques include object detection, face recognition, and 3D reconstruction. Qualcomm's Snapdragon processors support running computer vision algorithms for applications such as visual inertial odometry for VR/AR motion tracking, foveated rendering using eye tracking to reduce power in VR, video analytics for IP cameras and drones, and 3D scanning from depth sensors. The heterogeneous computing capabilities of Snapdragon allow running these computationally intensive workloads efficiently at the edge.

1. Copyright © 2017 Qualcomm Technologies, Inc. 1
Mike Mangan, Product Manager Staff
May 2017
Computer Vision and Machine
Learning at the Edge

2. Copyright © 2017 Qualcomm Technologies, Inc. 2
Computer Vision & Machine Learning Enhance
UX in many different industries
Mobile
Drones
IOT
Cameras
VR/AR
Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc.
Automotive

3. Copyright © 2017 Qualcomm Technologies, Inc. 3
CV & ML makes cameras smarter, making end-user experiences more interesting
Computer Vision Industry Overview
HMDs
In this presentation we’ll
explain how:
• Eye-tracking
• CV Pose estimation
help VR/AR
IOT Cameras
(Security, Sports, etc.)
DronesSmartphones
• CV Enhancing these markets:
• Tracking
• 3D reconstruction
• Face recognition

4. Copyright © 2017 Qualcomm Technologies, Inc. 4
How does CV enhance VR / AR?

5. Copyright © 2017 Qualcomm Technologies, Inc. 5
For accurate and intuitive interactions with the virtual world
CV provides precise motion tracking of head
movements
• “In which direction am I looking”
• Detect rotational movement
• Main benefit: look around the virtual world
from a fixed point
3 degrees of freedom (3-DOF)
X
Z
Y
Pitch
Yaw
Roll
• “Where I am and in which direction am I
looking”
• Detect rotational movement and translational
movement
• Main benefit: move freely in the virtual world
and look around corners
6 degrees of freedom (6-DOF)

6. Copyright © 2017 Qualcomm Technologies, Inc. 6
Visual Inertial Odometry (VIO) for rapid and accurate 6-DOF pose estimation
Achieving precise head motion
tracking on the device
6-DOF
position &
orientation
(aka “6-DOF pose”)
Captured from tracking
camera image sensor at
~30 fps
Monocular or
binocular
camera data
Accelerometer &
gyroscope data
Sampled from external
sensors at 800 / 1000 Hz
Camera feature processing
Inertial data processing
Snapdragon “VIO” subsystem
New frame
accurately displayed
Qualcomm® Hexagon™
DSP algorithms
• Camera and inertial sensor
data fusion
• Continuous localization
• Accurate, high-rate “pose”
generation & prediction
Qualcomm Hexagon is a product of Qualcomm Technologies, Inc.

7. Copyright © 2017 Qualcomm Technologies, Inc. 7
CV eye-tracking lowers power and enhances
the VR experience
• Rather than rendering with high
resolution throughout an image,
just render high where the eye is
fixated
• We use Eye-tracking algorithms to
know where an eye is looking
• The GPU renders a small
rectangle at a high resolution and
the rest of the FOV at a lower
resolution
• Foveated rendering will help
minimize power, while improving
performance and visual quality
High resolution everywhere
Foveated rendering based on the eyes being fixated on the paraglider
High resolution
Low resolution

8. Copyright © 2017 Qualcomm Technologies, Inc. 8
Lag prevents immersion and can cause discomfort
Minimizing motion to photon latency is crucial
Low latency Noticeable latency

9. Copyright © 2017 Qualcomm Technologies, Inc. 9
Many workloads must run efficiently for an immersive VR experience
An comprehensive approach is required
to minimize latency
9
Visual processing
• View generation
• Render / decode
Motion detection
• Sensor sampling
• Sensor fusion
Display
• Image correction
• Quality enhancement and display
“Motion” “Photon”
(new pixels’ light emitted from the screen)
Total time (motion to photon latency) for all steps above must be less than 20 milliseconds

10. Copyright © 2017 Qualcomm Technologies, Inc. 10
CV Tracking at the Edge:
IP Cameras, Drones, Sports Cameras

11. Copyright © 2017 Qualcomm Technologies, Inc. 11
Video analytics with background subtraction and motion blob/object detection and tracking

12. Copyright © 2017 Qualcomm Technologies, Inc. 12Video analytics with object detection and classification in DCN

13. Copyright © 2017 Qualcomm Technologies, Inc. 13Qualcomm Proprietary and Confidential
Enhanced automatic zoom based on T2T
Tracking in drone based on T2T

14. Copyright © 2017 Qualcomm Technologies, Inc. 14
Face Recognition —
At the Edge

15. Copyright © 2017 Qualcomm Technologies, Inc. 15
Face Recognition

16. Copyright © 2017 Qualcomm Technologies, Inc. 16
3D Reconstruction—
At the Edge

17. Copyright © 2017 Qualcomm Technologies, Inc. 17
Using:
1. QTI 3DR software, AND
2. A high resolution depth
scanner system we can
scan a small toy, human
face/body
or an object
All of this is designed to
happen easily on the
Snapdragon, thanks to its
powerful heterogeneous
computational framework
Some 3DR Scanning Results

18. Copyright © 2017 Qualcomm Technologies, Inc. 18
3D scanner block diagram
Scan
starts
Scan
finishes
USE CASE:
3D printing,
social networking,
gaming avatars, etc.
Color
correction
3D mesh
generation
HD texture
generation
USER MOVES USER STOPS
Color + Depth
(Structure light depth
based generation)
Computer vision based
initial pose estimation
Inertial motion
sensor fusion
Bundle
adjustment
Live 3D
renderer/viewer
TRACKING/ALIGNMENT

19. Copyright © 2017 Qualcomm Technologies, Inc. 19
3D scanner software architecture
Active Sensing Module
RGBD Image Grabber
Depth Engine
(DSP/HVX)
3D Scanner Engine
(CPU/GPU)
SysFS Camera HAL Camera HAL
Raw
RGB Data
Raw
NIR Data
Driver
Laser NIR Camera RGB Camera
Note: Arrows indicate dependency,
not dataflow
Apps(Java)Middleware(C++)Drivers(C)Hardware
3D Scanner Application
Camera 2 APIDepth JNI 3D Scanner JNI
RGB
Grabber
NIR
Grabber

20. Copyright © 2017 Qualcomm Technologies, Inc. 20
3D Reconstruction requires running
several computational demanding
processes simultaneously:
3DR workload summary—running on
Snapdragon 835
Qualcomm Kryo, Qualcomm Adreno, and Qualcomm Spectra are products of Qualcomm Technologies, Inc.
Thanks to the heterogeneous computational
framework, the Snapdragon 835 is engineered to do all
of this at 15 FPS:
Qualcomm®
Kryo™—
CPU/Neon:
• Pose tracking
• Bundle adjustment
• Sensor fusion
• Mesh generation
Qualcomm®
Adreno™—
GPU:
• Rendering
• Texture
mapping
Hexagon Vector
Extensions (HVX):
• Depth from
structured light
Qualcomm
Spectra™
ISP:
• RGB sensor
processing
• Depth sensor
interface
1. Camera pose tracking
2. Sensor fusion
3. Bundle adjustment
4. Rendering
5. Mesh generation
6. Texture mapping
7. Structured light
sensor decoding
3DR powered by
Snapdragon 835

21. Follow us on:
For more information, visit us at:
www.qualcomm.com & www.qualcomm.com/blog
All data and information contained in or disclosed by this document is confidential and proprietary information of Qualcomm Technologies, Inc.
and/or its affiliated companies and all rights therein are expressly reserved. By accepting this material the recipient agrees that this material
and the information contained therein is to be held in confidence and in trust and will not be used, copied, reproduced in wh ole or in part, nor
its contents revealed in any manner to others without the express written permission of Qualcomm Technologies, Inc. Nothing i n these
materials
is an offer to sell any of the components or devices referenced herein.
©2017 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved.
Qualcomm, Snapdragon, Kryo, Adreno and Hexagon are trademarks of Qualcomm Incorporated, registered in the United States and o ther
countries. Qualcomm Spectra is a trademark of Qualcomm Incorporated. Other products and brand names may be trademarks or registered
trademarks of their respective owners.
References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsi diaries
or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business,
QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated,
operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and s ubstantially all
of its product and services businesses, including its semiconductor business, QCT.
Thank you