"Snapdragon Hybrid Computer Vision/Deep Learning Architecture for Imaging Applications," a Presentation from Qualcomm
•
4 likes•1,832 views
For the full video of this presentation, please visit: https://www.embedded-vision.com/platinum-members/qualcomm/embedded-vision-training/videos/pages/may-2019-embedded-vision-summit-lay For more information about embedded vision, please visit: http://www.embedded-vision.com Robert Lay, Computer Vision and Camera Product Manager at Qualcomm, presents the "Snapdragon Hybrid Computer Vision/Deep Learning Architecture for Imaging Applications" tutorial at the May 2019 Embedded Vision Summit. Advances in imaging quality and features are accelerating, thanks to hybrid approaches that combine classical computer vision and deep learning algorithms and that take advantage of the powerful heterogeneous computing capability of Qualcomm Snapdragon mobile platforms. Qualcomm’s dedicated computer vision hardware accelerator, combined with the flexible deep learning capabilities of the Qualcomm AI Engine, help developers meet performance goals for the most complex on-device vision use cases with lower power consumption and processor load. In this talk, Lay reviews the Snapdragon heterogeneous architecture and how Qualcomm's dedicated computer vision processor and tools accelerate the realization of high performance, efficient imaging and vision applications.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to "Snapdragon Hybrid Computer Vision/Deep Learning Architecture for Imaging Applications," a Presentation from Qualcomm
Similar to "Snapdragon Hybrid Computer Vision/Deep Learning Architecture for Imaging Applications," a Presentation from Qualcomm (20)
More from Edge AI and Vision Alliance
More from Edge AI and Vision Alliance (20)
Recently uploaded
Recently uploaded (20)
"Snapdragon Hybrid Computer Vision/Deep Learning Architecture for Imaging Applications," a Presentation from Qualcomm
- 1. © 2019 Qualcomm Qualcomm® Snapdragon™ Hybrid Computer Vision/ Deep Learning Architecture for Imaging Applications Robert Lay, Product Manager, Staff Qualcomm Technologies, Inc (QTI) May 2019 Qualcomm Snapdragon is a product of Qualcomm Technologies, Inc. and or its subsidiaries.
- 2. © 2019 Qualcomm 2 Today and tomorrow’s imaging challenges The challenges of Camera workloads Very compute intensive Complex algorithms and Data flows Many concurrencies Low latency, high bandwidths Must be thermally efficient for sleek, ultra-light designs Requires long battery life for all-day use Storage / Memory Compute/ Bandwidth limitations
- 3. © 2019 Qualcomm Extreme Speed Boost Up to 4x* Power Savings Multi-object Classification Multi-Object Tracking Object Segmentation Depth Sensing at 60fps 6DoF XR Body Tracking CV Stabilization QTI has enabled the world’s first announced CV-ISP Computer Vision HW Acceleration A Powerful Heterogeneous Platform for Advanced Imaging Experiences *vs running on cDSP, Q6 Qualcomm Adreno, Qualcomm Hexagon, Qualcomm Kryo, Qualcomm Spectra and Qualcomm Snapdragon are products of Qualcomm Technologies Inc., and/or its subsidiaries.
- 4. © 2019 Qualcomm 4 Heterogeneous Approach at QTI to Computer Vision ISP Algorithmic advancements Neural network algorithm design optimized for hardware Optimization for space and runtime efficiency Software tools PC and target based simulators Flexible SW frameworks Sample code Efficient hardware Efficient architecture design Selecting the right compute engine for the right task
- 5. © 2019 Qualcomm Qualcomm® SpectraTM CV HW Processor - CVP CVP Acceleration Applications and Benefits Preprocessing & Detection • Pyramidal scaler • Downscaler • Feature descriptor Tracking/ Motion processing • Optical flow • Feature point extraction • Binary feature descriptor and inline descriptor matching • Normalized cross correlation (NCC) Classification • Non-linear classifier Geometric Correction Engine • Low-power high-quality image warping and alignment engine Depth Processing • Depth from Stereo Embedded Processor • Scheduling and concurrency handling
- 7. © 2019 Qualcomm Example for CVP accelerated real time video bokeh/segmentation Blue blocks accelerated using CVP R+L Images RT Bokeh DL Post-Proc Depth from Stereo DFS Rectification GCE, Scalers DL Post Proc Blur Map DL Segmentation RT Segmented RT Recalibration Descriptors, OF Scalers Factory Calibration
- 8. © 2019 Qualcomm Depth sensing at 60fps World’s First announced 4K HDR with portrait mode Example for CVP accelerated real time video bokeh
- 9. © 2019 Qualcomm XR Case: CVP acceleration for 3D scene understanding Book: Ireland, ISBN, <url@amazon> Table: IKEA #, dimensions W x H x D unknown: 3D model Unknown: cube W x H x D Known 3D coordinates of everything Room floor plan and furniture layout
- 10. © 2019 Qualcomm Blue blocks accelerated using CVP 3D reconstruction Camera pose Geometry Semantic information … Viewer perspective 6DOF Depth Image Data Planes Per object extraction Room layout 3D Segmentation DL fusion DL semantic segmentation Obstacle warningIR RGB Mono
- 11. © 2019 Qualcomm 11 XR 6DoF accelerated with CVP Feature Detector and Tracker (Major tasks offloaded to CVP) Extended Kalman Filter (Done in cDSP and optionally mDSP) Predicted 2D feature positions Measured 2D feature position Pose New Feature Inertial Sensor calibration Pose 3D Map Gravity Camera Frames Camera intrinsic & extrinsic calibration Accelerometers Gyroscopes
- 12. © 2019 Qualcomm XR 6DoF accelerated with CVP CVP Left Camera Frames Right Camera Frames DSP Pyramid Scaler 3 Levels Feature Tracking 3 Levels NCC Keypoint Detection Predict features location Camera Pose EKF Update Key Frame? Stereo Matching NCC Coordinates Coordinates
- 13. © 2019 Qualcomm 6DoF Body Tracking Object Detection and Tracking CPU+DSP+GPU CVP 2xPower Savings 4xPower Savings Power savings with CVP* *Qualcomm R&D characterization on Snapdragon 855 mobile platform
- 14. © 2019 Qualcomm 6DoF Body Tracking Object Detection and Tracking CPU+DSP+GPU CVP 39%Compute Savings 96%Compute Savings Compute savings with CVP * *Qualcomm R&D characterization on Snapdragon 855 mobile platform
- 15. © 2019 Qualcomm 15 Conclusions Computer Vision underpins cutting edge imaging experiences Classic camera and video experiences enhanced with: • Motion compensated temporal filtering • Multi-frame techniques • Depth enabled features and algorithms • Video superresolution • Video frame rate upconversion Emerging industries enabled by vision: QTI is taking a heterogeneous approach to computer vision enhanced camera experiences Meeting KPIs by combining general purpose compute engines with dedicated accelerators: IVI/ADASXR IoT Power Latency Processor and memory capacity
- 16. © 2019 Qualcomm 16 Resources Snapdragon Computer Vision SDK Please contact Robert Lay: rlay@qti.qualcomm.com FastCV Computer Vision SDK https://developer.qualcomm.com/software/fastcv-sdk Snapdragon On-Device AI https://www.qualcomm.com/invention/artificial-intelligence https://www.qualcomm.com/snapdragon/artificial-intelligence Qualcomm® Neural Processing SDK https://developer.qualcomm.com/software/qualcomm-neural-processing-sdk Hexagon SDK https://developer.qualcomm.com/software/hexagon-dsp-sdk Qualcomm Neural Processing SDK is a product of Qualcomm Technologies, Inc. and/or its subsidiaries.
- 17. © 2019 Qualcomm CVP SDK for Customer Enablement • Documentation: • CVP 1.0 (SD855) API Document: Available now • Computer Vision Processor (CVP) v 1.x Programming Guide • CVP 2.0 (SD865) API Document: May 2019 • Computer Vision Processor (CVP) v 2.x Programming Guide • Bit-exact PC simulator (Windows and Linux) for algorithm development: • CVP 1.0 simulator: Available now • CVP 2.0 simulator: Q2/2019) • Reference examples code for each of CVP blocks: CVP1.0 now, CVP2.0 Q2/2019 • Offline training tool for Object Detection: CVP1.0 now, CVP2.0 Q3/2019 • This tool works with annotations same as used for training DL networks
- 18. © 2019 Qualcomm Thank you Nothing in these materials is an offer to sell any of the components or devices referenced herein. Qualcomm, Snapdragon, Adreno, Hexagon, Kryo and Qualcomm Spectra are trademarks of Qualcomm Incorporated, registered in the United States and other countries. Other products and brand names may be a trademark or registered trademark of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes Qualcomm’s licensing business, QTL, and the vast majority of its patent portfolio. Qualcomm Technologies, Inc., a wholly-owned subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of Qualcomm’s engineering, research and development functions, and substantially all of its product and services businesses, including its semiconductor business, QCT.