“The Future of AI is Here Today: Deep Dive into Qualcomm’s On-Device AI Offerings,” a Presentation from Qualcomm
•
0 likes•211 views
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2022/06/the-future-of-ai-is-here-today-deep-dive-into-qualcomms-on-device-ai-offerings-a-presentation-from-qualcomm/ Vinesh Sukumar, Senior Director and Head of AI/ML Product Management at Qualcomm, presents the “Future of AI is Here Today: Deep Dive into Qualcomm’s On-Device AI Offerings” tutorial at the May 2022 Embedded Vision Summit. As a leader in on-device AI, Qualcomm is in a unique position to deliver optimized and now personalized AI experiences to consumers, made possible via innovation in hardware technology and investment across the entire software stack. This investment is now deeply rooted in all of our product offerings, spread across multiple verticals from mobile to automotive. In this talk, Sukumar explores the high-performance, low-power Hexagon processor — the core of his company’s latest 7th Generation AI Engine — and shows how the company scales it across the range of products that Qualcomm offers. He also highlights Qualcomm’s investment in advanced techniques such as the latest quantization approaches and neural architecture search to accelerate AI deployment. Finally, he shares details on how his company incorporates these technologies into AI solutions that power Qualcomm’s vision of on-device AI — and shows how these solutions are employed in real-world use cases across many verticals.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to “The Future of AI is Here Today: Deep Dive into Qualcomm’s On-Device AI Offerings,” a Presentation from Qualcomm
Similar to “The Future of AI is Here Today: Deep Dive into Qualcomm’s On-Device AI Offerings,” a Presentation from Qualcomm (20)
More from Edge AI and Vision Alliance
More from Edge AI and Vision Alliance (20)
Recently uploaded
Recently uploaded (20)
“The Future of AI is Here Today: Deep Dive into Qualcomm’s On-Device AI Offerings,” a Presentation from Qualcomm
- 1. The Future of AI is Here Today: Deep Dive into Qualcomm’s On-Device AI Offering Vinesh Sukumar Sr. Director, Product Management (AI/ML) Qualcomm Technologies, Inc.
- 2. Privacy Reliability Low latency Efficient use of network bandwidth Process data closest to the source, complement the cloud Center of Gravity Moving to the Edge... 2 Autonomy Personalization Efficiency Security Historically Increased Demand © 2022 Qualcomm Technologies, Inc.
- 3. At Qualcomm – AI Deployed Across Various Technologies & Verticals 3 Enhance BU VERTICALS Create Superior solutions Enabling developer ecosystems MORE THAN A DECADE Of AI R&D TECHNOLOGIES QUALCOMM® AI ENGINE Software layer Hardware layer Application layer Modem Camera Video Audio Voice AR/VR Windows 11 – PC Computing Smart Camera – IoT Markets ADAS Markets © 2022 Qualcomm Technologies, Inc.
- 5. AI Applications: Across Various Segments Expanding beyond modalities of computer vision to linguistics, communication, commerce and language understanding 5 Mobile IoT Compute Cloud Auto AI Assisted Imaging • AI 3A • Scene-based Camera Selection Image Understanding • Face Detection / Tracking / Features • Object Detection / Tracking • Body Detection / Tracking / Pose • Human Segmentation • Sky Segmentation • Multi-Class Segmentation • Depth Estimation Beautify / Augment • Scene-based Image Enhancement Image Processing • AI based NR or Image SR • Scene-based Camera Selection Audio • Real time language • Natural language processing (NLP) Modem • Parameter optimization • Robust sequence predictions Robotics • Autonomous navigation • Obstacle Avoidance • Picking and Sorting Productivity • Background based noise cancellation on Audio (inbound and outbound) • Segmentation/Blur/Super Resolution on Video • Voice activation without keywords • Face tracking • Smart photo categorization Data Centers • Natural language processing • Computer vision • Recommendation system IVI • Occupancy monitoring system (OMS) • Driver monitoring system (DMS) • Surround perception • Audio Command & Control Retail • Visitor/Face/Gesture Recognition • Object/People Detection and Counting • Barcode decoding • Empty shelf detection • Dwell time Privacy & Security • Automatic screen unlock and login • Privacy alert • Guard mode ADAS (Up to L4) • Highway driving assist • Front collision warning • lane departure, • Traffic jam assist • Auto lane change • Auto lane merge • Traffic light recognition • Construction zones • Urban autonomous driving • Parking assist • Person detection, • Perception • Valet parking • Driver monitoring Transportation • License plate recognition • Face and facial landmark detection • Drowsiness detection Content Creation & Gaming Gaming with gesture control Gaming with voice commands Intelligent highlight videos Game play improvement Edge Compute • Theft detection • Face/body/license plate detection / recognition • Image classification and segmentation Smart Devices • Object/People detection • Speaker detection • Gun shot detection Smart Buildings • People Tracking • Access Control Performance & Efficiency • Power and Screen optimization Manufacturing/Logistics • Predictive maintenance • Energy management with Asset demand Innovation centered around supporting high accuracy, latency & heterogeneity Innovation centered around energy efficiency & personalization Innovation centered around making AI relevant in PC Battery Life Latency Focused Throughput Heavy Concurrently Enabled Few examples
- 6. Challenges of AI Applications 6 Power and thermal efficiency are essential for on-device AI Very compute intensive Large, complicated neural network models Complex concurrencies Always-on Real-time Must be thermally efficient for sleek, ultra-light designs Storage/memory bandwidth limitations Requires long battery life for all-day use Ability to drive high concurrency and utilization Lays the foundation for Qualcomm AI Hardware © 2022 Qualcomm Technologies, Inc.
- 7. AI Hardware
- 8. Vision: Drive Leadership Capability Across All Markets 8 Performance Scalability Co-Design Innovation Invest in performance (Inf/Sec) and Power efficiency (Inf/Sec/W) Leverage existing AI HW engines to scale across various TDP points Feature innovation to drive leadership (Datatypes, Sparsity, Streaming…) HW & SW co-design to make programmability easier (NAS, Compatibility) © 2022 Qualcomm Technologies, Inc.
- 9. 7th Generation : Introducing Qualcomm AI Engine for All Verticals 9 © 2022 Qualcomm Technologies, Inc.
- 10. 7th Generation : Qualcomm AI Engine 10 Large-shared memory Computational performance 2X 2X © 2022 Qualcomm Technologies, Inc.
- 11. Scaling – Dedicated Investments in AI HW Engines 11 Investment in sparsity modules for supporting Auto ADAS usages Optimize design for sustained low power with the ability to enable high concurrency Investment in dedicated datatypes for higher performance and TCO (Total Cost of Ownership) Snapdragon is a product of Qualcomm Technologies, Inc. and/or its subsidiaries Using Mobile Design as an Anchor Point © 2022 Qualcomm Technologies, Inc.
- 12. Performance Leadership – Using MLPerf ™ 2.0 RESNET50 2221 Inf/sec MOSAIC 752 Inf/sec BERT 101 Inf/sec 16 75W cards in one server 20 server units in one server rack Gigabyte Platform 371,473 Inf/sec 7.4+M Inf/sec Cloud AI 100 servers achieve 3.7x higher rack-level ResNet-50 inference performance than Nvidia A100 servers 8C GEN 1 achieves an average of about 26% better latency than Exynos devices across various categories Scaling from Mobile to Cloud.. Qualcomm® 8C GEN 1 Mobile Qualcomm® Cloud AI 100 Xiaomi Mi12 Platform RESNET50 Qualcomm Cloud AI 100 is a product of Qualcomm Technologies, Inc. and/or its subsidiaries. © 2022 Qualcomm Technologies, Inc. 12
- 13. AI Software
- 14. Vision: Accelerate AI Innovation and Solution Deployment 14 Performance Scalability Innovation Tools Accelerate “out of box” operator functionality and performance Ability to have programming consistency from Cloud to Edge Accelerate AI Solution deployment with investment in Tools Innovation to drive product leadership (Pre-emption, DFS, Multi Chaining) © 2022 Qualcomm Technologies, Inc.
- 15. Qualcomm AI SW MLOps Cycle (Inception to Monitoring) 15 Data Preparation (Data labeling, datasets, feature store) Model Preparation (Training/HP-tuning, HW-aware NAS) Model Deployment (Quantization, compression, Inference) Model Monitoring (Detect drift, training- serving skew) Focus for SW Discussion Today © 2022 Qualcomm Technologies, Inc.
- 16. AI Software Workflow 16 Choose env., config., model and framework Does the model meet performance metrics? Model Compilation/ Runner Accuracy Evaluation Does the model compile? Is the model’s accuracy acceptable? Is the model’s output & latency acceptable? Integrate model into Application or pipeline Deploy Application Train & Finetune model Add custom layers and/or fix errors Debug and fix errors Qualcomm® Neural Processing SDK Converter and Quantizer Custom Ops LLVM (C/C++) TVM (Python) AI model Efficiency Toolkit (AIMET) Formats supported Profile Model Performance Did that work? Use advanced optimization techniques Hexagon Profiler & Trace Analyzer Data Scientist ML Training Engineer ML Inference Engineer App Developer DevOps Engineer Environment Training Compilation Accuracy analysis Optimizations Integration Deployment Qualcomm® Neural Processing SDK Profilers Legend Customer chosen SW Customer ML Workflow Target Usage SNPE & Qualcomm AI Engine Direct Workflow Tools for performance & accuracy optimization © 2022 Qualcomm Technologies, Inc.
- 17. NAS : Model Optimization Mapped to HW Intrinsics 17 Partnering with Google Vertex AI Team - Integrated into Qualcomm Software Stack Results Note : DL architecture choices influence results STEP : 1 Space of allowable architectures (Structure, operations, connectivity) Sampling populations of good architecture candidates Estimate performance of sampled architecture Search Space Search Algorithm Evaluation Strategy How • 8-10 % Accuracy Improvements • ~20 to 30% Latency Reduction © 2022 Qualcomm Technologies, Inc.
- 18. AIMET : Quantize & Compress AI Models 18 Trained AI model AI Model Efficiency Toolkit (AIMET) Optimized AI model TensorFlow or PyTorch Compression Quantization Deployed AI model State of the art network compression and quantization tools for various DL architectures (CNN, BERT, GAN’s..) Automated way of enabling reduction in precision of weights and activation while maintaining accuracy Original Image FP16 Segmented Map Quantized Segmented Map <0.5% Drop in Accuracy (Across various DL Models) -> Using QAT and PTQ techniques from AIMET Results Why STEP : 2
- 19. Your–NN-Framework Training .onnx .onnx .onnx .onnx .onnx .onnx .pb CPU Qualcomm Neural Network Library QML NEON KERNELS KERNELS eAI Open CL Open CL Performance Qualcomm® Neural Processing SDK ONNXRT PT Mobile TF- Lite NNAPI TF-Lite µ GPU Hexagon Processor Qualcomm Sensing Hub General purpose compute engines AI engines Profiler Debugger Visualizer Compilers Scalability Runtime framework Low level Library AI Engine On-Device Execution/Inference Run Time: Qualcomm AI Software Stack for Performance and Scalability Support - Application Deployment STEP : 3 19
- 20. • Meaningful improvements after initial chip shipment ; achieved through software investment / innovation • Consequences - • Additional gains across the entire SoC portfolio for downstream chips and adjacent BUs from a common investment • Platform software that can be delivered incrementally to already released products can continue to improve • Improvements directly leveraged into next generation chips • Improvements in current generation can lead to relative performance gap compression with next-gen devices 0 200 400 600 800 1000 1200 1400 1600 1800 2000 Tech Day MLPerf 1.0 MLPerf 1.1 Tech Day MLPerf 1.0 MLPerf 1.1 Tech Day MLPerf 1.0 MLPerf 1.1 MobilenetEdgeTPU DeeplabV3+ MobilenetEdgeTPU - Offline Premium Mobile MLPerf Improvements 0 200000 400000 600000 800000 1000000 1200000 Dec-20 May-21 Dec-20 May-21 Oct-20 Dec-20 AIMark AITutu AIB4 Premium Mobile – Other Benchmark Improvements 0 50 100 150 200 250 300 350 400 450 500 ML Perf 1.0 ML Perf 1.1 Resnet34-SSD Data Center- Improvement for Resnet34-SSD STEP : 4 Continuous Improvements in SW Stack - For Performance Leadership 20 © 2022 Qualcomm Technologies, Inc.
- 21. Conclusions
- 22. • AI Applications expanding beyond modalities of computer vision to linguistics, communication, commerce and language understanding • With evolution of AI Applications across many verticals, continued push for innovation around latency, heterogeneity, concurrency and user personalization • This is putting a lot of emphasis for the need of custom HW modules in several BU verticals in Qualcomm • Qualcomm silicon continues to show leadership in performance and energy efficiency in industry leading benchmarks • High investment in software continues to accelerate AI solution deployment across all verticals Conclusion 22 © 2022 Qualcomm Technologies, Inc.
- 23. Resources 23 2022 Embedded Vision Summit • “Powering the Intelligent Connected Edge and the Future of On-Device AI” Ziad Asghar May 18 9:30 - 10:00 AM PT • “A Practical Guide to Getting the DNN Accuracy You Need and the Performance You Deserve” Felix Baum May 18 2:40 - 3:10 PM PT • “Tools for Creating Next-Gen Computer Vision Apps on Snapdragon” Judd Heape May 18 10:50 - 11:20 AM PT • “Seamless Deployment of Multimedia and Machine Learning Applications at the Edge” Megha Daga May 17 2:40 - 3:10 PM PT Qualcomm Mobile AI Mobile AI | On-Device AI | Qualcomm® Qualcomm & Google NAS Qualcomm Technologies and Google Cloud Announce Collaboration on Neural Architecture Search for the Connected Intelligent Edge | Qualcomm Vinesh Sukumar Senior Director, Product Management – AI/ML vinesuku@qti.qualcomm.com © 2022 Qualcomm Technologies, Inc.
- 24. Thank You