Qualcomm Research, profile picture
Uploaded byQualcomm Research
PDF, PPTX1,011 views

Presentation - Model Efficiency for Edge AI

This document summarizes a presentation given by Chirag Patel and Tijmen Blankevoort of Qualcomm AI Research on model efficiency techniques for edge AI. They discuss why model efficiency is important for on-device AI due to constraints like power and thermal limits. They overview techniques like quantization, conditional compute, neural architecture search, and compilation that can shrink AI models and efficiently run them on hardware. Specifically, they find that integer quantization through techniques like post-training and quantization-aware training can achieve similar accuracy as floating point models but provide much better performance per watt. Overall, the presentation advocates that integer quantization is the best approach for efficient AI inference on edge devices.

Technology◦
Related topics:
Edge Computing•

Embed presentation

Download as PDF, PPTX
Chirag Patel Engineer, Principal/Manager Qualcomm AI Research September 21, 2022 @QCOMResearch The future of model efficiency for edge AI Tijmen Blankevoort Director of Engineering Qualcomm AI Research Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc
2 Our presenters Why model efficiency is important for on-device AI Overview of integer quantization (INT) versus floating point (FP) 3 1 2 4 Agenda Chirag Patel Engineer, Principal/Manager, Qualcomm AI Research Tijmen Blankevoort Director, Engineering, Qualcomm AI Research 5 Questions? Open-source tools: AI Model Efficiency Toolkit (AIMET) and AIMET Model Zoo AIMET and AIMET Model Zoo are products of Qualcomm Innovation Center, Inc. Improving low-bit quantization
3 3 Video monitoring Extended reality Smart cities Smart factories Autonomous vehicles Video conferencing Smart homes Smartphone 3 AI is being used all around us increasing productivity, enhancing collaboration, and transforming industries
4 4 Source: Welling Will we have reached the capacity of the human brain? Energy efficiency of the human brain is estimated to be 100,000x better than current hardware 2025 Weight parameter count 1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 1943: First NN (+/- N=10) 1988: NetTalk (+/- N=20K) 2009: Hinton’s Deep Belief Net (+/- N=10M) 2013: Google/Y! (N=+/- 1B) 2025: N = 100T = 1014 2017: Very large neural networks (N=137B) 1012 1010 108 106 1014 104 102 100 Deep neural networks are energy hungry and growing fast AI is being powered by the explosive growth of deep neural networks 2021: Extremely large neural networks (N=1.6T)
5 Power and thermal efficiency are essential for on-device AI The challenge of AI workloads Constrained mobile environment Very compute intensive Large, complicated neural network models Complex concurrencies Always-on Real-time Must be thermally efficient for sleek, ultra-light designs Storage/memory bandwidth limitations Requires long battery life for all-day use
6 Holistic model efficiency research Multiple axes to shrink AI models and efficiently run them on hardware Quantization Learning to reduce bit-precision while keeping desired accuracy Conditional compute Learning to execute only parts of a large inference model based on the input Neural architecture search Learning to design smaller neural networks that are on par or outperform hand-designed architectures on real hardware Compilation Learning to compile AI models for efficient hardware execution
7 7 AIMET and AIMET Model Zoo are products of Qualcomm Innovation Center, Inc. Leading AI research and fast commercialization Driving the industry towards integer inference and power-efficient AI AI Model Efficiency Toolkit (AIMET) AIMET Model Zoo Relaxed Quantization (ICLR 2019) Data-free Quantization (ICCV 2019) AdaRound (ICML 2020) Bayesian Bits (NeurIPS 2020) Quantization research Quantization open-sourcing Overcoming Oscillations (ICML 2022) Transformer Quantization (EMNLP 2021) Joint Pruning and Quantization (ECCV 2020) FP8 Quantization (NeurIPS 2022)
8 1: FP32 model compared to quantized model Leading research to efficiently quantize AI models Promising results show that low-precision integer inference can become widespread Virtually the same accuracy between a FP32 and quantized AI model through: • Automated, data free, post-training methods • Automated training-based mixed-precision method Significant performance per watt improvements through quantization Automated reduction in precision of weights and activations while maintaining accuracy Models trained at high precision 32-bit floating point 3452.3194 8-bit Integer 255 Increase in performance per watt from savings in memory and compute1 Inference at lower precision 16-bit Integer 3452 01010101 Increase in performance per watt from savings in memory and compute1 up to 4X 4-bit Integer 15 Increase in performance per watt from savings in memory and compute1 01010101 up to 16X up to 64X 01010101 0101 01010101 01010101 01010101 01010101
9 What does it mean to quantize a neural network? Weight and activation quantization can have different bit-precisions to maintain accuracy Biases Choose quantization bit-width Choose quantization bit-width • Simulated quantization ops are added in the neural network after each usage of weights and activations, and after every ‘operation’ • Quantization is generally simulated in floating point instead of running in integer math • Weights and activations can be quantized with the same or different precisions within a model layer • For example, W8A16 uses quantized 8-bit weights and 16-bit activations. INT8 means quantized 8-bit weight and 8-bit activations. Act quant Input Conv / FC + RELU Output Weights Wt quant
10 10 What algorithm to choose to improve accuracy? Post-training quantization (PTQ) Quantization-aware training (QAT) Take pre-trained FP32 model and convert it directly into fixed-point network Train/fine-tune the network with the simulated quantization operations in place No need for the original training pipeline Requires access to the training pipeline and labelled data Data-free or small (unlabeled) calibration set needed Simple usage (⇔ single API call) Longer training times Might not reach as high accuracy as QAT Hyper-parameter tuning Achieves higher accuracy, especially for lower bit-widths
11 Which is the better format for quantizing neural networks? Floating point vs integer
12 12 2 2 S 4 S 2 S 2 𝑠: scale; 𝑚: mantissa; 𝑒: exponent; S: sign INT8 and FP8 have the same number of values but different distributions Multiple FP8 formats exist, and they consume more power than INT8 FP: 𝑧 = 𝑠 ⋅ 𝑚 ⋅ 2! Formats 7 S 6 1 S 5 2 S 4 3 S 3 4 S 2 5 S INT8 INT8 FP8 5/2 FP8 4/3 FP8 3/4 FP8 2/5 Formats most commonly proposed in the industry INT: 𝑧 = 𝑠 ⋅ 𝑚 Mantissa Exponent
13 8 7 6 5 4 2 0 3 1 10 8 6 4 2 0 Most layers of models do not have large outliers FP8 may be useful for model layers with large outliers Normal Outlier-heavy distribution Uniform Weight distribution Signal-to-noise ratio (SNR) higher better INT8 5M2E 4M3E 3M4E 2M5E INT8 5M2E 4M3E 3M4E 2M5E SNR INT8 5M2E 4M3E 3M4E 2M5E 12 10 8 6 4 2 0 SNR SNR Normal Outlier-heavy distribution Uniform Some outliers
14 14 Several FP8 formats are required to get the best PTQ inference results For different networks, different formats are better — it depends on the amount of outliers Supporting multiple formats in hardware is expensive Model FP32 Best FP8 format Best result Worst FP8 format Worst FP8 format ResNet18 69.72% 69.66% 64.92% MobileNetV2 71.70% 71.06% 49.51% BERT 83.06 82.80 71.56 SalsaNext 55.80 55.67 55.12 HRNet 81.05 81.04 80.77 DeepLabV3 (MobileNetV2) 72.91 72.58 37.93 ViT 77.75% 77.71% 76.69 5 2 5 2 3 4 4 3 5 2 4 3 4 3 2 5 3 4 3 4 3 4 “FP8 Quantization: The Power of the Exponent”, NeurIPS 2022 2 5 5 2 2 5
15 FP32 INT8 Best FP8* 69.72% 69.55% 69.66% 70.43% 69.82% ResNet FP32 INT8 Best FP8* 71.70% 70.94% 71.06% 71.82% 71.54% MobileNet V2 FP32 INT8 Best FP8* 83.06 71.03 82.80 83.26 83.70 Bert FP32 INT8 Best FP8* 72.91 71.24 72.58 73.99 72.41 DeepLabV3 *: Best FP8 is the best result from testing the different FP8 formats. “FP8 Quantization: The Power of the Exponent”, NeurIPS 2022 INT8 has similar results as FP8 with QAT Outliers can be suitably trained with QAT PTQ QAT • No PTQ tricks – Per-channel • All QAT results per-tensor quantization • FP8 mantissa and exponent format was optimized for this comparison
16 FP32 INT (W8A8) INT (W8A16) Best FP8 result ResNet18 69.72% 69.55% 69.75% 69.66% HRNet 81.05 80.93 81.08 81.04 INT W8A16 accuracy is better than FP8 for all models with PTQ No real gap FP8/INT8 FP32 INT (W8A8) INT (W8A16) Best FP8 result BERT 83.06 71.03 82.90 82.80 SalsaNext 55.80 54.22 55.82 55.67 ViT 77.75% 76.39% 77.73% 77.71% MobileNetV2 69.72% 69.55% 69.75% 69.66% • Min-max range setting • Per-channel quantization
17 INT16 performs better than FP16 unless there are large outliers • 1,000 samples of 𝑋 ~ 𝑁𝑜𝑟𝑚𝑎𝑙(0, 1) • We add one outlier and vary its value • INT13 performs comparable to FP16 in terms of MSE 10−4 10−5 10−7 10−6 0 250 500 750 1000 Outlier 1250 1500 1750 2000 MSE MSE Error for w/o activation function for sigma =1.0, 100 neurons int16 float16
18 INT16 outperforms FP16 in accuracy and runs faster in hardware MobileNetV2 71.74% FP32 71.69% FP16 71.74% INT16 EfficientDet-D1 40.08 FP32 40.07 FP16 40.07 INT16
19 19 Integer quantization is the way to do AI inference Enabled through PTQ and QAT techniques Mixed precision gives the best of both worlds, using extra precision only when necessary INT 4 INT 8 INT 16 Best Better Good Power efficiency and latency Best Better Good Accuracy
Overcoming oscillations in quantization-aware training Improving quantization-aware training at lower bit-widths
21 21 Poor validation accuracy is consistent across various learning rates and epochs during QAT “Overcoming Oscillations in Quantization-Aware Training” (ICML 2022) Validation accuracy for QAT is typically unstable Why do we see the validation accuracy drop for 4-bit QAT? Training accuracy Epoch Different learning rates (LR) 71% 67% 62% 2 4 6 8 10 12 14 16 18 20 Validation accuracy Epoch 44% 70% 60% 50% 42% 2 4 6 8 10 12 14 16 18 20
22 22 2 0 −2 −4 Oscillations are present in QAT Example of MobileNetV2 training (last 1000 iterations of training) Quantized weights, 𝑞 𝑤 (Lowest bit of 4−bit weight) Latent weights, 𝑤 (Positive FP values zoomed on 0.5) Sign and Bit0 weights Iterations during QAT FP weights 0.5004 0.5002 0.5000 0.4998 0.4996 0 200 400 600 1000
23 Network Bits Pre-BN Acc. Post-BN Acc. MobileNetV2 8 71.79 0.07 71.89 0.05 MobileNetV2 4 68.99 0.44 71.01 0.05 MobileNetV2 3 64.97 1.23 69.50 0.04 Method Train Loss Val. Acc. (%) Baseline 1.3566 69.50 SR (mean + std) 1.3547 0.0053 69.58 0.09 SR (best) 1.3391 69.85 AdaRound 1.3070 70.12 +2.02 +4.53 Corrupts batch norm statistics • At inference, BN uses running statistics from training • Oscillations lead to big changes in statistics -> running statistics are not a good estimate • Solution: BN re-estimation Disrupts model convergence • At the end of training, oscillating weights may not be on the correct ‘side’ • Stochastic rounding (SR) and binary optimization (AdaRound) show that they are indeed not in the best possible state. • Oscillations prevent network from converging to best local minimum Oscillating weights are harmful when training a model 23
24 EMA = Exponential Moving Average Higher oscillation frequencies during QAT negatively impact accuracy Oscillation occurs The integer value changes & Its direction is opposite to its previous one
25 25 “Overcoming Oscillations in Quantization-Aware Training” (ICML 2022) Oscillation dampening and iterative freezing fix the QAT issue Dampening takes a regularizing approach: the weights are forced closer to the bin center Freezing the oscillating weights stabilizes training and mitigates the unwanted effects of oscillations 100 80 60 40 20 0 −0.4 −0.2 0.0 0.2 0.4 wint − w/s wint − w/s −0.4 −0.2 0.0 0.2 0.4 400 300 200 100 0 Dampening Freezing Frozen Not frozen
26 1: “Overcoming Oscillations in Quantization-Aware Training” (ICML 2022) 2: “Learned step size quantization” (ICLR,2020) We achieve SOTA results for INT4 quantization1 • Train with learned step-size quantization (LSQ2) and re-estimation • Dampening and freezing perform on par with each other Method W/A Val. Acc. (%) Full-precision 32/32 65.1 LSQ* (Esser et al., 2020) 4/4 61.0 (-4.1) LSQ + BR (Han et al., 2021) 4/4 61.5 (-3.6) LSQ + Dampen (ours) 4/4 63.7 (-1.4) LSQ + Freeze (ours) 4/4 63.6 (-1.5) Method W/A Val. Acc. (%) Full-precision 32/32 71.7 LSQ* (Esser et al., 2020) 4/4 69.5 (−2.3) LSQ + BR (Han et al., 2021) 4/4 70.4 (−1.4) LSQ + Dampen (ours) 4/4 70.5 (−1.2) LSQ + Freeze (ours) 4/4 70.6 (−1.1) MobileNetV3 MobileNetV2
27 Open-source projects to scale energy-efficient AI to the masses AIMET & AIMET Model Zoo
28 28 AIMET makes AI models small Open-sourced GitHub project that includes state-of-the-art quantization and compression techniques from Qualcomm AI Research Features: State-of-the-art network compression tools State-of-the-art quantization tools Support for both TensorFlow and PyTorch Benchmarks and tests for many models Developed by professional software developers If interested, please join the AIMET GitHub project: https://github.com/quic/aimet Trained AI model AI Model Efficiency Toolkit (AIMET) Optimized AI model TensorFlow or PyTorch Compression Quantization Deployed AI model
AIMET Providing advanced model efficiency features and benefits Benefits Lower memory bandwidth Lower power Lower storage Higher performance Maintains model accuracy Simple ease of use Features Quantization Compression State-of-the-art INT8 and INT4 performance Quantization-aware training (QAT) Efficient tensor decomposition and removal of redundant channels in convolution layers Spatial singular value decomposition (SVD) Channel pruning Visualization Analysis tools for drawing insights for quantization and compression Weight ranges Per-layer compression sensitivity Quantization simulation Post-training quantization (PTQ) methods: • Data-Free Quantization • Adaptive Rounding (AdaRound), • Automatic Mixed Precision (AMP) • AutoQuant 29
30 30 AIMET features and APIs are easy to use Designed to fit naturally in the AI model development workflow for researchers, developers, and ISVs PyTorch model PyTorch Model Train No change Same API PyTorch Model Train Create QuantSim Evaluate Typical model training workflow User-friendly QAT workflow in AIMET No change Same API Evaluate User-friendly APIs invoked directly from the existing model pipeline Example Jupyter notebooks on AIMET GitHub AIMET extensions extensions Model optimization library (techniques to compress & quantize models) Framework specific API Algorithm API Other frameworks
31 Low resolution Super resolution First 4K super-resolution demo at 100+ FPS on mobile Our new machine-learning based super-resolution method 8-bit quantized model created using AIMET QAT
32 With better PTQ and QAT techniques, more models will achieve better power efficiency AIMET enables accurate INT W4A8 for wide range of use cases Task Model FP32 INT W4A8 Classification ResNet50 76.10% 75.4% ResNet18 69.75% 68.96% EfficientNet-Lite 75.31% 74.33% Regnext 78.3% 77.2% Segmentation Deeplabv3 (RN-50) 76.07% 75.91% Super-resolution ABPN 31.97 dB 31.67 (dB) Pose detection PoseNet (HRNet-32) 0.765 0.763
33 33 Comparison between FP32 model and model quantized with AIMET AIMET quantizes transformers with high accuracy, comparable to FP32 Top-1 accuracy 81.30 FP32 80.88 INT8/ W8A16 (PTQ) ViT base GLUE 84.99 FP32 84.60 INT8 (QAT) RoBERTa base GLUE 82.73 FP32 81.95 INT8 (QAT) BERT base (uncased) GLUE 79.21 FP32 78.61 INT8 (QAT) DistilBERT base (uncased)
34 34 AIMET Model Zoo Accurate pre-trained 8-bit quantized models Image classification Semantic segmentation Pose estimation Speech recognition Object detection Super resolution
35 35 35 *: Comparison between FP32 model and INT8 model quantized with AIMET. For further details, check out: https://github.com/quic/aimet-model-zoo/ AIMET Model Zoo includes popular quantized AI models Accuracy is maintained for INT8 models — less than 1% loss* Top-1 accuracy* 75.21% FP32 74.96% INT8 ResNet-50 (v1) Top-1 accuracy* 75% FP32 74.21% INT8 MobileNet-v2-1.4 Top-1 accuracy* 74.93% FP32 74.99% INT8 EfficientNet Lite mAP* 0.2469 FP32 0.2456 INT8 ResNet-50 (v1) mAP* 0.35 FP32 0.349 INT8 RetinaNet mAP* 0.383 FP32 0.379 INT8 Pose estimation PSNR* 25.45 FP32 24.78 INT8 SRGAN Top-1 accuracy* 71.67% FP32 71.14% INT8 MobileNetV2 Top-1 accuracy* 75.42% FP32 74.44% INT8 EfficientNet-lite0 mIoU* 72.62% FP32 72.22% INT8 DeepLabV3+ mAP* 68.7% FP32 68.6% INT8 MobileNetV2-SSD-Lite mAP* 0.364 FP32 0.359 INT8 Pose estimation PSNR 25.51 FP32 25.5 INT8 SRGAN WER* 9.92% FP32 10.22% INT8 DeepSpeech2 PSNR 32.75 FP32 32.69 INT8 ABPN <1% Loss in accuracy*
36 *: Comparison between FP32 model and INT8 model quantized with AIMET. For further details, check out: https://github.com/quic/aimet-model-zoo/ Super resolution model suite Wide variety of models, suited for fast, energy-efficient INT8 inference • Virtually no accuracy loss compared to FP32 • Simple and convenient for developer integration • Useful across diverse applications, from gaming and photography to XR and autonomous driving 1 Anchor-based Plain Net (ABPN) 2 Robust Real-Time Single-Image Super Resolution (XLSR) 3 Super-Efficient Super Resolution (SESR) PSNR (dB) 32.66 FP32 32.58 INT8 SESR-M73 PSNR (dB) 32.41 FP32 32.25 INT8 SESR-M33 PSNR (dB) 32.71 FP32 32.64 INT8 ABPN1 PSNR (dB) 32.57 FP32 32.30 INT8 XLSR2 PSNR (dB) 33.03 FP32 32.92 INT8 SESR-XL3 INT8 PSNR and visual quality comparable to FP32*
37 37 Explore our open-source projects and tools AIMET State-of-the-art quantization and compression techniques github.com/quic/aimet AIMET Model Zoo Accurate pre-trained 8-bit quantized models github.com/quic/aimet-model-zoo Quantization whitepaper arxiv.org/abs/2201.08442
AI Frameworks Qualcomm® Neural Processing SDK TF Lite TF Lite Micro Direct ML AI Runtimes Programming Languages Virtual platforms Core Libraries Math Libraries Profilers & Debuggers Compilers System Interface SoC, accelerator drivers Auto XR Robotics IoT ACPC Smartphones Cloud Platforms Qualcomm® AI Engine Direct (QNN) Tools: Emulation Support Qualcomm Neural Processing SDK, Qualcomm AI Model Studio, and Qualcomm AI Engine Direct are products of Qualcomm Technologies, Inc. and/or its subsidiaries Qualcomm AI Model Studio AIMET AIMET Model Zoo NAS Model analyzers Infrastructure:
39 Model efficiency is key for enabling on-device AI and accelerating the growth of the connected intelligent edge INT8/16 perform better than FP8/16 Qualcomm AI Research is enabling 4-bit integer models AIMET is making fixed-point quantization possible at scale without sacrificing accuracy
40 www.qualcomm.com/research/ artificial-intelligence @QCOMResearch www.qualcomm.com/news/onq www.youtube.com/c/QualcommResearch www.slideshare.net/qualcommwirelessevolution Connect with us Questions
Nothing in these materials is an offer to sell any of the components or devices referenced herein. ©2018-2022 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Hexagon, and Snapdragon are trademarks or registered trademarks of Qualcomm Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes our licensing business, QTL, and the vast majority of our patent portfolio. Qualcomm Technologies, Inc., a subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of our engineering, research and development functions, and substantially all of our products and services businesses, including our QCT semiconductor business. Follow us on: For more information, visit us at: qualcomm.com & qualcomm.com/blog Thank you

More Related Content

PDF
Enabling Power-Efficient AI Through Quantization
byQualcomm Research
 
PDF
Enabling on-device learning at scale
byQualcomm Research
 
PDF
Understanding the world in 3D with AI.pdf
byQualcomm Research
 
PDF
AI firsts: Leading from research to proof-of-concept
byQualcomm Research
 
PDF
Introduction to Deep Learning (NVIDIA)
byRakuten Group, Inc.
 
PDF
Device-level AI for 5G and beyond
by3G4G
 
PDF
The future of AI is hybrid
byQualcomm Research
 
PDF
How 5G Will Transform Industrial IoT
byQualcomm Research
 
Enabling Power-Efficient AI Through Quantization
byQualcomm Research
 
Enabling on-device learning at scale
byQualcomm Research
 
Understanding the world in 3D with AI.pdf
byQualcomm Research
 
AI firsts: Leading from research to proof-of-concept
byQualcomm Research
 
Introduction to Deep Learning (NVIDIA)
byRakuten Group, Inc.
 
Device-level AI for 5G and beyond
by3G4G
 
The future of AI is hybrid
byQualcomm Research
 
How 5G Will Transform Industrial IoT
byQualcomm Research
 

What's hot

PPTX
5G NR-based C-V2X
byQualcomm Research
 
PPTX
Transformers in Vision: From Zero to Hero
byBill Liu
 
PDF
FPGAs and Machine Learning
byinside-BigData.com
 
PDF
Transfer Learning
byHichem Felouat
 
PDF
Intelligence at scale through AI model efficiency
byQualcomm Research
 
PDF
OpenGL 4.4 - Scene Rendering Techniques
byNarann29
 
PPTX
GS-4106 The AMD GCN Architecture - A Crash Course, by Layla Mah
byAMD Developer Central
 
PDF
FPGA Hardware Accelerator for Machine Learning
byDr. Swaminathan Kathirvel
 
PDF
Hardware Acceleration for Machine Learning
byCastLabKAIST
 
PPTX
Machine Learning - Convolutional Neural Network
byRichard Kuo
 
PDF
The Rendering Technology of 'Lords of the Fallen' (Game Connection Europe 2014)
byPhilip Hammer
 
PDF
Trip down the GPU lane with Machine Learning
byRenaldas Zioma
 
PDF
Bluetooth Broadcasting
byijtsrd
 
PDF
Advanced Scenegraph Rendering Pipeline
byNarann29
 
PDF
“How Transformers are Changing the Direction of Deep Learning Architectures,”...
byEdge AI and Vision Alliance
 
PDF
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
byUniversitat Politècnica de Catalunya
 
PDF
Why and what you need to know about 6G in 2022
byQualcomm Research
 
PPT
Secrets of CryENGINE 3 Graphics Technology
byTiago Sousa
 
PDF
Rendering Tech of Space Marine
byPope Kim
 
PDF
COSCUP 2023 - Make Your Own Ray Tracing GPU with FPGA
byOwen Wu
 
5G NR-based C-V2X
byQualcomm Research
 
Transformers in Vision: From Zero to Hero
byBill Liu
 
FPGAs and Machine Learning
byinside-BigData.com
 
Transfer Learning
byHichem Felouat
 
Intelligence at scale through AI model efficiency
byQualcomm Research
 
OpenGL 4.4 - Scene Rendering Techniques
byNarann29
 
GS-4106 The AMD GCN Architecture - A Crash Course, by Layla Mah
byAMD Developer Central
 
FPGA Hardware Accelerator for Machine Learning
byDr. Swaminathan Kathirvel
 
Hardware Acceleration for Machine Learning
byCastLabKAIST
 
Machine Learning - Convolutional Neural Network
byRichard Kuo
 
The Rendering Technology of 'Lords of the Fallen' (Game Connection Europe 2014)
byPhilip Hammer
 
Trip down the GPU lane with Machine Learning
byRenaldas Zioma
 
Bluetooth Broadcasting
byijtsrd
 
Advanced Scenegraph Rendering Pipeline
byNarann29
 
“How Transformers are Changing the Direction of Deep Learning Architectures,”...
byEdge AI and Vision Alliance
 
The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...
byUniversitat Politècnica de Catalunya
 
Why and what you need to know about 6G in 2022
byQualcomm Research
 
Secrets of CryENGINE 3 Graphics Technology
byTiago Sousa
 
Rendering Tech of Space Marine
byPope Kim
 
COSCUP 2023 - Make Your Own Ray Tracing GPU with FPGA
byOwen Wu
 

Similar to Presentation - Model Efficiency for Edge AI

PDF
"Quantizing Deep Networks for Efficient Inference at the Edge," a Presentatio...
byEdge AI and Vision Alliance
 
PDF
“DNN Quantization: Theory to Practice,” a Presentation from AMD
byEdge AI and Vision Alliance
 
PDF
A White Paper On Neural Network Quantization
byApril Knyff
 
PDF
Pushing the boundaries of AI research
byQualcomm Research
 
PDF
Integer quantization for deep learning inference: principles and empirical ev...
byjemin lee
 
PDF
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic...
byRyo Takahashi
 
PDF
“Practical DNN Quantization Techniques and Tools,” a Presentation from Facebook
byEdge AI and Vision Alliance
 
PDF
“Practical Approaches to DNN Quantization,” a Presentation from Magic Leap
byEdge AI and Vision Alliance
 
PDF
Julien Simon - Deep Dive - Quantizing LLMs
byJulien SIMON
 
PDF
Extremely Low Bit Transformer Quantization for On-Device NMT
byInsoo Chung
 
PDF
Model Quantization Technologies with AIMET.pdf
byZHUORANGUO2
 
PDF
"Efficient Deployment of Quantized ML Models at the Edge Using Snapdragon SoC...
byEdge AI and Vision Alliance
 
PDF
Tutorial-on-DNN-07-Co-design-Precision.pdf
byDuy-Hieu Bui
 
PDF
Efficient execution of quantized deep learning models a compiler approach
byjemin lee
 
PDF
“Introduction to Data Types for AI: Trade-offs and Trends,” a Presentation fr...
byEdge AI and Vision Alliance
 
PDF
Fast, Scalable Quantized Neural Network Inference on FPGAs with FINN and Logi...
byKTN
 
PPTX
quantized spike driven transformer presentation
byericzhang0328
 
PDF
“Introduction to Shrinking Models with Quantization-aware Training and Post-t...
byEdge AI and Vision Alliance
 
PDF
“Powering the Connected Intelligent Edge and the Future of On-Device AI,” a P...
byEdge AI and Vision Alliance
 
PDF
Accelerating algorithmic and hardware advancements for power efficient on-dev...
byQualcomm Research
 
"Quantizing Deep Networks for Efficient Inference at the Edge," a Presentatio...
byEdge AI and Vision Alliance
 
“DNN Quantization: Theory to Practice,” a Presentation from AMD
byEdge AI and Vision Alliance
 
A White Paper On Neural Network Quantization
byApril Knyff
 
Pushing the boundaries of AI research
byQualcomm Research
 
Integer quantization for deep learning inference: principles and empirical ev...
byjemin lee
 
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic...
byRyo Takahashi
 
“Practical DNN Quantization Techniques and Tools,” a Presentation from Facebook
byEdge AI and Vision Alliance
 
“Practical Approaches to DNN Quantization,” a Presentation from Magic Leap
byEdge AI and Vision Alliance
 
Julien Simon - Deep Dive - Quantizing LLMs
byJulien SIMON
 
Extremely Low Bit Transformer Quantization for On-Device NMT
byInsoo Chung
 
Model Quantization Technologies with AIMET.pdf
byZHUORANGUO2
 
"Efficient Deployment of Quantized ML Models at the Edge Using Snapdragon SoC...
byEdge AI and Vision Alliance
 
Tutorial-on-DNN-07-Co-design-Precision.pdf
byDuy-Hieu Bui
 
Efficient execution of quantized deep learning models a compiler approach
byjemin lee
 
“Introduction to Data Types for AI: Trade-offs and Trends,” a Presentation fr...
byEdge AI and Vision Alliance
 
Fast, Scalable Quantized Neural Network Inference on FPGAs with FINN and Logi...
byKTN
 
quantized spike driven transformer presentation
byericzhang0328
 
“Introduction to Shrinking Models with Quantization-aware Training and Post-t...
byEdge AI and Vision Alliance
 
“Powering the Connected Intelligent Edge and the Future of On-Device AI,” a P...
byEdge AI and Vision Alliance
 
Accelerating algorithmic and hardware advancements for power efficient on-dev...
byQualcomm Research
 

More from Qualcomm Research

PDF
Generative AI at the edge.pdf
byQualcomm Research
 
PDF
Qualcomm Webinar: Solving Unsolvable Combinatorial Problems with AI
byQualcomm Research
 
PDF
Enabling the metaverse with 5G- web.pdf
byQualcomm Research
 
PDF
Bringing AI research to wireless communication and sensing
byQualcomm Research
 
PDF
How will sidelink bring a new level of 5G versatility.pdf
byQualcomm Research
 
PDF
Scaling 5G to new frontiers with NR-Light (RedCap)
byQualcomm Research
 
PDF
Realizing mission-critical industrial automation with 5G
byQualcomm Research
 
PDF
3GPP Release 17: Completing the first phase of 5G evolution
byQualcomm Research
 
PDF
Setting off the 5G Advanced evolution with 3GPP Release 18
byQualcomm Research
 
PDF
5G positioning for the connected intelligent edge
byQualcomm Research
 
PDF
The essential role of AI in the 5G future
byQualcomm Research
 
PDF
How AI research is enabling next-gen codecs
byQualcomm Research
 
PDF
Role of localization and environment perception in autonomous driving
byQualcomm Research
 
PDF
Pioneering 5G broadcast
byQualcomm Research
 
PDF
How to build high performance 5G networks with vRAN and O-RAN
byQualcomm Research
 
PDF
What's in the future of 5G millimeter wave?
byQualcomm Research
 
PDF
Efficient video perception through AI
byQualcomm Research
 
PDF
Enabling the rise of the smartphone: Chronicling the developmental history at...
byQualcomm Research
 
PDF
5G spectrum innovations and global update
byQualcomm Research
 
PDF
Transforming enterprise and industry with 5G private networks
byQualcomm Research
 
Generative AI at the edge.pdf
byQualcomm Research
 
Qualcomm Webinar: Solving Unsolvable Combinatorial Problems with AI
byQualcomm Research
 
Enabling the metaverse with 5G- web.pdf
byQualcomm Research
 
Bringing AI research to wireless communication and sensing
byQualcomm Research
 
How will sidelink bring a new level of 5G versatility.pdf
byQualcomm Research
 
Scaling 5G to new frontiers with NR-Light (RedCap)
byQualcomm Research
 
Realizing mission-critical industrial automation with 5G
byQualcomm Research
 
3GPP Release 17: Completing the first phase of 5G evolution
byQualcomm Research
 
Setting off the 5G Advanced evolution with 3GPP Release 18
byQualcomm Research
 
5G positioning for the connected intelligent edge
byQualcomm Research
 
The essential role of AI in the 5G future
byQualcomm Research
 
How AI research is enabling next-gen codecs
byQualcomm Research
 
Role of localization and environment perception in autonomous driving
byQualcomm Research
 
Pioneering 5G broadcast
byQualcomm Research
 
How to build high performance 5G networks with vRAN and O-RAN
byQualcomm Research
 
What's in the future of 5G millimeter wave?
byQualcomm Research
 
Efficient video perception through AI
byQualcomm Research
 
Enabling the rise of the smartphone: Chronicling the developmental history at...
byQualcomm Research
 
5G spectrum innovations and global update
byQualcomm Research
 
Transforming enterprise and industry with 5G private networks
byQualcomm Research
 

Recently uploaded

PDF
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 
PPTX
Strategic Shelf Planning for the New Year Turning Resolutions into Revenue .pptx
byAnoop Ashok
 
PPTX
AI in Marine Insurance Future of Smarter Risk, Faster Claims & Safer Shipping...
byAutomationEdge Technologies
 
PDF
LUXHUB: a detailed look at 2025...and fast forward to 2026
byalexandrekeilmann1
 
PPTX
Storage-and-HCI-Positioning-update-sales.pptx
byhungungphu123
 
PPTX
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
PPT
Database Management Systems: Basics, History, Data Models, etc.
byParithi Thamizh
 
PDF
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
PDF
apidays Australia 2025 | User Brainpower vs. AI Blind Spots in API Docs
byapidays
 
PPTX
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
PDF
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
PDF
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
PDF
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
PDF
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
PDF
How to Build a Crypto Wallet that Excels in 2026.pdf
byimoliviabennett
 
PDF
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
PPTX
CRM for the Insurance Industry Lessons for Insurance CIOs and Digital Leaders...
byKerry Millar
 
PPTX
Achieve enterprise automation with SAP BTP solutions and SAP Signavio
bydarrellkiwi
 
PPTX
Single Cell Protein from Methane or Methanol - Process development research c...
byJohn Downs
 
PPTX
Dell poweredge-customer-presentation.pptx
byhungungphu123
 
Kubernetes Cloud Native Indonesia Meetup - January 2026
byPrasta Maha
 
Strategic Shelf Planning for the New Year Turning Resolutions into Revenue .pptx
byAnoop Ashok
 
AI in Marine Insurance Future of Smarter Risk, Faster Claims & Safer Shipping...
byAutomationEdge Technologies
 
LUXHUB: a detailed look at 2025...and fast forward to 2026
byalexandrekeilmann1
 
Storage-and-HCI-Positioning-update-sales.pptx
byhungungphu123
 
Mastering SQL Server Replication: Types, Setup & Troubleshooting
byGeoPITS Global Pvt Ltd
 
Database Management Systems: Basics, History, Data Models, etc.
byParithi Thamizh
 
Deploying Windows Clients & Managing Identities
byVICTOR MAESTRE RAMIREZ
 
apidays Australia 2025 | User Brainpower vs. AI Blind Spots in API Docs
byapidays
 
Spacecraft Guidance Quick Research Guide by Arthur Morgan
byArthur Morgan
 
Safer’s Picks: Recent FME Enhancements with Big Everyday Impact
bySafe Software
 
5G Explained! A High Level Overview [Introduction]
byLuciano Motta
 
Azure DevOps Managed Services: Driving Speed, Security, and Business Growth
byjohncarterjn
 
Transcript: What ONIX can do: Leveraging metadata to support the discoverabil...
byBookNet Canada
 
How to Build a Crypto Wallet that Excels in 2026.pdf
byimoliviabennett
 
How PayPal Account Verification Works – Complete Guide for Online Businesses
byjhdhj3989
 
CRM for the Insurance Industry Lessons for Insurance CIOs and Digital Leaders...
byKerry Millar
 
Achieve enterprise automation with SAP BTP solutions and SAP Signavio
bydarrellkiwi
 
Single Cell Protein from Methane or Methanol - Process development research c...
byJohn Downs
 
Dell poweredge-customer-presentation.pptx
byhungungphu123
 

Presentation - Model Efficiency for Edge AI

  • 1.
    Chirag Patel Engineer, Principal/Manager QualcommAI Research September 21, 2022 @QCOMResearch The future of model efficiency for edge AI Tijmen Blankevoort Director of Engineering Qualcomm AI Research Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc
  • 2.
    2 Our presenters Why modelefficiency is important for on-device AI Overview of integer quantization (INT) versus floating point (FP) 3 1 2 4 Agenda Chirag Patel Engineer, Principal/Manager, Qualcomm AI Research Tijmen Blankevoort Director, Engineering, Qualcomm AI Research 5 Questions? Open-source tools: AI Model Efficiency Toolkit (AIMET) and AIMET Model Zoo AIMET and AIMET Model Zoo are products of Qualcomm Innovation Center, Inc. Improving low-bit quantization
  • 3.
    3 3 Video monitoring Extended realitySmart cities Smart factories Autonomous vehicles Video conferencing Smart homes Smartphone 3 AI is being used all around us increasing productivity, enhancing collaboration, and transforming industries
  • 4.
    4 4 Source: Welling Will wehave reached the capacity of the human brain? Energy efficiency of the human brain is estimated to be 100,000x better than current hardware 2025 Weight parameter count 1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 1943: First NN (+/- N=10) 1988: NetTalk (+/- N=20K) 2009: Hinton’s Deep Belief Net (+/- N=10M) 2013: Google/Y! (N=+/- 1B) 2025: N = 100T = 1014 2017: Very large neural networks (N=137B) 1012 1010 108 106 1014 104 102 100 Deep neural networks are energy hungry and growing fast AI is being powered by the explosive growth of deep neural networks 2021: Extremely large neural networks (N=1.6T)
  • 5.
    5 Power and thermal efficiencyare essential for on-device AI The challenge of AI workloads Constrained mobile environment Very compute intensive Large, complicated neural network models Complex concurrencies Always-on Real-time Must be thermally efficient for sleek, ultra-light designs Storage/memory bandwidth limitations Requires long battery life for all-day use
  • 6.
    6 Holistic model efficiency research Multiple axesto shrink AI models and efficiently run them on hardware Quantization Learning to reduce bit-precision while keeping desired accuracy Conditional compute Learning to execute only parts of a large inference model based on the input Neural architecture search Learning to design smaller neural networks that are on par or outperform hand-designed architectures on real hardware Compilation Learning to compile AI models for efficient hardware execution
  • 7.
    7 7 AIMET and AIMETModel Zoo are products of Qualcomm Innovation Center, Inc. Leading AI research and fast commercialization Driving the industry towards integer inference and power-efficient AI AI Model Efficiency Toolkit (AIMET) AIMET Model Zoo Relaxed Quantization (ICLR 2019) Data-free Quantization (ICCV 2019) AdaRound (ICML 2020) Bayesian Bits (NeurIPS 2020) Quantization research Quantization open-sourcing Overcoming Oscillations (ICML 2022) Transformer Quantization (EMNLP 2021) Joint Pruning and Quantization (ECCV 2020) FP8 Quantization (NeurIPS 2022)
  • 8.
    8 1: FP32 modelcompared to quantized model Leading research to efficiently quantize AI models Promising results show that low-precision integer inference can become widespread Virtually the same accuracy between a FP32 and quantized AI model through: • Automated, data free, post-training methods • Automated training-based mixed-precision method Significant performance per watt improvements through quantization Automated reduction in precision of weights and activations while maintaining accuracy Models trained at high precision 32-bit floating point 3452.3194 8-bit Integer 255 Increase in performance per watt from savings in memory and compute1 Inference at lower precision 16-bit Integer 3452 01010101 Increase in performance per watt from savings in memory and compute1 up to 4X 4-bit Integer 15 Increase in performance per watt from savings in memory and compute1 01010101 up to 16X up to 64X 01010101 0101 01010101 01010101 01010101 01010101
  • 9.
    9 What does itmean to quantize a neural network? Weight and activation quantization can have different bit-precisions to maintain accuracy Biases Choose quantization bit-width Choose quantization bit-width • Simulated quantization ops are added in the neural network after each usage of weights and activations, and after every ‘operation’ • Quantization is generally simulated in floating point instead of running in integer math • Weights and activations can be quantized with the same or different precisions within a model layer • For example, W8A16 uses quantized 8-bit weights and 16-bit activations. INT8 means quantized 8-bit weight and 8-bit activations. Act quant Input Conv / FC + RELU Output Weights Wt quant
  • 10.
    10 10 What algorithm tochoose to improve accuracy? Post-training quantization (PTQ) Quantization-aware training (QAT) Take pre-trained FP32 model and convert it directly into fixed-point network Train/fine-tune the network with the simulated quantization operations in place No need for the original training pipeline Requires access to the training pipeline and labelled data Data-free or small (unlabeled) calibration set needed Simple usage (⇔ single API call) Longer training times Might not reach as high accuracy as QAT Hyper-parameter tuning Achieves higher accuracy, especially for lower bit-widths
  • 11.
    11 Which is thebetter format for quantizing neural networks? Floating point vs integer
  • 12.
    12 12 2 2 S 4 S 2 S 2 𝑠:scale; 𝑚: mantissa; 𝑒: exponent; S: sign INT8 and FP8 have the same number of values but different distributions Multiple FP8 formats exist, and they consume more power than INT8 FP: 𝑧 = 𝑠 ⋅ 𝑚 ⋅ 2! Formats 7 S 6 1 S 5 2 S 4 3 S 3 4 S 2 5 S INT8 INT8 FP8 5/2 FP8 4/3 FP8 3/4 FP8 2/5 Formats most commonly proposed in the industry INT: 𝑧 = 𝑠 ⋅ 𝑚 Mantissa Exponent
  • 13.
    13 8 7 6 5 4 2 0 3 1 10 8 6 4 2 0 Most layers ofmodels do not have large outliers FP8 may be useful for model layers with large outliers Normal Outlier-heavy distribution Uniform Weight distribution Signal-to-noise ratio (SNR) higher better INT8 5M2E 4M3E 3M4E 2M5E INT8 5M2E 4M3E 3M4E 2M5E SNR INT8 5M2E 4M3E 3M4E 2M5E 12 10 8 6 4 2 0 SNR SNR Normal Outlier-heavy distribution Uniform Some outliers
  • 14.
    14 14 Several FP8 formatsare required to get the best PTQ inference results For different networks, different formats are better — it depends on the amount of outliers Supporting multiple formats in hardware is expensive Model FP32 Best FP8 format Best result Worst FP8 format Worst FP8 format ResNet18 69.72% 69.66% 64.92% MobileNetV2 71.70% 71.06% 49.51% BERT 83.06 82.80 71.56 SalsaNext 55.80 55.67 55.12 HRNet 81.05 81.04 80.77 DeepLabV3 (MobileNetV2) 72.91 72.58 37.93 ViT 77.75% 77.71% 76.69 5 2 5 2 3 4 4 3 5 2 4 3 4 3 2 5 3 4 3 4 3 4 “FP8 Quantization: The Power of the Exponent”, NeurIPS 2022 2 5 5 2 2 5
  • 15.
    15 FP32 INT8 BestFP8* 69.72% 69.55% 69.66% 70.43% 69.82% ResNet FP32 INT8 Best FP8* 71.70% 70.94% 71.06% 71.82% 71.54% MobileNet V2 FP32 INT8 Best FP8* 83.06 71.03 82.80 83.26 83.70 Bert FP32 INT8 Best FP8* 72.91 71.24 72.58 73.99 72.41 DeepLabV3 *: Best FP8 is the best result from testing the different FP8 formats. “FP8 Quantization: The Power of the Exponent”, NeurIPS 2022 INT8 has similar results as FP8 with QAT Outliers can be suitably trained with QAT PTQ QAT • No PTQ tricks – Per-channel • All QAT results per-tensor quantization • FP8 mantissa and exponent format was optimized for this comparison
  • 16.
    16 FP32 INT (W8A8) INT (W8A16) Best FP8 result ResNet1869.72% 69.55% 69.75% 69.66% HRNet 81.05 80.93 81.08 81.04 INT W8A16 accuracy is better than FP8 for all models with PTQ No real gap FP8/INT8 FP32 INT (W8A8) INT (W8A16) Best FP8 result BERT 83.06 71.03 82.90 82.80 SalsaNext 55.80 54.22 55.82 55.67 ViT 77.75% 76.39% 77.73% 77.71% MobileNetV2 69.72% 69.55% 69.75% 69.66% • Min-max range setting • Per-channel quantization
  • 17.
    17 INT16 performs better thanFP16 unless there are large outliers • 1,000 samples of 𝑋 ~ 𝑁𝑜𝑟𝑚𝑎𝑙(0, 1) • We add one outlier and vary its value • INT13 performs comparable to FP16 in terms of MSE 10−4 10−5 10−7 10−6 0 250 500 750 1000 Outlier 1250 1500 1750 2000 MSE MSE Error for w/o activation function for sigma =1.0, 100 neurons int16 float16
  • 18.
    18 INT16 outperforms FP16 in accuracy andruns faster in hardware MobileNetV2 71.74% FP32 71.69% FP16 71.74% INT16 EfficientDet-D1 40.08 FP32 40.07 FP16 40.07 INT16
  • 19.
    19 19 Integer quantization isthe way to do AI inference Enabled through PTQ and QAT techniques Mixed precision gives the best of both worlds, using extra precision only when necessary INT 4 INT 8 INT 16 Best Better Good Power efficiency and latency Best Better Good Accuracy
  • 20.
    Overcoming oscillations inquantization-aware training Improving quantization-aware training at lower bit-widths
  • 21.
    21 21 Poor validation accuracyis consistent across various learning rates and epochs during QAT “Overcoming Oscillations in Quantization-Aware Training” (ICML 2022) Validation accuracy for QAT is typically unstable Why do we see the validation accuracy drop for 4-bit QAT? Training accuracy Epoch Different learning rates (LR) 71% 67% 62% 2 4 6 8 10 12 14 16 18 20 Validation accuracy Epoch 44% 70% 60% 50% 42% 2 4 6 8 10 12 14 16 18 20
  • 22.
    22 22 2 0 −2 −4 Oscillations are presentin QAT Example of MobileNetV2 training (last 1000 iterations of training) Quantized weights, 𝑞 𝑤 (Lowest bit of 4−bit weight) Latent weights, 𝑤 (Positive FP values zoomed on 0.5) Sign and Bit0 weights Iterations during QAT FP weights 0.5004 0.5002 0.5000 0.4998 0.4996 0 200 400 600 1000
  • 23.
    23 Network Bits Pre-BNAcc. Post-BN Acc. MobileNetV2 8 71.79 0.07 71.89 0.05 MobileNetV2 4 68.99 0.44 71.01 0.05 MobileNetV2 3 64.97 1.23 69.50 0.04 Method Train Loss Val. Acc. (%) Baseline 1.3566 69.50 SR (mean + std) 1.3547 0.0053 69.58 0.09 SR (best) 1.3391 69.85 AdaRound 1.3070 70.12 +2.02 +4.53 Corrupts batch norm statistics • At inference, BN uses running statistics from training • Oscillations lead to big changes in statistics -> running statistics are not a good estimate • Solution: BN re-estimation Disrupts model convergence • At the end of training, oscillating weights may not be on the correct ‘side’ • Stochastic rounding (SR) and binary optimization (AdaRound) show that they are indeed not in the best possible state. • Oscillations prevent network from converging to best local minimum Oscillating weights are harmful when training a model 23
  • 24.
    24 EMA = ExponentialMoving Average Higher oscillation frequencies during QAT negatively impact accuracy Oscillation occurs The integer value changes & Its direction is opposite to its previous one
  • 25.
    25 25 “Overcoming Oscillations inQuantization-Aware Training” (ICML 2022) Oscillation dampening and iterative freezing fix the QAT issue Dampening takes a regularizing approach: the weights are forced closer to the bin center Freezing the oscillating weights stabilizes training and mitigates the unwanted effects of oscillations 100 80 60 40 20 0 −0.4 −0.2 0.0 0.2 0.4 wint − w/s wint − w/s −0.4 −0.2 0.0 0.2 0.4 400 300 200 100 0 Dampening Freezing Frozen Not frozen
  • 26.
    26 1: “Overcoming Oscillationsin Quantization-Aware Training” (ICML 2022) 2: “Learned step size quantization” (ICLR,2020) We achieve SOTA results for INT4 quantization1 • Train with learned step-size quantization (LSQ2) and re-estimation • Dampening and freezing perform on par with each other Method W/A Val. Acc. (%) Full-precision 32/32 65.1 LSQ* (Esser et al., 2020) 4/4 61.0 (-4.1) LSQ + BR (Han et al., 2021) 4/4 61.5 (-3.6) LSQ + Dampen (ours) 4/4 63.7 (-1.4) LSQ + Freeze (ours) 4/4 63.6 (-1.5) Method W/A Val. Acc. (%) Full-precision 32/32 71.7 LSQ* (Esser et al., 2020) 4/4 69.5 (−2.3) LSQ + BR (Han et al., 2021) 4/4 70.4 (−1.4) LSQ + Dampen (ours) 4/4 70.5 (−1.2) LSQ + Freeze (ours) 4/4 70.6 (−1.1) MobileNetV3 MobileNetV2
  • 27.
    27 Open-source projects toscale energy-efficient AI to the masses AIMET & AIMET Model Zoo
  • 28.
    28 28 AIMET makes AImodels small Open-sourced GitHub project that includes state-of-the-art quantization and compression techniques from Qualcomm AI Research Features: State-of-the-art network compression tools State-of-the-art quantization tools Support for both TensorFlow and PyTorch Benchmarks and tests for many models Developed by professional software developers If interested, please join the AIMET GitHub project: https://github.com/quic/aimet Trained AI model AI Model Efficiency Toolkit (AIMET) Optimized AI model TensorFlow or PyTorch Compression Quantization Deployed AI model
  • 29.
    AIMET Providing advanced model efficiency featuresand benefits Benefits Lower memory bandwidth Lower power Lower storage Higher performance Maintains model accuracy Simple ease of use Features Quantization Compression State-of-the-art INT8 and INT4 performance Quantization-aware training (QAT) Efficient tensor decomposition and removal of redundant channels in convolution layers Spatial singular value decomposition (SVD) Channel pruning Visualization Analysis tools for drawing insights for quantization and compression Weight ranges Per-layer compression sensitivity Quantization simulation Post-training quantization (PTQ) methods: • Data-Free Quantization • Adaptive Rounding (AdaRound), • Automatic Mixed Precision (AMP) • AutoQuant 29
  • 30.
    30 30 AIMET features andAPIs are easy to use Designed to fit naturally in the AI model development workflow for researchers, developers, and ISVs PyTorch model PyTorch Model Train No change Same API PyTorch Model Train Create QuantSim Evaluate Typical model training workflow User-friendly QAT workflow in AIMET No change Same API Evaluate User-friendly APIs invoked directly from the existing model pipeline Example Jupyter notebooks on AIMET GitHub AIMET extensions extensions Model optimization library (techniques to compress & quantize models) Framework specific API Algorithm API Other frameworks
  • 31.
    31 Low resolution Superresolution First 4K super-resolution demo at 100+ FPS on mobile Our new machine-learning based super-resolution method 8-bit quantized model created using AIMET QAT
  • 32.
    32 With better PTQ andQAT techniques, more models will achieve better power efficiency AIMET enables accurate INT W4A8 for wide range of use cases Task Model FP32 INT W4A8 Classification ResNet50 76.10% 75.4% ResNet18 69.75% 68.96% EfficientNet-Lite 75.31% 74.33% Regnext 78.3% 77.2% Segmentation Deeplabv3 (RN-50) 76.07% 75.91% Super-resolution ABPN 31.97 dB 31.67 (dB) Pose detection PoseNet (HRNet-32) 0.765 0.763
  • 33.
    33 33 Comparison between FP32model and model quantized with AIMET AIMET quantizes transformers with high accuracy, comparable to FP32 Top-1 accuracy 81.30 FP32 80.88 INT8/ W8A16 (PTQ) ViT base GLUE 84.99 FP32 84.60 INT8 (QAT) RoBERTa base GLUE 82.73 FP32 81.95 INT8 (QAT) BERT base (uncased) GLUE 79.21 FP32 78.61 INT8 (QAT) DistilBERT base (uncased)
  • 34.
    34 34 AIMET Model Zoo Accurate pre-trained8-bit quantized models Image classification Semantic segmentation Pose estimation Speech recognition Object detection Super resolution
  • 35.
    35 35 35 *: Comparison betweenFP32 model and INT8 model quantized with AIMET. For further details, check out: https://github.com/quic/aimet-model-zoo/ AIMET Model Zoo includes popular quantized AI models Accuracy is maintained for INT8 models — less than 1% loss* Top-1 accuracy* 75.21% FP32 74.96% INT8 ResNet-50 (v1) Top-1 accuracy* 75% FP32 74.21% INT8 MobileNet-v2-1.4 Top-1 accuracy* 74.93% FP32 74.99% INT8 EfficientNet Lite mAP* 0.2469 FP32 0.2456 INT8 ResNet-50 (v1) mAP* 0.35 FP32 0.349 INT8 RetinaNet mAP* 0.383 FP32 0.379 INT8 Pose estimation PSNR* 25.45 FP32 24.78 INT8 SRGAN Top-1 accuracy* 71.67% FP32 71.14% INT8 MobileNetV2 Top-1 accuracy* 75.42% FP32 74.44% INT8 EfficientNet-lite0 mIoU* 72.62% FP32 72.22% INT8 DeepLabV3+ mAP* 68.7% FP32 68.6% INT8 MobileNetV2-SSD-Lite mAP* 0.364 FP32 0.359 INT8 Pose estimation PSNR 25.51 FP32 25.5 INT8 SRGAN WER* 9.92% FP32 10.22% INT8 DeepSpeech2 PSNR 32.75 FP32 32.69 INT8 ABPN <1% Loss in accuracy*
  • 36.
    36 *: Comparison betweenFP32 model and INT8 model quantized with AIMET. For further details, check out: https://github.com/quic/aimet-model-zoo/ Super resolution model suite Wide variety of models, suited for fast, energy-efficient INT8 inference • Virtually no accuracy loss compared to FP32 • Simple and convenient for developer integration • Useful across diverse applications, from gaming and photography to XR and autonomous driving 1 Anchor-based Plain Net (ABPN) 2 Robust Real-Time Single-Image Super Resolution (XLSR) 3 Super-Efficient Super Resolution (SESR) PSNR (dB) 32.66 FP32 32.58 INT8 SESR-M73 PSNR (dB) 32.41 FP32 32.25 INT8 SESR-M33 PSNR (dB) 32.71 FP32 32.64 INT8 ABPN1 PSNR (dB) 32.57 FP32 32.30 INT8 XLSR2 PSNR (dB) 33.03 FP32 32.92 INT8 SESR-XL3 INT8 PSNR and visual quality comparable to FP32*
  • 37.
    37 37 Explore our open-sourceprojects and tools AIMET State-of-the-art quantization and compression techniques github.com/quic/aimet AIMET Model Zoo Accurate pre-trained 8-bit quantized models github.com/quic/aimet-model-zoo Quantization whitepaper arxiv.org/abs/2201.08442
  • 38.
    AI Frameworks Qualcomm® NeuralProcessing SDK TF Lite TF Lite Micro Direct ML AI Runtimes Programming Languages Virtual platforms Core Libraries Math Libraries Profilers & Debuggers Compilers System Interface SoC, accelerator drivers Auto XR Robotics IoT ACPC Smartphones Cloud Platforms Qualcomm® AI Engine Direct (QNN) Tools: Emulation Support Qualcomm Neural Processing SDK, Qualcomm AI Model Studio, and Qualcomm AI Engine Direct are products of Qualcomm Technologies, Inc. and/or its subsidiaries Qualcomm AI Model Studio AIMET AIMET Model Zoo NAS Model analyzers Infrastructure:
  • 39.
    39 Model efficiency iskey for enabling on-device AI and accelerating the growth of the connected intelligent edge INT8/16 perform better than FP8/16 Qualcomm AI Research is enabling 4-bit integer models AIMET is making fixed-point quantization possible at scale without sacrificing accuracy
  • 40.
  • 41.
    Nothing in thesematerials is an offer to sell any of the components or devices referenced herein. ©2018-2022 Qualcomm Technologies, Inc. and/or its affiliated companies. All Rights Reserved. Qualcomm, Hexagon, and Snapdragon are trademarks or registered trademarks of Qualcomm Incorporated. Other products and brand names may be trademarks or registered trademarks of their respective owners. References in this presentation to “Qualcomm” may mean Qualcomm Incorporated, Qualcomm Technologies, Inc., and/or other subsidiaries or business units within the Qualcomm corporate structure, as applicable. Qualcomm Incorporated includes our licensing business, QTL, and the vast majority of our patent portfolio. Qualcomm Technologies, Inc., a subsidiary of Qualcomm Incorporated, operates, along with its subsidiaries, substantially all of our engineering, research and development functions, and substantially all of our products and services businesses, including our QCT semiconductor business. Follow us on: For more information, visit us at: qualcomm.com & qualcomm.com/blog Thank you