“A Survey of Model Compression Methods,” a Presentation from Instrumental
•
0 likes•40 views
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/10/a-survey-of-model-compression-methods-a-presentation-from-instrumental/ Rustem Feyzkhanov, Staff Machine Learning Engineer at Instrumental, presents the “Survey of Model Compression Methods” tutorial at the May 2023 Embedded Vision Summit. One of the main challenges when deploying computer vision models to the edge is optimizing the model for speed, memory and energy consumption. In this talk, Feyzkhanov provides a comprehensive survey of model compression approaches, which are crucial for harnessing the full potential of deep learning models on edge devices. Feyzkhanov explores pruning, weight clustering and knowledge distillation, explaining how these techniques work and how to use them effectively. He also examines inference frameworks, including ONNX, TFLite and OpenVINO. Feyzkhanov discusses how these frameworks support model compression and explores the impact of hardware considerations on the choice of framework. He concludes with a comparison of the techniques presented, considering implementation complexity and typical efficiency gains.
Recommended
Recommended
More Related Content
Similar to “A Survey of Model Compression Methods,” a Presentation from Instrumental
Similar to “A Survey of Model Compression Methods,” a Presentation from Instrumental (20)
More from Edge AI and Vision Alliance
More from Edge AI and Vision Alliance (20)
Recently uploaded
Recently uploaded (20)
“A Survey of Model Compression Methods,” a Presentation from Instrumental
- 1. A Survey of Model Compression Methods Rustem Feyzkhanov Staff Machine Learning Engineer Instrumental
- 2. • Cloud-to-edge ML pipeline and optimizations • Model serialization • Pruning • Quantization • Weight clustering • Knowledge distillation • Architecture search • Summary Agenda 2 © 2023 Instrumental
- 3. • Latency • Memory usage • Energy usage • Disk usage Challenges with ML on Edge 3 © 2023 Instrumental
- 4. Cloud-to-edge ML pipeline 4 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference • Data acquisition • Data labeling • Feature selection • Model architecture • Cloud training pipeline • Parameter optimizations • Serialization • Post training optimizations • Compiling model in cloud for hardware • Serving model to the edge • Initializing model • Making predictions
- 5. Optimizations in ML pipeline 5 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference
- 6. Optimizations in ML pipeline 6 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference Easier to implement/add Affects accuracy less
- 7. 7 © 2023 Instrumental Model serialization
- 8. • Hardware specific optimizations • Framework built-in native model compression methods • Significantly affects speed Inference frameworks and model serialization 8 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference
- 9. Frameworks focused on running inference. Lightweight, focused on specific hardware, require separate serialization. • Hardware agnostic: e.g., TorchScript, ONNX, TFLite* • Hardware specific: • CPU: e.g., OpenVINO (Intel) • GPU: e.g., TensorRT • Mobile: e.g., CoreML • NPU: Check out Embedded Vision Alliance members :-) Inference frameworks 9 © 2023 Instrumental
- 10. • Object detection - YOLOv5s Inference frameworks - example 10 © 2023 Instrumental Framework Size, MB CPU inference, ms GPU (V100) inf, ms PyTorch 29.5 127.61 10.19 TorchScript 29.4 131.23 6.85 TensorRT 33.3 N/A 1.89 ONNX 29.3 69.34 14.63 OpenVINO 29.3 66.52 N/A TFLite 29.0 316.61 N/A From https://docs.ultralytics.com/yolov5/tutorials/model_export/
- 12. • Eliminating redundant or unimportant parameters • Affects accuracy, but can be addressed by finetuning • Affects size more than speed Pruning 12 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference
- 13. • Weight pruning • Structural pruning • Neuron • Layer • Filter • Channel Types of pruning 13 © 2023 Instrumental Weight pruning Neuron pruning
- 14. Weight pruning example 14 © 2023 Instrumental • Image classification example – weight pruning Configuration Number of parameters Top-1 accuracy, ImageNet InceptionV3 Original 27.1M 78.1% 50% sparsity 13.6M 78.0% 75% sparsity 6.8M 76.1% 87.5% sparsity 3.3M 74.6% From https://www.tensorflow.org/model_optimization/guide/pruning
- 16. • Reducing numerical precision of weights and activations • Affects accuracy, but can be addressed during training • Checkout the talk “Practical Approaches to DNN Quantization” later today Quantization 16 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference
- 17. Quantization example 17 © 2023 Instrumental • Image classification example – quantization Configuration Size, MB Inference (CPU), ms Top-1 accuracy, ImageNet InceptionV3 Original 95.7 1130 78.1% Post training quantization 23.9 845 77.2% Quantization aware training 23.9 543 77.5% From https://www.tensorflow.org/lite/performance/model_optimization
- 18. 18 © 2023 Instrumental Weight clustering
- 19. • Cluster model weights and use indices • Only optimizes model size • Similar to quantization, but doesn’t change computation complexity Weight clustering 19 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference
- 20. Weight clustering example 20 © 2023 Instrumental • Replaces weights with reference to the closest centroids • Centroids are usually rounded, but not quantized so main gain is model size 0.86 -1.88 -1.44 -1.7 1.58 0.12 1.9 0.46 1.37 1 2 2 2 3 1 3 1 3 Centroid Index 0.33 1 -1.26 2 1.92 3 Weight matrix
- 21. Weight clustering example 21 © 2023 Instrumental • Image classification example Configuration Size, MB Top-1 accuracy, ImageNet MobileNetV2 Original 12.38 71.7% Last 3 layers, 32 clusters 7.03 70.9% Last 3 layers, 16 clusters 6.68 70.7% All layers, 32 clsuters 4.05 69.7% From https://www.tensorflow.org/model_optimization/guide/clustering
- 22. 22 © 2023 Instrumental Knowledge distillation
- 23. • Smaller, more efficient "student" model learns to mimic the behavior of a larger, pre-trained "teacher" model Knowledge distillation 23 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference
- 24. Student Teacher Training data Knowledge distillation - example 24 © 2023 Instrumental • Only student model is trained • Student model and teacher model run on the same images • Error is propagated back for student mode Predictions Predictions True label
- 25. Knowledge distillation - example 25 © 2023 Instrumental • Image classification example From https://www.analyticsvidhya.com/blog/2022/01/knowledge-distillation-theory-and-end-to-end-case-study/ Base model No. of parameters Test accuracy Teacher model VGG16 27 M 77% Student model with Distillation VGG16 pruned 296 k 75% Student model without Distillation VGG16 pruned 296 k 64%
- 26. • Finding smaller models which have less parameters and have faster predictions Optimizing model architecture 26 © 2023 Instrumental Data gathering Architecture search Model training Model compilation Deployment and model inference
- 27. Model architecture - example 27 © 2023 Instrumental • Object detection – YOLOv5 family From https://github.com/ultralytics/yolov5 Model Size, MB mAP, COCO CPU inf, ms GPU (V100) inf, ms YOLOv5n 4.1 45.7 45 6.3 YOLOv5s 14.8 56.8 98 6.4 YOLOv5m 42.8 64.1 224 8.2 YOLOv5l 93.6 67.3 430 10.1 YOLOv5x 174.1 68.9 766 12.1
- 29. Comparing optimization techniques 29 © 2023 Instrumental Optimization Size decrease Speed increase Inference framework Low High Pruning High Low Quantization High High Weight clustering Low Low Architecture search High High
- 30. • Use a test dataset to assess performance changes • Impacts vary (e.g., less for image classification, more for object detection) • Integrate compilation optimizations into training for optimal model selection • Test on target hardware for deployment • Balance trade-offs: Understand acceptable accuracy loss and business metric impact Recommendations on choosing model compression techniques 30 © 2023 Instrumental
- 31. • Open-source • Inference or training frameworks with built-in solutions • Commercial • A number of companies specifically focus on optimizing your models (some of them are Alliance Members) Open-source vs commercial 31 © 2023 Instrumental
- 32. • Begin with an open-source solution for quick optimization gains • Consider commercial for specialized hardware (e.g., mobile) with easy trial options • Use a test set to validate accuracy trade-offs for both approaches Open-source vs commercial 32 © 2023 Instrumental
- 33. • Model compression is vital for cloud-to-edge ML pipelines • Streamlined training pipeline enables easy exploration of approaches • Integrating compression in training pipeline ensures optimal accuracy Conclusions 33 © 2023 Instrumental
- 34. • Inference framework guides • OpenVINO https://docs.openvino.ai/latest/openvino_docs_model_optimization_guide.html • TFLite https://www.tensorflow.org/lite/performance/model_optimization • ONNX https://onnxruntime.ai/docs/performance/model-optimizations/ • Books • TinyML https://www.oreilly.com/library/view/tinyml/9781492052036/ • Deep Learning with PyTorch https://livebook.manning.com/book/deep-learning-with-pytorch/ Resources 34 © 2023 Instrumental