“Efficient Many-function Video ML at the Edge,” a Presentation from Cisco Systems
•
0 likes•3 views
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2023/09/efficient-many-function-video-ml-at-the-edge-a-presentation-from-cisco-systems/ Chris Rowen, Vice President of AI Engineering for Webex Collaboration at Cisco Systems, presents the “Efficient Many-function Video ML at the Edge,” tutorial at the May 2023 Embedded Vision Summit. Video streams are so rich, and video workloads are so sophisticated, that we may now expect video ML to supply many simultaneous insights and transformations. It will be increasingly common to need video segmentation, object and motion recognition, SLAM, 3D model extraction, relighting, avatarization and neural compression in parallel. Conventionally, this combination would overwhelm edge compute resources, but novel multi-headed ML models and unified video pipelines make this feasible on existing personal devices and embedded compute subsystems. In this talk, Rowen discusses the goals for advanced video intelligence in secure, edge-powered video communications, and shows how new model structures can achieve very high accuracy, resolution and frame rate at low cost per function. He also discusses improved objective and subjective quality metrics, training set synthesis and his company’s optimized portable edge implementation methodology. Rowen wraps up with some observations on the challenges of even larger video workloads at the edge.
Recommended
Recommended
More Related Content
Similar to “Efficient Many-function Video ML at the Edge,” a Presentation from Cisco Systems
Similar to “Efficient Many-function Video ML at the Edge,” a Presentation from Cisco Systems (20)
More from Edge AI and Vision Alliance
More from Edge AI and Vision Alliance (20)
Recently uploaded
Recently uploaded (20)
“Efficient Many-function Video ML at the Edge,” a Presentation from Cisco Systems
- 1. Efficient Many-Function Video ML at the Edge Chris Rowen Will Reed Collaboration AI Cisco Systems
- 2. • Many ML tasks • Limited capacity: • Compute • Memory • Development time The Problem
- 3. • Many tasks, 1 model • Share common paths • Architecture • Data • Testing • Deployment Our Solution
- 4. © 2023 Cisco Systems 4 Before: 2 Tasks, 2 Models Segmentation Gestures
- 5. © 2023 Cisco Systems After: 2 Tasks, 1 Model Backbone Gestures Segmentation 5
- 6. Model Size Comparison: 2 functions 6 2.4M 2 300k 50k 5.15M 2.4M 1 300k 50k 2.75M Before: After: # Encoder Parameters # Models # Head Parameters TOTAL © 2023 Cisco Systems
- 7. © 2023 Cisco Systems 7 Architecture Encoder Features Decoder Head 1 Head 2 Head 3 Head 4 Head N • Configurable input size • Swappable encoders • 200 K, 420 K and 2.4 M param options • Configurable training • Train encoder, features and decoder jointly • Freeze encoder and train heads • Train encoder, lower learning rate and train heads video frames
- 8. • Data efficient architecture • Tasks can learn from each other’s data through generalization in the encoder • Similar to how models benefit from pre-training • Implicit regularization • Multiple tasks discourage the model from overfitting on any one task • Benefit from diversity of data in related tasks • Only need to label data for the task you care about • Quick and cheap to add a new task Data Strategy 8 © 2023 Cisco Systems
- 9. Training 9 Function Labels Inputs Back prop • Skip losses for unlabeled functions Encoder Features Decoder Head 1 Head 3 Head 4 Head N Loss1 Loss2 Loss3 Loss4 Weighted sum loss © 2023 Cisco Systems
- 10. © 2023 Cisco Systems Adding a New Task: Face Landmarks Detection Click to edit Master text styles • Second level • Third level • Fourth level • Fifth level 10
- 11. • Perform face and keypoints detection in one pass • Based on YOLO v2 (transitioning to v3 soon) • Remove classification loss • Add landmark localization loss • Computation is bounded SSLD: Single Shot Landmark Detection face center face box face classification (presence) classification confidence landmark locations 11 © 2023 Cisco Systems
- 12. 12 2.4M 4 300k (segment) 50k (gesture) 50k (landmarks) 30k (face loc) 11.2M 2.4M 1 2.8M 300k (segment) 50K (gesture) 50K (landmark+face loc) Model Size Comparison: 4 functions segmentation + gesture + face location+ face landmarks Separate: Unified: # Encoder Parameters # Models # Head Parameters TOTAL © 2023 Cisco Systems
- 13. Multi-Function Model Performance 13 0.00000 0.00002 0.00004 0.00006 0.00008 0.00010 0.80 0.82 0.84 0.86 0.88 0.90 0.92 0.94 0.96 0.98 1.00 0 0.1 0.2 0.3 0.4 0.5 0.6 Error (Keypoints) Accuracy (segmentation, gesture, face loc) Compute GMACs per frame - tiny, small, medium models Ladon Model Performance (4 tasks) Segmentation IoU Gesture Fscore Face box IoU Face keypoints mean-squared error tiny model small model medium model © 2023 Cisco Systems
- 14. Examples 14
- 15. Optimized Portable Edge Implementation 15 • Cross-platform • C++: Supports Windows, Mac, Linux, iOS, Android • Javascript: Recent versions of major browsers • Common ML framework • ONNX, CoreML, OpenVINO • CPU and GPU mode • Consistent results • Less conversion hell • High and low level API’s • Get high level predictions like a fully segmented and blurred output, 2 and 3D filter effects • Also low level access to segmentation masks, landmark coordinates, etc… Overlays & Backgrounds Avatars Be Right Back Encoder Features Decoder Foreground Segmentation Gestures Face Landmarks Gesture Actions Face Location & Size Relighting LUT Studio Lighting © 2023 Cisco Systems
- 16. • Two powerful methods • Teachers: • Build big teacher model from limited hand- labeled data • use it to bootstrap big data set • Train range of small models • Synthetic Scenes: • Diverse • 3D environments • faces • poses • lighting • Programmatically animate 3D avatars Training Label & Data Synthesis 16 Labeled data Teacher model 1.Initial training Unlabeled data 2.Rough labeling Human selection/correction Semi-automatic labeled data 3.Training bootstrap Production models 4.Product training © 2023 Cisco Systems
- 17. 1. Richer applications explosion of video ML needs compute crisis? 2. Multi-headed vision model is a form of ”foundation model” like GPT-4 robustness through task diversity 3. Smart algorithmic labeling can replace much hand labeling 4. Estimating algorithm FLOPS is an imperfect predictor of implementation throughput – not every layer is a convolution 5. Implementing N functions together complicates loss function design and training 6. Raising system functionality may span many platforms portability 7. Emergence of edge CPU neural accelerators may open door to more aggressive video ML workloads, but uneven time-lines for availability 8. Conventional wisdom says diverse training tasks together often doesn’t work. Conventional wisdom is often wrong. Lessons Learned 17 © 2023 Cisco Systems
- 18. • Semantic Segmentation: https://www.v7labs.com/blog/semantic-segmentation-guide • Adaptive Re-lighting: https://arxiv.org/pdf/2009.14468.pdf • Original YOLO paper: https://arxiv.org/abs/1506.02640 • Knowledge distillation: https://www.v7labs.com/blog/knowledge-distillation-guide • Multi-task learning: https://towardsdatascience.com/multi-task-learning-in-machine-learning- 20a37c796c9c • Dataset Distillation: https://ai.googleblog.com/2021/12/training-machine-learning-models- more.html • Intro to self-supervised learning: https://ai.facebook.com/blog/self-supervised-learning-the-dark- matter-of-intelligence/ Resources 18 © 2023 Cisco Systems