Train & Sustain: Why data leaders need to pay attention to HITL
•
0 likes•33 views
Humans are still very much needed in AI, ML, and automation. In this presentation, our Senior Solutions Consultant, Matt Beale, discusses why deep learning needs human expertise to succeed and why blending human intelligence and technology to build scalable models for production is the best approach.
Recommended
Recommended
More Related Content
Similar to Train & Sustain: Why data leaders need to pay attention to HITL
Similar to Train & Sustain: Why data leaders need to pay attention to HITL (20)
Recently uploaded
Recently uploaded (20)
Train & Sustain: Why data leaders need to pay attention to HITL
- 1. 1 Train & Sustain: Why data leaders need to pay attention to HITL May 9, 2023
- 2. Place image here within cropped circle ©2019 CloudFactory Confidential 2 ©2023 CloudFactory Confidential Hi, I’m Matt Originally from Cambridge, Matt now helps clients move to a data centric ML approach as a Senior Solutions Consultant at CloudFactory. He’s worked with clients across autonomous vehicles, green energy and fintech whilst providing meaningful work in the developing world.
- 3. Place image here within cropped circle ©2019 CloudFactory Confidential3 ©2023 CloudFactory Confidential About CloudFactory CloudFactory is a global leader in combining people and technology to support the AI development lifecycle, from data curation and annotation, to quality assurance and model optimization. Our human-in-the-loop AI solutions are trusted by AI leaders at 700+ companies globally. 40M+ 7,000+ 64 $78M project hours delivered data analysts Client NPS score in growth funding
- 5. Place image here within cropped circle In 2012, Computer computer vision pioneer Geoffrey Hinton won important Large Scale Visual Recognition Challenge (ILSVRC2012) competition with Deep Learning. Deep Learning and ILSVRC 2012
- 6. 6 ©2023 CloudFactory Confidential Traditional wisdom since that time was that more layers (and more data) was better Source: Research Gate Shallow 152 Layers
- 7. “One thing ImageNet changed in the field of AI is suddenly people realized the thankless work of making a dataset was at the core of AI research.” Fei-Fei Li Professor, Computer Science Co-Director, Stanford Institute of Human-Centered AI (HAI) Stanford University
- 8. 8 ©2023 CloudFactory Confidential But at a certain point, that approach stopped being as effective and attention turned to the data instead “20% of activities are automatable by AI advances in areas like visual object recognition. - McKinsey “87% of applied ML projects never make it to production” - Gartner A small group of teams emerges succeeding with applied ML due to a data-centric approach.” - Andrew Ng 2017 2019 2021
- 9. 9 ©2023 CloudFactory Confidential When you focus on having humans label the right things, you can dramatically increase model performance Active learning used to prioritize workflow
- 10. 10 Deep Learning Isn’t Just About Training Data
- 11. 11 ©2023 CloudFactory Confidential Historically, companies have used HITL primarily for labeling training data Label Data QA Data Train Model Monitor Model Deploy Model Evaluate Model TRAIN SUSTAIN
- 12. 12 ©2023 CloudFactory Confidential Though automation can speed the initial labeling process, it can’t do everything without human help
- 13. 13 ©2023 CloudFactory Confidential Though automation can speed the initial labeling process, it can’t do everything without human help
- 14. 14 ©2023 CloudFactory Confidential Though automation can speed the initial labeling process, it can’t do everything without human help
- 15. 15 ©2023 CloudFactory Confidential Though automation can speed the initial labeling process, it can’t do everything without human help
- 16. 16 ©2023 CloudFactory Confidential You still need both humans and automation to get AI to production reliably Level 1: Out-of-the-Box Annotation with out-of-the-box assistants Level 2: Custom Trained Annotation with adaptive, custom assistants Level 3: Automated Annotation with custom trained model Input data* IMAGES VIA API Active learning ensures only useful images are labeled Output data IMAGES + LABELS VIA API
- 17. 17 ©2023 CloudFactory Confidential The future is to use humans for train and for sustain Label Data QA Data Train Model Monitor Model Deploy Model Evaluate Model TRAIN SUSTAIN
- 18. 18 ©2023 CloudFactory Confidential Getting your AI to production reliably Level 1: Out-of-the-Box Annotation with out-of-the-box assistants Level 2: Custom Trained Annotation with adaptive, custom assistants Level 3: Automated Annotation with custom trained model Input data* IMAGES VIA API Data insight report Capture insights and progress for review Output data IMAGES + LABELS VIA API cabbage potato grain grain grain AI Consensus scoring Review to ensure quality and resolve ambiguity Active learning ensures only useful images are labeled
- 19. 19 ©2023 CloudFactory Confidential AI Consensus Scoring Using Bayesian Neural Networks and Confident Learning to check every label at least twice Edge cases are reviewed by a human. The results generate quality reports. Our AI checks the data for a variety of error types. Dog Wrong class Low IoU Artefacts Missing labels
- 20. 20 ©2023 CloudFactory Confidential Inference Validation Example
- 21. 21 Finding the right HITL expertise at the right time
- 24. 24 ©2023 CloudFactory Confidential The decision to add humans-in-the-loop depends on your place in the AI lifecycle: incubation, scale training, or production. When you’re ready - here are four things to consider: Criteria for selecting the right HITL #1 #2 #3 #4 Use case experience Flexibility (new use cases, speed, etc) Workforce management technology Ethics and risk management
- 25. 25 Questions?