This document discusses evaluating the quality of machine learning models. It begins with an agenda that includes evaluating the quality process, techniques for evaluation, and an introduction to Snitch AI. Key points made include: - ML model quality is important for relevance, maintainability, ethics, and reliability. - Characteristics of quality include the ability to learn, generalization, robustness, and avoiding biases and discrimination. - Techniques discussed for evaluating quality include measuring a model's ability to learn, robustness to noise, feature contributions, and evolution over time. - Snitch AI is introduced as a tool that can open the black box of models, mitigate data drift, and provide visualizations of model quality.

1. Evaluate Your ML Model Quality

2. Work on the development of ISO standards for artificial
intelligence with the Standards Council of Canada.
Mentor at Creative Destruction Labs where he helps startups
across Canada succeed with AI.
A speaker on digital transformation and the use of artificial
intelligence in business, he was appointed 30/30 in 2019.
Who Am I?

3. Ø Quality evaluation process
Ø Quality evaluation techniques
Ø Introduction to Snitch AI
Agenda

4. ML Model Quality Evaluation
Process

6. Project team
Data Science Team
Stakeholders
Society
Why Is Quality Evaluation Important

7. Relevance MaintainabilityEthicsReliability
Characteristics of ML Model Quality

8. Ø Ability to learn
Ø Generalization
Relevance MaintainabilityEthicsReliability
Characteristics of ML Model Quality

9. Ø Ability to learn
Ø Generalization
Ø Robustness
Ø Complexity
Relevance MaintainabilityEthicsReliability
Characteristics of ML Model Quality

10. Ø Robustness
Ø Complexity
Ø Data biases
Ø Discrimination
Relevance MaintainabilityEthicsReliability
Characteristics of ML Model Quality
Ø Ability to learn
Ø Generalization

11. Ø Data biases
Ø Discrimination
Ø Data drift
Ø Model degradation
Relevance MaintainabilityEthicsReliability
Characteristics of ML Model Quality
Ø Robustness
Ø Complexity
Ø Ability to learn
Ø Generalization

12. Evaluation Process During Project Delivery
Continuous Improvement
Monitor data and model degradation
Implementation
Evaluate overall quality prior to deployment
Proof of Concept
Evaluate relevance and reliability
Project kickoff
Define project risks

13. ML Model Quality Evaluation
Techniques

14. Measure a System’s Ability to
Learn

15. Evaluate Ability to Learn
Performance
Complexity
Dummy model
Simple model
Complex model
Random
Theorical threshold
Minimum threshold

16. Measure Model’s
Robustness

17. Analysis of Robustness to Random Noise

18. Extreme Noise (Adversarial Noise)
+ 0.7% x =
« Panda »
57.7% confidence
« Gibbon »
99.3% confidence

19. Analysis of Robustness to Extreme Noise

20. Visualize Discrimination in
Model’s Assumptions

21. Measure Feature’s Contribution to Predictions
High
value
Low
value

22. Measure Model’s
Evolution

23. Measure Population Stability (PSI)
Baseline population
New population
Buckets
1 2 3 4 5 6 7 8 9 10
0%
20%
10%
Population’sperentage

24. Data Separability
Proportionofobservations
Probabillity that the data in the baseline or new dataset
Baseline dataset New Dataset
The goal is to reach
an AUC of 50% !

25. Uncertainty Evolution – MC Dropout !

26. Uncertainty Evolution
Predictedvalue

27. Intro to Snitch AI

28. Open the Black Box
Relevance MaintainabilityEthicsReliability

29. Mitigate Inevitable Data Drift
Model Training Model deployment in production

30. What we Evaluate
• Data drift
• Feature biases
• Model robustness to random noise
• Model robustness to extreme noise
• Underfitting and overfitting *
• Labelling errors *
• Data leakage *
• Model simplification opportunities *
• Feature pruning opportunities *
* Upcoming

31. How Does Snitch AI Work ?
Save model and
data
Import into
Snitch
Creation of a
report/ json

32. Model Quality Visualization

33. Model Quality Visualization

34. Model Quality Visualization

35. Thank you !
Question ?
olivier.blais@moov.ai