Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
You vs Ryu(r) algorithm - Be the smartest
- 1. Be the smartest Virginie Mathivet R&D Officer Ryu(r) Algorithm https://danbooru.donmai.us/posts/948811 http://vsbattles.wikia.com/wiki/File:RyuRender.png
- 2. dotAI’18 - Virginie Mathivet – R&D Officer The accuracy paradox
- 3. dotAI’18 - Virginie Mathivet – R&D Officer The accuracy paradox
- 4. dotAI’18 - Virginie Mathivet – R&D Officer Fault detection Infrared HQ images Bubble Scratch Micro defectBorder defect
- 5. dotAI’18 - Virginie Mathivet – R&D Officer First results • Accuracy: 99.1%!!! (in 2 epochs)
- 6. dotAI’18 - Virginie Mathivet – R&D Officer First results • Accuracy: 99.1%!!! (in 2 epochs) Real Predicted OK (1665) KO (15) Accuracy OK 1665 0 100% KO 15 0 0% Total 1680 0 99.1% • Confusion matrix:
- 7. dotAI’18 - Virginie Mathivet – R&D Officer The accuracy paradox and imbalanced classes 0 10 20 30 40 50 60 70 80 90 100 Classes Real Pred. 1 2 3 Accuracy 1 90 0 0 100% 2 5 0 0 0% 3 5 0 0 0% TOTAL 100 0 0 90%
- 8. dotAI’18 - Virginie Mathivet – R&D Officer What to do?
- 9. dotAI’18 - Virginie Mathivet – R&D Officer Data augmentation … New dataset: • 1665 images “OK” (94.9%) • 90 images “KO” (5.1%)
- 10. dotAI’18 - Virginie Mathivet – R&D Officer Data augmentation … New dataset: • 1665 images “OK” (94.9%) • 90 images “KO” (5.1%) Accuracy: 99.15%
- 11. dotAI’18 - Virginie Mathivet – R&D Officer Data augmentation Real Predicted OK (1665) KO (90) Accuracy OK 1665 0 100% KO 15 75 83.3% Total 1680 75 99.15% … New dataset: • 1665 images “OK” (94.9%) • 90 images “KO” (5.1%) Accuracy: 99.15%
- 12. dotAI’18 - Virginie Mathivet – R&D Officer Data augmentation Real Predicted OK (1665) KO (90) Accuracy OK 1665 0 100% KO 15 75 83.3% Total 1680 75 99.15% … New dataset: • 1665 images “OK” (94.9%) • 90 images “KO” (5.1%) Accuracy: 99.15%
- 13. dotAI’18 - Virginie Mathivet – R&D Officer The evaluation function definition
- 14. dotAI’18 - Virginie Mathivet – R&D Officer The evaluation function definition
- 15. dotAI’18 - Virginie Mathivet – R&D Officer The choice of the evaluation function Classification Real Predicted 1 0 1 A (true positive) C (false negative) 0 B (false positive) D (true negative) Accuracy Precision Recall / Sensitivity F1-score (A+D)/(A+B+C+D) A/(A+B) A/(A+C) 2*precision*recall/(precision+recall) Specificity Detection rate Youden’s J Index Balanced accuracy D/(B+D) A/(A+B+C+D) Sensitivity+specificity - 1 (sensitivity+specificity) / 2
- 16. dotAI’18 - Virginie Mathivet – R&D Officer The choice of the evaluation function Regression • R-squared (R2) • Root Mean Squared Error (RMSE) • Residual Standard Error (RSE) or Model Sigma • Mean Absolute Error (MAE) • Akaike’s Information Criteria (AIC) • Bayesian Information Criteria (BIC) • …
- 17. dotAI’18 - Virginie Mathivet – R&D Officer The choice of the evaluation function 0 5 10 15 20 25 0 1 2 3 4 5 6 7 8 9 10 Behavioral task
- 18. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 1st try: distance made in 10sec
- 19. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 1st try: distance made in 10sec https://gymnasticselectronicportfolio13.weebly.com/gymnastics-skills.html
- 20. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 1st try: distance made in 10sec https://gymnasticselectronicportfolio13.weebly.com/gymnastics-skills.html https://viatcheslav.livejournal.com/231564.html
- 21. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 2nd try: distance made in 10sec without rolling
- 22. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 2nd try: distance made in 10sec without rolling https://blog.myagilepartner.fr/wp-content/uploads/2017/08/scared_bug.jpg
- 23. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 2nd try: distance made in 10sec without rolling https://www.memecenter.com/fun/1717889/flying-spiderhttps://blog.myagilepartner.fr/wp-content/uploads/2017/08/scared_bug.jpg
- 24. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 3rd try: idem but after the bug correction
- 25. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 3rd try: idem but after the bug correction https://www.nemusloren.it/2013/11/30/come-se-fosse-possibile/
- 26. dotAI’18 - Virginie Mathivet – R&D Officer Teach a (virtual) spider how to walk • 3rd try: idem but after the bug correction https://www.nemusloren.it/2013/11/30/come-se-fosse-possibile/
- 27. dotAI’18 - Virginie Mathivet – R&D Officer Rule of thumb: speak to a child Eat with a fork:
- 28. dotAI’18 - Virginie Mathivet – R&D Officer Rule of thumb: speak to a child Eat with a fork:
- 29. dotAI’18 - Virginie Mathivet – R&D Officer Rule of thumb: speak to a child Eat with a fork:
- 30. dotAI’18 - Virginie Mathivet – R&D Officer Rule of thumb: speak to a child Eat with a fork:
- 31. dotAI’18 - Virginie Mathivet – R&D Officer The overfitting spotting
- 32. dotAI’18 - Virginie Mathivet – R&D Officer The overfitting spotting
- 33. dotAI’18 - Virginie Mathivet – R&D Officer Overfitting spotting Error Time Training Validation Underfitting Overfitting Optimum
- 34. dotAI’18 - Virginie Mathivet – R&D Officer Teaching a robot how to avoid walls • Evaluation function : • 1st try: time until a collision with a wall (max 10sec)
- 35. dotAI’18 - Virginie Mathivet – R&D Officer Teaching a robot how to avoid walls • Evaluation function : • 1st try: time until a collision with a wall (max 10sec)
- 36. dotAI’18 - Virginie Mathivet – R&D Officer Teaching a robot how to avoid walls • Evaluation function : • 1st try: time until a collision with a wall (max 10sec) • 2nd try: time moving until a collision with a wall (max 10sec)
- 37. dotAI’18 - Virginie Mathivet – R&D Officer Teaching a robot how to avoid walls • Evaluation function : • 1st try: time until a collision with a wall (max 10sec) • 2nd try: time moving until a collision with a wall (max 10sec)
- 38. dotAI’18 - Virginie Mathivet – R&D Officer Teaching a robot how to avoid walls • Evaluation function : • 1st try: time until a collision with a wall (max 10sec) • 2nd try: time moving until a collision with a wall (max 10sec) • 3rd try: time moving forward until a collision with a wall (max 10sec)
- 39. dotAI’18 - Virginie Mathivet – R&D Officer Teaching a robot how to avoid walls • Evaluation function : • 1st try: time until a collision with a wall (max 10sec) • 2nd try: time moving until a collision with a wall (max 10sec) • 3rd try: time moving forward until a collision with a wall (max 10sec)
- 40. dotAI’18 - Virginie Mathivet – R&D Officer Theory vs Reality • Real robot and real cage => not working at all!! • Unseen overfitting!! (no dataset = no curves)
- 41. dotAI’18 - Virginie Mathivet – R&D Officer Theory vs Reality • Real robot and real cage => not working at all!! • Unseen overfitting!! (no dataset = no curves)
- 42. dotAI’18 - Virginie Mathivet – R&D Officer Theory vs Reality • Real robot and real cage => not working at all!! • Unseen overfitting!! (no dataset = no curves) • Solution : • Randomness • Test on real robots as soon as possible
- 43. dotAI’18 - Virginie Mathivet – R&D Officer