The document discusses the development and evaluation of algorithms for detecting faults and teaching virtual agents, using a series of experiments to highlight accuracy and performance metrics. Initial results showed a high accuracy of 99.1% in classifying images, with further improvements through data augmentation. It also addresses the challenges of overfitting in real robot applications and emphasizes the importance of testing on real robots early in the process.

1. Be the smartest
Virginie Mathivet
R&D Officer
Ryu(r) Algorithm
https://danbooru.donmai.us/posts/948811
http://vsbattles.wikia.com/wiki/File:RyuRender.png

2. dotAI’18 - Virginie Mathivet – R&D Officer
The accuracy paradox

3. dotAI’18 - Virginie Mathivet – R&D Officer
The accuracy paradox

4. dotAI’18 - Virginie Mathivet – R&D Officer
Fault detection
Infrared HQ images Bubble Scratch
Micro defectBorder defect

5. dotAI’18 - Virginie Mathivet – R&D Officer
First results
• Accuracy: 99.1%!!! (in 2 epochs)

6. dotAI’18 - Virginie Mathivet – R&D Officer
First results
• Accuracy: 99.1%!!! (in 2 epochs)
Real Predicted OK (1665) KO (15) Accuracy
OK 1665 0 100%
KO 15 0 0%
Total 1680 0 99.1%
• Confusion matrix:

7. dotAI’18 - Virginie Mathivet – R&D Officer
The accuracy paradox and imbalanced classes
0
10
20
30
40
50
60
70
80
90
100
Classes
Real
Pred.
1 2 3 Accuracy
1 90 0 0 100%
2 5 0 0 0%
3 5 0 0 0%
TOTAL 100 0 0 90%

8. dotAI’18 - Virginie Mathivet – R&D Officer
What to do?

9. dotAI’18 - Virginie Mathivet – R&D Officer
Data augmentation
…
New dataset:
• 1665 images “OK” (94.9%)
• 90 images “KO” (5.1%)

10. dotAI’18 - Virginie Mathivet – R&D Officer
Data augmentation
…
New dataset:
• 1665 images “OK” (94.9%)
• 90 images “KO” (5.1%)
Accuracy: 99.15%

11. dotAI’18 - Virginie Mathivet – R&D Officer
Data augmentation
Real Predicted OK (1665) KO (90) Accuracy
OK 1665 0 100%
KO 15 75 83.3%
Total 1680 75 99.15%
…
New dataset:
• 1665 images “OK” (94.9%)
• 90 images “KO” (5.1%)
Accuracy: 99.15%

12. dotAI’18 - Virginie Mathivet – R&D Officer
Data augmentation
Real Predicted OK (1665) KO (90) Accuracy
OK 1665 0 100%
KO 15 75 83.3%
Total 1680 75 99.15%
…
New dataset:
• 1665 images “OK” (94.9%)
• 90 images “KO” (5.1%)
Accuracy: 99.15%

13. dotAI’18 - Virginie Mathivet – R&D Officer
The evaluation function definition

14. dotAI’18 - Virginie Mathivet – R&D Officer
The evaluation function definition

15. dotAI’18 - Virginie Mathivet – R&D Officer
The choice of the evaluation function
Classification
Real Predicted 1 0
1 A (true positive) C (false negative)
0 B (false positive) D (true negative)
Accuracy Precision Recall / Sensitivity F1-score
(A+D)/(A+B+C+D) A/(A+B) A/(A+C) 2*precision*recall/(precision+recall)
Specificity Detection rate Youden’s J Index Balanced accuracy
D/(B+D) A/(A+B+C+D) Sensitivity+specificity - 1 (sensitivity+specificity) / 2

16. dotAI’18 - Virginie Mathivet – R&D Officer
The choice of the evaluation function
Regression
• R-squared (R2)
• Root Mean Squared Error (RMSE)
• Residual Standard Error (RSE) or
Model Sigma
• Mean Absolute Error (MAE)
• Akaike’s Information Criteria (AIC)
• Bayesian Information Criteria (BIC)
• …

17. dotAI’18 - Virginie Mathivet – R&D Officer
The choice of the evaluation function
0
5
10
15
20
25
0 1 2 3 4 5 6 7 8 9 10
Behavioral task

18. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 1st try: distance made in 10sec

19. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 1st try: distance made in 10sec
https://gymnasticselectronicportfolio13.weebly.com/gymnastics-skills.html

20. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 1st try: distance made in 10sec
https://gymnasticselectronicportfolio13.weebly.com/gymnastics-skills.html
https://viatcheslav.livejournal.com/231564.html

21. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 2nd try: distance made in 10sec without rolling

22. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 2nd try: distance made in 10sec without rolling
https://blog.myagilepartner.fr/wp-content/uploads/2017/08/scared_bug.jpg

23. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 2nd try: distance made in 10sec without rolling
https://www.memecenter.com/fun/1717889/flying-spiderhttps://blog.myagilepartner.fr/wp-content/uploads/2017/08/scared_bug.jpg

24. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 3rd try: idem but after the bug correction

25. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 3rd try: idem but after the bug correction
https://www.nemusloren.it/2013/11/30/come-se-fosse-possibile/

26. dotAI’18 - Virginie Mathivet – R&D Officer
Teach a (virtual) spider how to walk
• 3rd try: idem but after the bug correction
https://www.nemusloren.it/2013/11/30/come-se-fosse-possibile/

27. dotAI’18 - Virginie Mathivet – R&D Officer
Rule of thumb: speak to a child
Eat with a fork:

28. dotAI’18 - Virginie Mathivet – R&D Officer
Rule of thumb: speak to a child
Eat with a fork:

29. dotAI’18 - Virginie Mathivet – R&D Officer
Rule of thumb: speak to a child
Eat with a fork:

30. dotAI’18 - Virginie Mathivet – R&D Officer
Rule of thumb: speak to a child
Eat with a fork:

31. dotAI’18 - Virginie Mathivet – R&D Officer
The overfitting spotting

32. dotAI’18 - Virginie Mathivet – R&D Officer
The overfitting spotting

33. dotAI’18 - Virginie Mathivet – R&D Officer
Overfitting spotting
Error
Time
Training
Validation
Underfitting Overfitting
Optimum

34. dotAI’18 - Virginie Mathivet – R&D Officer
Teaching a robot how to avoid walls
• Evaluation function :
• 1st try: time until a collision with a wall (max 10sec)

35. dotAI’18 - Virginie Mathivet – R&D Officer
Teaching a robot how to avoid walls
• Evaluation function :
• 1st try: time until a collision with a wall (max 10sec)

36. dotAI’18 - Virginie Mathivet – R&D Officer
Teaching a robot how to avoid walls
• Evaluation function :
• 1st try: time until a collision with a wall (max 10sec)
• 2nd try: time moving until a collision with a wall (max 10sec)

37. dotAI’18 - Virginie Mathivet – R&D Officer
Teaching a robot how to avoid walls
• Evaluation function :
• 1st try: time until a collision with a wall (max 10sec)
• 2nd try: time moving until a collision with a wall (max 10sec)

38. dotAI’18 - Virginie Mathivet – R&D Officer
Teaching a robot how to avoid walls
• Evaluation function :
• 1st try: time until a collision with a wall (max 10sec)
• 2nd try: time moving until a collision with a wall (max 10sec)
• 3rd try: time moving forward until a collision with a wall (max 10sec)

39. dotAI’18 - Virginie Mathivet – R&D Officer
Teaching a robot how to avoid walls
• Evaluation function :
• 1st try: time until a collision with a wall (max 10sec)
• 2nd try: time moving until a collision with a wall (max 10sec)
• 3rd try: time moving forward until a collision with a wall (max 10sec)

40. dotAI’18 - Virginie Mathivet – R&D Officer
Theory vs Reality
• Real robot and real cage => not working at all!!
• Unseen overfitting!! (no dataset = no curves)

41. dotAI’18 - Virginie Mathivet – R&D Officer
Theory vs Reality
• Real robot and real cage => not working at all!!
• Unseen overfitting!! (no dataset = no curves)

42. dotAI’18 - Virginie Mathivet – R&D Officer
Theory vs Reality
• Real robot and real cage => not working at all!!
• Unseen overfitting!! (no dataset = no curves)
• Solution :
• Randomness
• Test on real robots as soon as possible

43. dotAI’18 - Virginie Mathivet – R&D Officer