Evaluation metrics for machine learning models

1. Model Evaluation

2. Goals
● Scenario-1: In a course project or competition with given evaluation scheme
and metric, I want to understand the metric to do achieve better result.
● Scenario-2: In a practical project, I need to choose an evaluation scheme and
metric that fit the model and the business need.

3. Popular metrics in Kaggle competitions
Data source: https://www.kaggle.com/kaggle/meta-kaggle

4. Popular metrics in Kaggle competitions
Data source: https://www.kaggle.com/kaggle/meta-kaggle

5. Popular metrics in Kaggle competitions
Data source: https://www.kaggle.com/kaggle/meta-kaggle

6. Regression Metrics
● Root Mean Squared Error
● Mean Absolute Error
● Root Mean Squared Logarithmic Error

7. Root Mean Squared Error v.s. Mean Absolute Error

8. Root Mean Squared Error v.s. Mean Absolute Error

9. Root Mean Squared Error v.s. Mean Absolute Error

10. Root Mean Squared Error v.s. Mean Absolute Error

11. Root Mean Squared Error v.s. Mean Absolute Error

12. Root Mean Squared Error v.s. Mean Absolute Error

13. Root Mean Squared Error v.s. Mean Absolute Error

14. Examples using Root Mean Squared Error
● Relevance score of a product search result chosen from 1, 2, 3 (is RMSE and
MAE the same in this case?)
● A 0-100 rating of a person for a piece of music
● Travel time on highway
● Probabilities of a galaxy belonging to a certain class

15. Examples using Mean Absolute Error
● How much loan in $ defaulted
● Certain part on computers of a certain model, how many repairs are required
each month
● Tens of target variables transformed to be on the same scale, meanings
unknown
● Salary of a job
● Solar radiation
● ...

16. Regression Metrics
● Root Mean Squared Error
● Mean Absolute Error
● Root Mean Squared Logarithmic Error

17. Root Mean Squared Error v.s. Root Mean Squared Logarithmic Error

18. Root Mean Squared Error v.s. Root Mean Squared Logarithmic Error

19. Root Mean Squared Error v.s. Root Mean Squared Logarithmic Error

20. ● Days in hospital a patient will spend in a year
● Number of edits a person will make on Wikipedia in 5 months
● Online sales for a product in a year
● Bulldozer prices
Examples using Root Mean Squared Logarithmic Error

21. ● Days in hospital a patient will spend in a year
● Number of edits a person will make on Wikipedia in 5 months
● Online sales for a product in a year
● Bulldozer prices
Examples using Root Mean Squared Logarithmic Error

22. Regression Metrics - a word on R2

23. Regression Metrics - a word on R2

24. Regression Metrics - a word on R2

25. Classification Metrics
● Categorization Accuracy
● Log Loss
● Area Under Receiver Operating Characteristic Curve (AUC ROC)
● Metrics for multi-label classification

26. Categorization Accuracy
https://www.kaggle.com/kaggle/meta-kaggle

27. Categorization Accuracy

28. Categorization Accuracy

29. Log Loss

30. Log Loss

31. Log Loss

32. Log Loss

33. Log Loss

34. Log Loss severely punish over confidence

35. Log Loss severely punish over confidence

36. Log Loss severely punish over confidence

37. Log Loss severely punish over confidence

38. Log Loss severely punish over confidence

39. Log Loss is a.k.a. cross entropy

41. Examples using Log Loss
● Does an insurance claim need additional information
● Game outcomes in March Madness
● Will a user click an add
● Image classification tasks

42. Multilabel Classification
● Average of the AUCs of each label
● MAP@K
E.g., Which categories does an online comment belong to
(toxic,severe_toxic,obscene,threat,insult,identity_hate)

43. AUC ROC
...

44. AUC ROC

45. AUC ROC

46. Average Precision v.s. Average Precision @K

47. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

48. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

49. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

50. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

51. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

52. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

53. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

54. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

55. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

56. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

57. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

58. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

59. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Average Precision

60. True Targets: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Average Precision

61. AP@K
AP@K does not change increasingly or decreasingly with K.