Understand Precision and Recall with Spam Detection Model Evaluation
•
1 like•3,237 views
Introduction to ML for analysts and product managers Unit 3: model evaluation
Recommended
Recommended
More Related Content
More from Alexey Grigorev
More from Alexey Grigorev (20)
Recently uploaded
Recently uploaded (20)
Understand Precision and Recall with Spam Detection Model Evaluation
- 1. Introduction to Machine Learning for Analysts and Product Managers Machine Learning Bookcamp Alexey Grigorev Unit 3: Model Evaluation
- 3. Plan ● Example: spam detection ● Accuracy ● Precision ● Recall
- 10. 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ Validation 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉✉ ✉✉✉✉✉✉✉ Pretend we don’t know the truth
- 11. Validation ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉✉ ✉✉✉✉✉✉✉ Pretend we don’t know the truth
- 14. ❌✔✔✔✔✔✔ ✔✔✔✔✔❌✔
- 15. 14 in total ❌ 2 incorrect ✔ 12 correct ❌✔✔✔✔✔✔ ✔✔✔✔✔❌✔
- 16. 14 in total ❌ 2 incorrect ✔ 12 correct 86% accuracy
- 17. 14 in total ❌ 2 incorrect ✔ 12 correct 86% accuracy is it good?
- 18. 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ Validation ✉✉✉✉✉✉✉ ✉✉✉✉✉✉✉ Actual Prediction (dummy model) Always predict “not spam”
- 20. 14 in total ❌ 3 incorrect ✔ 11 correct 80% accuracy (only 6% worse!)
- 21. Accuracy ● Easy to understand ● Misleading in cases of imbalance (our case: only 20% are spam) ● 86% seems quite good — until compared to baseline with 80% Is there anything else?
- 22. Precision Among predicted as spam, how many messages are indeed spam
- 23. Precision Among predicted as spam, how many messages are indeed spam 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction
- 24. Precision Among predicted as spam, how many messages are indeed spam 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction
- 25. Precision Among predicted as spam, how many messages are indeed spam 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction
- 26. Precision Among predicted as spam, how many messages are indeed spam 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction
- 27. Precision Among predicted as spam, how many messages are indeed spam 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction ✔ ✔ ❌
- 28. Precision Among predicted as spam, how many messages are indeed spam ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉✔ ✔ ❌ Precision = 2 / 3 = 66%
- 29. Recall Among all the spam messages, how many are classified correctly
- 30. Recall Among all the spam messages, how many are classified correctly 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction
- 31. Recall Among all the spam messages, how many are classified correctly 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction
- 32. Recall Among all the spam messages, how many are classified correctly 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction
- 33. Recall Among all the spam messages, how many are classified correctly 💌✉✉✉✉✉💌 ✉💌✉✉✉✉✉ ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉ Predict Actual Prediction ✔ ✔❌
- 34. Recall Among all the spam messages, how many are classified correctly ✉✉✉✉✉✉💌 ✉💌✉✉✉💌✉✔ ✔❌ Recall = 2 / 3 = 66%
- 35. Precision vs Recall Precision: Fraction of correctly classified spam messages in spam. We check only the spam folder.
- 36. Precision vs Recall Recall: Fraction of all spam messages that ended up in spam. We check both inbox and spam. Precision: Fraction of correctly classified spam messages in spam. We check only the spam folder.
- 37. Baseline 3 2 R = 3 2 P = Model Dummy baseline (always predict not spam) 3 0 R = 0 0 P =
- 38. Baseline Dummy baseline (always predict not spam) Home task: Check these numbers! Do it for two baselines: ● Always predict “spam” ● Always predict “no spam” 3 0 R = 0 0 P =
- 39. Summary ● Accuracy may be misleading in cases of class imbalance ● Precision: number of actual spam messages in the spam folder ● Recall: number of spam messages that correctly ended up in the spam folder
- 40. mlbookcamp.com ● Learn Machine Learning by doing projects ● http://bit.ly/mlbookcamp ● Get 40% off with code “grigorevpc” Machine Learning Bookcamp
- 41. Course page: mlbookcamp.com/course/ml-pm @Al_Grigoragrigorev alexeygrigorev.com Icons from flaticon by Pixel perfect, Roundicons, turkkub