SlideShare a Scribd company logo

2 Receiver Operating Characteristic (ROC)

•
•
Dmytro Fishman
Dmytro Fishman

The second lecture from the Machine Learning course series of lectures. This lecture discusses ROC metric for evaluating machine learning model's performance. In particular, two ways of building ROC are discussed. A link to my github (https://github.com/skyfallen/MachineLearningPracticals) with practicals that I have designed for this course in both R and Python. I can share keynote files, contact me via e-mail: dmytro.fishman@ut.ee.

Education
1 of 55
Download to read offline
Introduction to Machine Learning (ROC) Dmytro Fishman (dmytro@ut.ee)
ROCK
ROCK
Receiver Operating Characteristic
(1, 0, 1, 0 ,1) True labels
(1, 0, 1, 0 ,1) True labels (0.6,0.2,0.7,0.5,0.4) Classifier predicts
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) There are as many marks on y-axis as there are 1’s in our true labels
(1, 1, 0, 1, 0) True labels There are as many marks on x-axis as there are 0’s in our true labels (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels Go through true labels one by one, if 1 go up, if 0 go right (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) 1 Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is called Receiver Operating Characteristic (ROC) (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is square has sides of length 1 and 1 (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curve (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curveAUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels 1 1 0 1 0 Here is another way to do it (not always you can count labels yourself)AUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/N TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/(FP + TN) TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) TPR = TP/P FPR = FP/(FP + TN) TPR FPR
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR We would like to evaluate different strictness levels of our classifier (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = ? FPR = ?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = 1/3 FPR = 0/(0 + 2)
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 Let’s plot this point on a graph 1/3 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 We shall do this procedure for all possible thresholds 1/3 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = ? FPR = ? How about TPR and FPR?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ?
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ? Oops, this is a false positive!
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 And so on…
(1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
AUC = 0.83 (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 AUC is considered to be more adequate performance measure than accuracy
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 0.5 AUC of 0.5 means random guess
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification
(1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification overfitting 🙄
References • Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine- learning) • Introduction to Machine Learning by Pascal Vincent given at Deep Learning Summer School, Montreal 2015 (http://videolectures.net/ deeplearning2015_vincent_machine_learning/) • Welcome to Machine Learning by Konstantin Tretyakov delivered at AACIMP Summer School 2015 (http://kt.era.ee/lectures/aacimp2015/1-intro.pdf) • Stanford CS class: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy (http://cs231n.github.io/) • Data Mining Course by Jaak Vilo at University of Tartu (https://courses.cs.ut.ee/ MTAT.03.183/2017_spring/uploads/Main/DM_05_Clustering.pdf) • Machine Learning Essential Conepts by Ilya Kuzovkin (https:// www.slideshare.net/iljakuzovkin) • From the brain to deep learning and back by Raul Vicente Zafra and Ilya Kuzovkin (http://www.uttv.ee/naita?id=23585&keel=eng)
www.biit.cs.ut.ee www.ut.ee www.quretec.ee
You, guys, rock!

Recommended

AI and ML Training for non technical
AI and ML Training for non technicalAI and ML Training for non technical
AI and ML Training for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Universitat Politècnica de Catalunya
 
AI and ML for non technical
AI and ML for non technicalAI and ML for non technical
AI and ML for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la modaDeep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la moda
Javier Abadía
 
Deep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial IntelligenceDeep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial Intelligence
Lukas Masuch
 
DOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biologyDOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biology
Dmytro Fishman
 
Tips for effective presentations
Tips for effective presentationsTips for effective presentations
Tips for effective presentations
Dmytro Fishman
 
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WPAutonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Dmytro Fishman
 

Recommended

AI and ML Training for non technical
AI and ML Training for non technicalAI and ML Training for non technical
AI and ML Training for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)Deep Learning for Computer Vision: Image Classification (UPC 2016)
Deep Learning for Computer Vision: Image Classification (UPC 2016)
Universitat Politècnica de Catalunya
 
AI and ML for non technical
AI and ML for non technicalAI and ML for non technical
AI and ML for non technical
Shiv Onkar Deepak Kumar, Ph.D.
 
Deep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la modaDeep learning image classification aplicado al mundo de la moda
Deep learning image classification aplicado al mundo de la moda
Javier Abadía
 
Deep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial IntelligenceDeep Learning - The Past, Present and Future of Artificial Intelligence
Deep Learning - The Past, Present and Future of Artificial Intelligence
Lukas Masuch
 
DOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biologyDOME: Recommendations for supervised machine learning validation in biology
DOME: Recommendations for supervised machine learning validation in biology
Dmytro Fishman
 
Tips for effective presentations
Tips for effective presentationsTips for effective presentations
Tips for effective presentations
Dmytro Fishman
 
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WPAutonomous Driving Lab - Simultaneous Localization and Mapping WP
Autonomous Driving Lab - Simultaneous Localization and Mapping WP
Dmytro Fishman
 
Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
Dmytro Fishman
 
Introduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/BoltIntroduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/Bolt
Dmytro Fishman
 
Introduction to Gaussian Processes
Introduction to Gaussian ProcessesIntroduction to Gaussian Processes
Introduction to Gaussian Processes
Dmytro Fishman
 
Biit group 2018
Biit group 2018Biit group 2018
Biit group 2018
Dmytro Fishman
 
Detecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep LearningDetecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep Learning
Dmytro Fishman
 
Deep Learning in Healthcare
Deep Learning in HealthcareDeep Learning in Healthcare
Deep Learning in Healthcare
Dmytro Fishman
 
5 Introduction to neural networks
5 Introduction to neural networks5 Introduction to neural networks
5 Introduction to neural networks
Dmytro Fishman
 
4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)
Dmytro Fishman
 
3 Unsupervised learning
3 Unsupervised learning3 Unsupervised learning
3 Unsupervised learning
Dmytro Fishman
 
1 Supervised learning
1 Supervised learning1 Supervised learning
1 Supervised learning
Dmytro Fishman
 
What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?
Dmytro Fishman
 
Machine Learning in Bioinformatics
Machine Learning in BioinformaticsMachine Learning in Bioinformatics
Machine Learning in Bioinformatics
Dmytro Fishman
 
Making and Justifying Mathematical Decisions.pdf
Making and Justifying Mathematical Decisions.pdfMaking and Justifying Mathematical Decisions.pdf
Making and Justifying Mathematical Decisions.pdf
Chris Hunter
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introduction
Maksud Ahmed
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
TechSoup
 
ICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptxICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptx
AreebaZafar22
 
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
National Information Standards Organization (NISO)
 
psychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docxpsychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docx
PoojaSen20
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdf
Admir Softic
 
INDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptx
INDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptxINDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptx
INDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptx
RAM LAL ANAND COLLEGE, DELHI UNIVERSITY.
 
Gardella_Mateo_IntellectualProperty.pdf.
Gardella_Mateo_IntellectualProperty.pdf.Gardella_Mateo_IntellectualProperty.pdf.
Gardella_Mateo_IntellectualProperty.pdf.
MateoGardella
 
Unit-V; Pricing (Pharma Marketing Management).pptx
Unit-V; Pricing (Pharma Marketing Management).pptxUnit-V; Pricing (Pharma Marketing Management).pptx
Unit-V; Pricing (Pharma Marketing Management).pptx
VishalSingh1417
 

More Related Content

More from Dmytro Fishman

More from Dmytro Fishman (12)

Introduction to Deep Learning
Introduction to Deep LearningIntroduction to Deep Learning
Introduction to Deep Learning
 
Introduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/BoltIntroduction to Machine Learning for Taxify/Bolt
Introduction to Machine Learning for Taxify/Bolt
 
Introduction to Gaussian Processes
Introduction to Gaussian ProcessesIntroduction to Gaussian Processes
Introduction to Gaussian Processes
 
Biit group 2018
Biit group 2018Biit group 2018
Biit group 2018
 
Detecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep LearningDetecting Nuclei from Microscopy Images with Deep Learning
Detecting Nuclei from Microscopy Images with Deep Learning
 
Deep Learning in Healthcare
Deep Learning in HealthcareDeep Learning in Healthcare
Deep Learning in Healthcare
 
5 Introduction to neural networks
5 Introduction to neural networks5 Introduction to neural networks
5 Introduction to neural networks
 
4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)4 Dimensionality reduction (PCA & t-SNE)
4 Dimensionality reduction (PCA & t-SNE)
 
3 Unsupervised learning
3 Unsupervised learning3 Unsupervised learning
3 Unsupervised learning
 
1 Supervised learning
1 Supervised learning1 Supervised learning
1 Supervised learning
 
What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?What does it mean to be a bioinformatician?
What does it mean to be a bioinformatician?
 
Machine Learning in Bioinformatics
Machine Learning in BioinformaticsMachine Learning in Bioinformatics
Machine Learning in Bioinformatics
 

Recently uploaded

Making and Justifying Mathematical Decisions.pdf
Making and Justifying Mathematical Decisions.pdfMaking and Justifying Mathematical Decisions.pdf
Making and Justifying Mathematical Decisions.pdf
Chris Hunter
 
Gardella_Mateo_IntellectualProperty.pdf.
Gardella_Mateo_IntellectualProperty.pdf.Gardella_Mateo_IntellectualProperty.pdf.
Gardella_Mateo_IntellectualProperty.pdf.
MateoGardella
 
Activity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdfActivity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdf
ciinovamais
 
An Overview of Mutual Funds Bcom Project.pdf
An Overview of Mutual Funds Bcom Project.pdfAn Overview of Mutual Funds Bcom Project.pdf
An Overview of Mutual Funds Bcom Project.pdf
SanaAli374401
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptx
heathfieldcps1
 

Recently uploaded (20)

Making and Justifying Mathematical Decisions.pdf
Making and Justifying Mathematical Decisions.pdfMaking and Justifying Mathematical Decisions.pdf
Making and Justifying Mathematical Decisions.pdf
 
microwave assisted reaction. General introduction
microwave assisted reaction. General introductionmicrowave assisted reaction. General introduction
microwave assisted reaction. General introduction
 
Introduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The BasicsIntroduction to Nonprofit Accounting: The Basics
Introduction to Nonprofit Accounting: The Basics
 
ICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptxICT Role in 21st Century Education & its Challenges.pptx
ICT Role in 21st Century Education & its Challenges.pptx
 
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
Mattingly "AI & Prompt Design: Structured Data, Assistants, & RAG"
 
psychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docxpsychiatric nursing HISTORY COLLECTION .docx
psychiatric nursing HISTORY COLLECTION .docx
 
Key note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdfKey note speaker Neum_Admir Softic_ENG.pdf
Key note speaker Neum_Admir Softic_ENG.pdf
 
INDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptx
INDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptxINDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptx
INDIA QUIZ 2024 RLAC DELHI UNIVERSITY.pptx
 
Gardella_Mateo_IntellectualProperty.pdf.
Gardella_Mateo_IntellectualProperty.pdf.Gardella_Mateo_IntellectualProperty.pdf.
Gardella_Mateo_IntellectualProperty.pdf.
 
Unit-V; Pricing (Pharma Marketing Management).pptx
Unit-V; Pricing (Pharma Marketing Management).pptxUnit-V; Pricing (Pharma Marketing Management).pptx
Unit-V; Pricing (Pharma Marketing Management).pptx
 
Código Creativo y Arte de Software | Unidad 1
Código Creativo y Arte de Software | Unidad 1Código Creativo y Arte de Software | Unidad 1
Código Creativo y Arte de Software | Unidad 1
 
Holdier Curriculum Vitae (April 2024).pdf
Holdier Curriculum Vitae (April 2024).pdfHoldier Curriculum Vitae (April 2024).pdf
Holdier Curriculum Vitae (April 2024).pdf
 
Activity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdfActivity 01 - Artificial Culture (1).pdf
Activity 01 - Artificial Culture (1).pdf
 
Accessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impactAccessible design: Minimum effort, maximum impact
Accessible design: Minimum effort, maximum impact
 
Web & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdfWeb & Social Media Analytics Previous Year Question Paper.pdf
Web & Social Media Analytics Previous Year Question Paper.pdf
 
Nutritional Needs Presentation - HLTH 104
Nutritional Needs Presentation - HLTH 104Nutritional Needs Presentation - HLTH 104
Nutritional Needs Presentation - HLTH 104
 
An Overview of Mutual Funds Bcom Project.pdf
An Overview of Mutual Funds Bcom Project.pdfAn Overview of Mutual Funds Bcom Project.pdf
An Overview of Mutual Funds Bcom Project.pdf
 
Application orientated numerical on hev.ppt
Application orientated numerical on hev.pptApplication orientated numerical on hev.ppt
Application orientated numerical on hev.ppt
 
The basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptxThe basics of sentences session 2pptx copy.pptx
The basics of sentences session 2pptx copy.pptx
 
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptxBasic Civil Engineering first year Notes- Chapter 4 Building.pptx
Basic Civil Engineering first year Notes- Chapter 4 Building.pptx
 

2 Receiver Operating Characteristic (ROC)

  • 5. (1, 0, 1, 0 ,1) True labels
  • 6. (1, 0, 1, 0 ,1) True labels (0.6,0.2,0.7,0.5,0.4) Classifier predicts
  • 7. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2)
  • 8. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) There are as many marks on y-axis as there are 1’s in our true labels
  • 9. (1, 1, 0, 1, 0) True labels There are as many marks on x-axis as there are 0’s in our true labels (0.7,0.6,0.5,0.4,0.2)
  • 10. (1, 1, 0, 1, 0) True labels Go through true labels one by one, if 1 go up, if 0 go right (0.7,0.6,0.5,0.4,0.2)
  • 11. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 12. (1, 1, 0, 1, 0) True labels 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 13. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) 1 Go through true labels one by one, if 1 go up, if 0 go right
  • 14. (1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 15. (1, 1, 0, 1, 0) True labels 1 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 16. (1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 17. (1, 1, 0, 1, 0) True labels 1 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 18. (1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 19. (1, 1, 0, 1, 0) True labels 1 1 0 1 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 20. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 (0.7,0.6,0.5,0.4,0.2) Go through true labels one by one, if 1 go up, if 0 go right
  • 21. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is called Receiver Operating Characteristic (ROC) (0.7,0.6,0.5,0.4,0.2)
  • 22. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 This is square has sides of length 1 and 1 (0.7,0.6,0.5,0.4,0.2)
  • 23. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curve (0.7,0.6,0.5,0.4,0.2)
  • 24. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 We need to find a square of the area under the (ROC) curveAUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
  • 25. (1, 1, 0, 1, 0) True labels 1 1 0 1 0 Here is another way to do it (not always you can count labels yourself)AUC = 0.83 (0.7,0.6,0.5,0.4,0.2)
  • 26. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2)
  • 27. (1, 1, 0, 1, 0) True labelsTPR FPR TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
  • 28. (1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/N TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
  • 29. (1, 1, 0, 1, 0) True labelsTPR FPR FPR = FP/(FP + TN) TPR = TP/P (0.7,0.6,0.5,0.4,0.2)
  • 30. (1, 1, 0, 1, 0) True labels (0.7,0.6,0.5,0.4,0.2) TPR = TP/P FPR = FP/(FP + TN) TPR FPR
  • 31. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR We would like to evaluate different strictness levels of our classifier (0.7,0.6,0.5,0.4,0.2)
  • 32. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
  • 33. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2)
  • 34. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case?
  • 35. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = ? FPR = ?
  • 36. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR What if consider as positive (1) only instances that were predicted positive with >= 0.7 probability? (0.7,0.6,0.5,0.4,0.2) What would TPR and FPR be in this case? >= 0.7 TPR = 1/3 FPR = 0/(0 + 2)
  • 37. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0
  • 38. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 Let’s plot this point on a graph 1/3 0
  • 39. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 We shall do this procedure for all possible thresholds 1/3 0
  • 40. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = ? FPR = ? How about TPR and FPR?
  • 41. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0
  • 42. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ?
  • 43. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = ? FPR = ? Oops, this is a false positive!
  • 44. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
  • 45. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) >= 0.7 TPR = 1/3 FPR = 0 1/3 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 And so on…
  • 46. (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2
  • 47. AUC = 0.83 (1, 1, 0, 1, 0) True labels TPR = TP/P FPR = FP/(FP + TN) TPR FPR (0.7,0.6,0.5,0.4,0.2) 1/3 0 >= 0.4 TPR = 3/3 FPR =1/2 >= 0.2 TPR = 3/3 FPR =2/2 >= 0.7 TPR = 1/3 FPR = 0 >= 0.6 TPR = 2/3 FPR = 0 >= 0.5 TPR = 2/3 FPR =1/2 AUC is considered to be more adequate performance measure than accuracy
  • 48. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 0.5 AUC of 0.5 means random guess
  • 49. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification
  • 50. (1, 1, 0, 1, 0) True labelsTPR FPR (0.7,0.6,0.5,0.4,0.2) 0 AUC is considered to be more adequate performance measure than accuracy AUC = 1 AUC of 0.5 means random guess AUC of 1 means perfect classification overfitting 🙄
  • 51.
  • 52. References • Machine Learning by Andrew Ng (https://www.coursera.org/learn/machine- learning) • Introduction to Machine Learning by Pascal Vincent given at Deep Learning Summer School, Montreal 2015 (http://videolectures.net/ deeplearning2015_vincent_machine_learning/) • Welcome to Machine Learning by Konstantin Tretyakov delivered at AACIMP Summer School 2015 (http://kt.era.ee/lectures/aacimp2015/1-intro.pdf) • Stanford CS class: Convolutional Neural Networks for Visual Recognition by Andrej Karpathy (http://cs231n.github.io/) • Data Mining Course by Jaak Vilo at University of Tartu (https://courses.cs.ut.ee/ MTAT.03.183/2017_spring/uploads/Main/DM_05_Clustering.pdf) • Machine Learning Essential Conepts by Ilya Kuzovkin (https:// www.slideshare.net/iljakuzovkin) • From the brain to deep learning and back by Raul Vicente Zafra and Ilya Kuzovkin (http://www.uttv.ee/naita?id=23585&keel=eng)
  • 54.