This document discusses best practices for evaluating supervised machine learning models, including: 1) The importance of splitting data into training and testing sets to avoid "memorizing" the data and get an accurate performance measure. 2) Common dataset splitting methods like linear and random splits. 3) The importance of metrics like accuracy, and how they can be misleading, especially for imbalanced datasets. 4) How different domains may value reducing different types of mistakes, like preferring fewer false negatives for medical diagnosis.

1. BigML Education
Evaluations
August 2017

2. BigML Education Program 2Evaluations
In This Video
• Introduction to and justification of model evaluation
• Discussion of common missteps when evaluating a
supervised model
• Creation of a dataset split for training and evaluation of
a supervised model using BigML
• Creation and interpretation of an evaluation in the
BigML user interface

3. BigML Education Program 3Evaluations
Evaluation Workflow
TRAINING
DATASET
EVALUATION
TESTING
DATASET
+
ANALYZE PREDICTIONS
SUPERVISED
MODEL

4. BigML Education Program 4Evaluations
Memorizing Data
TRAINED MODEL
TRAINING
DATASET
TESTING
DATASET
EVALUATE
TRAINING
DATASET
EVALUATE
Excellent performance
Model already
“knows” the data
Poor performance
if model can’t
generalize

5. BigML Education Program 5Evaluations
Linear Dataset Split
Patient No.
Plasma
Glucose
BMI Pregnancies Diabetes?
1 80 28,5 4 No
2 111 20,7 2 No
3 156 41,0 0 Yes
4 131 35,1 2 Yes
5 70 32,2 1 No
6 86 21,8 4 No
7 42 18,6 0 No
8 150 33,1 6 Yes
9 92 25,9 3 No
10 145 30,7 5 Yes
Test Instances
Training Instances

6. BigML Education Program 6Evaluations
Random Dataset Split
Plasma Glucose BMI Pregnancies Diabetes?
80 28,5 4 No
111 20,7 2 No
156 41,0 0 Yes
131 35,1 2 Yes
70 32,2 1 No
86 21,8 4 No
42 18,6 0 No
150 33,1 6 Yes
92 25,9 3 No
145 30,7 5 Yes
Test Instances
Remainder used for
training

7. BigML Education Program 7Evaluations
Accuracy
Positive Example
Negative Example
Classified Positive Classified Negative
A trivial classifier (always predict negative)
achieves 95% accuracy on unbalanced data!

8. BigML Education Program 8Evaluations
Accuracy
Positive Example
Negative Example
Classified Positive Classified Negative
On a balanced dataset,
95% accuracy indicates a competent classifier

9. BigML Education Program 9Evaluations
Mistake Costs
• Which is worse in your domain, a false positive or a
false negative?
• Medical diagnosis
• Cost of a false postive: The patient has to
undergo more testing to discover they do not
have the disease (low cost?)
• Cost of a false negative: The patient is declared
healthy and the undetected disease progresses
(high cost?)
• Solution: Select a threshold for positive classification
that makes the appropriate trade-off between mistakes

10. BigML Education Program 10Evaluations
Review
• Evaluations, when done correctly allow you to assess
the performance of your learned supervised model
• You should never evaluate a model on any of the data
used to train that model
• BigML allows 1-click splitting of your dataset for proper
training and testing
• Depending on your domain, metrics other than
accuracy may be required to properly understand your
model’s performance
• The evaluation resource view can be used to select an
operating point for the model that makes the correct
trade-off between false positives and false negatives for
your domain