classifier evaluation of analysis

1. Evaluation and
Credibility
How much should we
believe in what was
learned?

2. 2
Estimating the Predictive Accuracy of a
Classifier( 3 Methods)
• Method 1: Separate Training and Test Sets
If the test set contains N instances of which C are correctly classified the
predictive accuracy of the classifier for the test set is p = C/N

3. The total number of instances correctly classified (in all k runs combined) is
divided by the total number of instances N to give an overall level of predictive
accuracy p,
Method 2: k-fold Cross-validation

4. 4
Method 3: N-fold Cross-validation
• N-fold cross-validation is an extreme case of k-fold cross-
validation, often known as ‘leave-one-out’ cross-validation,
where the dataset is divided into as many parts as there are
instances, each instance effectively forming a test set of one.
• N classifiers are generated, each from N − 1 instances, and
each is used to classify a single test instance. The predictive
accuracy p is the total number correctly classified divided by
the total number of instances.

5. Confusion Matrix

7. ROC Curves

8. ROC (Receiver Operating
Characteristic) curve
• ROC curves were developed in the 1950's as a by-product of research into
making sense of radio signals contaminated by noise. More recently it's
become clear that they are remarkably useful in decision-making.
• They are a performance graphing method.
• True positive and False positive fractions are plotted as we move the dividing
threshold. They look like:

9. True positives and False positives
True positive rate is
TP
= P correctly classified / P
False positive rate is
FP
= N incorrectly classified as P / N

11. ROC Space
• ROC graphs are two-dimensional
graphs in which TP rate is plotted on
the Y axis and FP rate is plotted on the
X axis.
• An ROC graph depicts relative trade-
offs between benefits (true positives)
and costs (false positives).
• Figure shows an ROC graph with five
classifiers labeled A through E.
• A discrete classier is one that outputs
only a class label.
• Each discrete classier produces an (fp
rate, tp rate) pair corresponding to a
single point in ROC space.
• Classifiers in figure are all discrete
classifiers.

12. Several Points in ROC Space
• Lower left point (0, 0) represents the
strategy of never issuing a positive
classification;
– such a classier commits no false positive
errors but also gains no true positives.
• Upper right corner (1, 1) represents the
opposite strategy, of unconditionally
issuing positive classifications.
• Point (0, 1) represents perfect
classification.
– D's performance is perfect as shown.
• Informally, one point in ROC space is
better than another if it is to the
northwest of the first
– tp rate is higher, fp rate is lower, or both.

13. “Conservative” vs. “Liberal”
• Classifiers appearing on the left hand-
side of an ROC graph, near the X axis,
may be thought of as “conservative”
– they make positive classifications
only with strong evidence so they
make few false positive errors,
– but they often have low true positive
rates as well.
• Classifiers on the upper right-hand side
of an ROC graph may be thought of as
“liberal”
– they make positive classifications
with weak evidence so they classify
nearly all positives correctly,
– but they often have high false
positive rates.
• In figure, A is more conservative than B.

14. Random Performance
• The diagonal line y = x represents the strategy
of randomly guessing a class.
• For example, if a classier randomly says
“Positive” half the time (regardless of the
instance provided), it can be expected to get
half the positives and half the negatives correct;
– this yields the point (0.5; 0.5) in ROC space.
• If it randomly say “Positive” 90% of the time
(regardless of the instance provided), it can be
expected to:
– get 90% of the positives correct, but
– its false positive rate will increase to 90% as
well, yielding (0.9; 0.9) in ROC space.
• A random classier will produce a ROC point
that "slides" back and forth on the diagonal
based on the frequency with which it guesses
the positive class.
C's performance is virtually
random.
At (0.7; 0.7), C is guessing
the positive class 70% of the
time.

15. Upper and Lower Triangular Areas
• To get away from the diagonal into the
upper triangular region, the classifier must
exploit some information in the data.
• Any classifier that appears in the lower
right triangle performs worse than random
guessing.
• This triangle is therefore usually empty
in ROC graphs.
• If we negate a classifier that is, reverse its
classification decisions on every instance,
then:
• its true positive classifications become
false negative mistakes, and
• its false positives become true negatives.
• A classifier below the diagonal may be
said to have useful information, but it is
applying the information incorrectly

16. Confusion matrix
• There are two types of errors
• Data mining methods usually minimize FP+FN
Predicted class
Yes No
Actual
class
Yes TP: True
positive
FN: False
negative
No FP: False
positive
TN: True
negative