This document discusses confusion matrices, which are used to evaluate machine learning algorithms. A confusion matrix shows the predictions made by a model versus the actual classifications. It allows users to see what types of mistakes or "confusions" are made. The document explains 2-class and multi-class confusion matrices through examples. A multi-class matrix for animal classification is shown, demonstrating how values on the diagonal represent correct predictions while off-diagonal values are errors. Code is also presented to calculate precision, recall, and macro-averaged precision from a confusion matrix in Python.

2. 2
Overview
• Machine learning is the kind of programming which gives
computers the capability to automatically learn from data
without being explicitly programmed.
• This means in other words that these programs change
their behavior by learning from data.
• In this course we will cover various aspects of machine
learning
• Everything will be related to Python. So it is Machine
Learning by using Python.
• What is the best programming language for machine
learning?
• Python is clearly one of the top players!

3. 3
Confusion Matrix
• In the previous lesson of our Machine Learning
(Neural Networks with Python and
Numpy and Neural Networks from Scratch ) we
implemented various algorithms, but we didn't
properly measure the quality of the output.
• The main reason was that we used very simple
and small datasets to learn and test. In the
lesson Neural Network: Testing with MNIST, we
will work with large datasets and ten classes, so
we need proper evaluations tools.
• We will introduce the concepts of the confusion
matrix:

4. 4
Confusion Matrix

5. 5
confusion matrix
• A confusion matrix is a matrix (table) that can be
used to measure the performance of an machine
learning algorithm, usually a supervised learning
one.
• Each row of the confusion matrix represents the
instances of an actual class and each column
represents the instances of a predicted class.
• This is the way we keep it like this but it can be
the other way around as well, i.e. rows for
predicted classes and columns for actual classes.
•

6. 6
Confusion Matrix
• The name confusion matrix reflects the fact that it
makes it easy for us to see what kind of
confusions occur in our classification algorithms.
• For example the algorithms should have predicted
a sample as Ci because the actual class is Ci, but
the algorithm came out with Cj. In this case of
mislabelling the element cm[i,j] will be incremented
by one, when the confusion matrix is constructed.
• We will define methods to calculate the confusion
matrix, precision and recall in the following class.

7. 7
2-class Case
• In a 2-class case, i.e. "negative" and
"positive", the confusion matrix may look
like this:

8. 8
2-class Case

9. 9
2-class Case

10. 10
Multi-class Case
• To measure the results of machine learning
algorithms, the previous confusion matrix will not
be sufficient.
• We will need a generalization for the multi-class
case.
• Let us assume that we have a sample of 25
animals, e.g. 7 cats, 8 dogs, and 10 snakes, most
probably Python snakes.
• The confusion matrix of our recognition algorithm
may look like the following table:

11. 11
Multi-class Case
• confusion matrix

12. 12
Multi-class Case
• In this confusion matrix, the system correctly predicted
six of the eight actual dogs, but in two cases it took a
dog for a cat.
• The seven actual cats were correctly recognized in six
cases but in one case a cat was taken to be a dog.
Usually, it is hard to take a snake for a dog or a cat, but
this is what happened to our classifier in two cases.
• Yet, eight out of ten snakes had been correctly
recognized. (Most probably this machine learning
algorithm was not written in a Python program, because
Python should properly recognize its own species :-) )

13. 13
Multi-class Case
• You can see that all correct predictions are located
in the diagonal of the table, so prediction errors
can be easily found in the table, as they will be
represented by values outside the diagonal.
• We can generalize this to the multi-class case.
• To do this we summarize over the rows and
columns of the confusion matrix.
• Given that the matrix is oriented as above, i.e., that
a given row of the matrix corresponds to specific
value for the "truth", we have:

14. 14

15. 15
Example
• We are ready now to code this into Python.
• The following code shows a confusion
matrix for a multi-class machine learning
problem with ten labels, so for example an
algorithms for recognizing the ten digits
from handwritten characters.
• Familiarity with Numpy and Numpy arrays,
are recommend .

16. 16
Numpy arrays

17. 17
• The functions 'precision' and 'recall'
calculate values for a label, whereas the
function 'precision_macro_average' the
precision for the whole classification
problem calculates.

18. 18

19. 19

20. 20
Machine Learning by
using Python
Confusion Matrix
Lesson 3
By: Professor Lili Saghafi