The document provides an overview of machine learning, including definitions, types of machine learning (supervised, unsupervised, reinforcement learning), and evaluation metrics for machine learning models. It discusses classification metrics like accuracy, precision, recall, F1 score, and confusion matrices. For regression problems, it covers metrics like mean absolute error, mean squared error, R2 score. It also provides examples of calculating many of these common metrics in Python.

1. www.SunilOS.com 1
Machine Learning
www.sunilos.com
www.raystec.com

2. www.SunilOS.com 2
What is Machine Learning?
❑ “Learning is any process by which a system improves
performance from experience.”(Herbert Simon)

3. Concept in machine learning
❑Definition by Tom Mitchell (1998):
❑Machine Learning is the study of algorithms that
❑• improve their performance P.
❑• at some task T
❑• with experience E.
❑A well-defined learning task
❑ is given by <P,T,E>
❑
www.SunilOS.com 3

4. Traditional programming v/s machine learning
www.SunilOS.com 4

5. Why Machine Learning?
❑Human expertise does not exist (Mission
Mangal).
www.SunilOS.com 5

6. Robotics
❑Human are not able to perform task with great
expertise.
www.SunilOS.com 6

7. Personalized Recommendation
❑Models must be customized (Netflix, Amazon)
www.SunilOS.com 7

8. Big Data
❑Models are based on huge amounts of data.
www.SunilOS.com 8

9. Why Now?
❑Flood of available data(especially with the use of
Internet)
❑Increasing Computational power.
❑Growing progress in available algorithms and
theory developed by researchers
❑Increasing support in Industries
www.SunilOS.com 9

10. Types of Machine Learning
www.SunilOS.com 10

11. Types of Learning
❑Supervised (inductive) learning:
o Given: training data + desired outputs (labels).
❑Unsupervised learning
o Given: training data (without desired outputs)
❑Reinforcement learning
o Rewards from sequence of actions
www.SunilOS.com 11

12. Supervised Learning
www.SunilOS.com 12
WEIGHT =
FEATURE
CURRENCY=
LABEL
03 Grams = 01 Rupee
Coin

13. Supervised Learning
www.SunilOS.com 13
Labeled Data

14. No Labeled
Data
Unsupervised Learning

15. Reinforcement Learning
www.SunilOS.com 15

16. Machine Learning Model

17. Applications of Machine Learning Model

18. Framing Learning Problem
www.SunilOS.com 18
❑Choose the data.
❑Choose exactly what is to be learned.
o i.e. the target function
❑Choose the model
❑Train the model
❑Evaluate the model

19. Machine Learning Process
www.SunilOS.com 19

20. Evaluation Criteria
❑Accuracy
❑Precision and recall
❑Squared error
❑Likelihood
❑Posterior probability
❑Cost/Utility
❑Margin
❑Entropy
❑etc.
www.SunilOS.com 20

21. Performance Measure of Model
❑The metrics that you choose to evaluate your machine
learning algorithms are very important.
❑Choice of metrics influences how the performance of
machine learning algorithms is measured and
compared.
o Classification Metrics
o Regression Metrics
o Multilabel ranking metrics
o Clustering metrics
www.SunilOS.com 21

22. Classification Metrics
❑Classification accuracy is the number of correct
predictions made as a ratio of all predictions made.
o Correct Prediction/Total Prediction
❑This is the most common evaluation metric for
classification problems.
❑It is really only suitable when
o There are an equal number of observations in each class
(which is rarely the case) and
o That all predictions and prediction errors are equally
important, which is often not the case.
www.SunilOS.com 22

23. Classification Accuracy
❑ import numpy as np
❑ from sklearn.metrics import accuracy_score
❑ y_pred = [0, 2, 1, 3]
❑ y_true = [0, 1, 2, 3]
❑ print("Acurracy",accuracy_score(y_true, y_p
red))
❑ print("Accuracy:",accuracy_score(y_true, y_
pred, normalize=False))
❑ #Output
o Acurracy 0.5
o Accuracy: 2
www.SunilOS.com 23

24. Cohen’s kappa
❑The kappa score is a number between -1 and 1. Scores
above .8 are generally considered good; zero or lower
means not good (practically random labels).
❑Kappa scores can be computed for binary or multiclass
problems, but not for multilabel problems
www.SunilOS.com 24

25. Cohen’s kappa (cont.)
❑ from sklearn.metrics import cohen_kappa_sco
re
❑ y_true = [2, 0, 2, 2, 0, 1]
❑ y_pred = [0, 0, 2, 2, 0, 2]
❑ print("Kappa Score=",cohen_kappa_score(y_tr
ue, y_pred))
❑ #Output
o 0.4285714285714286
www.SunilOS.com 25

26. Precision and Recall
❑ Precision and Recall both are the important to evaluate the
model. It is used for binary class labels.
❑ Precision can be said as a positive predictive value. The
precision is the ratio tp / (tp + fp) where
o tp is the number of true positives
o fp the number of false positives.
❑ The precision is the ability of the classifier not to label as
positive a sample that is negative.
❑ Recall is the ratio tp / (tp + fn)
o where tp is the number of true positives
o fn the number of false negatives.
o The recall is the ability of the classifier to find all the positive
samples.
www.SunilOS.com 26

27. Precision and Recall (cont.)
www.SunilOS.com 27

28. Precision and Recall (cont.)
❑ from sklearn import metrics
❑ y_pred = [0, 1, 0, 0]
❑ y_true = [0, 1, 0, 1]
❑ print("Precision:",metrics.precision_score(y_tr
ue, y_pred))
❑ # 1.0
❑ print("Recall:,metrics.recall_score(y_true, y_p
red))
❑ # 0.5
www.SunilOS.com 28

29. Confusion Matrix
❑ The confusion matrix is a presentation of the accuracy of a
model with two or more classes.
❑ The table presents predictions on the x-axis and accuracy
outcomes on the y-axis.
❑ The cells of the table are the number of predictions made by a
machine learning algorithm.
❑ For example, a machine learning algorithm can predict 0 or 1 and
each prediction may actually have been a 0 or 1.
o Predictions for 0 that were actually 0 , appear in the cell for prediction=0
and actual=0,
o whereas predictions for 0 that were actually 1, appear in the cell for
prediction = 0 and actual=1. and so on
www.SunilOS.com 29

30. Confusion Matrix (cont.)
❑ The parameter normalize allows to report ratios instead of counts. The
confusion matrix can be normalized in 3 different ways: 'pred', 'true',
and 'all' which will divide the counts by the sum of each columns,
rows, or the entire matrix, respectively.
❑ from sklearn.metrics import confusion_matrix
❑ y_true = [2, 0, 2, 2, 0, 1]
❑ y_pred = [0, 0, 2, 2, 0, 2]
❑ print(confusion_matrix(y_true, y_pred))
❑ #Output
❑ [[2 1]
❑ [2 3]]
www.SunilOS.com 30

31. Confusion Matrix with normalize parameter
❑ from sklearn.metrics import confusion_matrix
❑ y_true = [0, 0, 0, 1, 1, 1, 1, 1]
❑ y_pred = [0, 1, 0, 1, 0, 1, 0, 1]
❑ # print(confusion_matrix(y_true, y_pred, normal
ize="true"))
❑ print(confusion_matrix(y_true, y_pred, normaliz
e="all"))
❑ # print(confusion_matrix(y_true, y_pred, normal
ize="pred"))
❑ #Output
❑ [[0.25 0.125]
❑ [0.25 0.375]]
www.SunilOS.com 31

32. Get count of Values
❑For binary problems, we can get counts of true
negatives, false positives, false negatives and
true positives as follows:
o tn, fn, fp, tp=confusion_matrix(y_true, y
_pred).ravel())
www.SunilOS.com 32

33. Classification report
❑ The classification_report function builds a text report showing
the main classification metrics. Here is a small example with
custom target_names and inferred labels:
o from sklearn.metrics import classification_re
port
o y_true = [0, 1, 2, 2, 0]
o y_pred = [0, 0, 2, 1, 0]
o target_names = ['class 0', 'class 1', 'class
2']
o print(classification_report(y_true, y_pred, t
arget_names=target_names))
www.SunilOS.com 33

34. Classification Report Output
❑ precision recall f1-score support
❑ class 0 0.67 1.00 0.80 2
❑ class 1 0.00 0.00 0.00 1
❑ class 2 1.00 0.50 0.67 2
❑ accuracy 0.60 5
❑ macro avg 0.56 0.50 0.49 5
❑ weighted avg 0.67 0.60 0.59 5
www.SunilOS.com 34

35. Hamming Loss
❑ The hamming loss computes the average Hamming loss
or Hamming distance between two sets of samples.
❑ It is Used for multi class classification.
❑ Hamming Loss calculates loss generated in the bit string of class
labels during prediction.
❑ It does that by XOR between the actual and predicted labels and
then average across the dataset.
o from sklearn.metrics import hamming_loss
o y_pred = [1, 2, 3, 4,6,7,8]
o y_true = [2, 2, 3, 4,5,6,7]
o print(hamming_loss(y_true, y_pred))
www.SunilOS.com 35

36. F-measures
❑The F-Measure (Fβ and F1 measures) can be interpreted
as a weighted harmonic mean of the precision and
recall. A Fβ measure reaches its best value at 1 and its
worst score at 0.
o from sklearn import metrics
o y_pred = [0, 1, 0, 0]
o y_true = [0, 1, 0, 1]
o metrics.f1_score(y_true, y_pred)
www.SunilOS.com 36

37. Receiver operating characteristic
❑ The function roc_curve computes the receiver operating
characteristic curve, or ROC curve
❑ ROC curve, is a graphical plot which illustrates the performance
of a binary classifier system as its discrimination threshold is
varied.
❑ It is created by plotting the fraction of true positives out of the
positives (TPR = true positive rate) vs. the fraction of false
positives out of the negatives (FPR = false positive rate), at
various threshold settings.
❑ TPR is also known as sensitivity, and FPR is one minus the
specificity or true negative rate.
www.SunilOS.com 37

38. Area under ROC curve
❑ AUC ranges in value from 0 to 1. A model whose predictions are
100% wrong has an AUC of 0.0; one whose predictions are
100% correct has an AUC of 1.0.
❑ import numpy as np
❑ from sklearn.metrics import roc_auc_score
❑ y_true = np.array([0, 0, 1, 1])
❑ y_scores = np.array([0.1, 0.4, 0.35, 0.8])
❑ roc_auc_score(y_true, y_scores)
www.SunilOS.com 38

39. ROC curve
www.SunilOS.com 39

40. Regression Metrics
❑ We will review 3 of the most common metrics for evaluating
predictions on regression machine learning problems:
❑ Mean Absolute Error.
o The Mean Absolute Error (or MAE) is the average of the
absolute differences between predictions and actual values. It
gives an idea of how wrong the predictions were.
❑ Mean Squared Error.
o Same as the mean absolute error.
o Taking the square root of the mean squared error converts the
units back to the original units of the output variable and can
be meaningful for description and presentation. This is called
the Root Mean Squared Error (or RMSE).
www.SunilOS.com 40

41. Regression Metrics
❑ R^2
o The R^2 (or R Squared) metric provides an indication of the goodness of
fit of a set of predictions to the actual values.
o In statistical literature, this measure is called the coefficient of
determination.
o This is a value between 0 and 1 for no-fit and perfect fit respectively.
❑These functions have an multioutput keyword argument
which specifies the way the scores or losses for each
individual target should be averaged. Values for this
keyword
o uniform_average
o raw_values
www.SunilOS.com 41

42. Mean absolute error
❑
www.SunilOS.com 42

43. Mean absolute error (cont.)
❑ from sklearn.metrics import mean_absolute_error
❑ y_true = [3, -0.5, 2, 7]
❑ y_pred = [2.5, 0.0, 2, 8]
❑ mean_absolute_error(y_true, y_pred)
❑ y_true = [[0.5, 1], [-1, 1], [7, -6]]
❑ y_pred = [[0, 2], [-1, 2], [8, -5]]
❑ mean_absolute_error(y_true, y_pred))
❑ mean_absolute_error(y_true, y_pred, multioutput='raw
_values')
❑ mean_absolute_error(y_true, y_pred, multioutput=[0.3
, 0.7])
www.SunilOS.com 43

44. Mean squared error
❑
www.SunilOS.com 44

45. Mean squared error
❑ from sklearn.metrics import mean_squared_error
❑ y_true = [3, -0.5, 2, 7]
❑ y_pred = [2.5, 0.0, 2, 8]
❑ mean_squared_error(y_true, y_pred))
❑ y_true = [[0.5, 1], [-1, 1], [7, -6]]
❑ y_pred = [[0, 2], [-1, 2], [8, -5]]
❑ mean_squared_error(y_true, y_pred))
❑ Output:
❑ MSE: 0.375
❑ MSE: 0.7083333333333334
www.SunilOS.com 45

46. Mean squared logarithmic error
www.SunilOS.com 46

47. ❑ from sklearn.metrics import mean_squared_log_error
❑ y_true = [3, 5, 2.5, 7]
❑ y_pred = [2.5, 5, 4, 8]
❑ mean_squared_log_error(y_true, y_pred)
❑ y_true = [[0.5, 1], [1, 2], [7, 6]]
❑ y_pred = [[0.5, 2], [1, 2.5], [8, 8]]
❑ mean_squared_log_error(y_true, y_pred)
❑ Output:
❑ MSLE: 0.03973012298459379
❑ MSLE: 0.044199361889160536
www.SunilOS.com 47

48. Median absolute error
❑ The median_absolute_error is particularly interesting because it
is robust to outliers.
❑ The loss is calculated by taking the median of all absolute
differences between the target and the prediction.
❑ If yj is the predicted value of the i-th sample and yi is the
corresponding true value, then the median absolute error
(MedAE) estimated over nsamples is defined as
❑ It does not support multioutput.
o MeadAE(yi,yj)=median(|yi-yj|,……,|yi-yj|)
www.SunilOS.com 48

49. ❑from sklearn.metrics import
❑median_absolute_error
❑y_true = [3, -0.5, 2,7]
❑y_pred = [2.5, 0.0, 2, 8]
❑median_absolute_error(y_true, y_pred)
www.SunilOS.com 49

50. R² score, the coefficient of determination
❑The r2 function computes the coefficient of
determination, usually denoted as R².
❑It represents the proportion of variance (of y) that has
been explained by the independent variables in the
model.
❑ It provides an indication of goodness of fit and
therefore a measure of how well unseen samples are
likely to be predicted by the model, through the
proportion of explained variance.
www.SunilOS.com 50

51. ❑As such variance is dataset dependent, R² may not be
meaningfully comparable across different datasets.
❑Best possible score is 1.0 and it can be negative
(because the model can be arbitrarily worse).
❑A constant model that always predicts the expected
value of y, disregarding the input features, would get a
R² score of 0.0.
❑If yj is the predicted value of the i-th sample and yi is
the corresponding true value for total n samples, the
estimated R² is defined as:
www.SunilOS.com 51

52. ❑ from sklearn.metrics import r2_score
❑ y_true = [3, -0.5, 2, 7]
❑ y_pred = [2.5, 0.0, 2, 8]
❑ r2_score(y_true, y_pred)
❑ y_true = [[0.5, 1], [-1, 1], [7, -6]]
❑ y_pred = [[0, 2], [-1, 2], [8, -5]] r2_score(y_true,
y_pred, multioutput='variance_weighted')
❑ y_true = [[0.5, 1], [-1, 1], [7, -6]]
❑ y_pred = [[0, 2], [-1, 2], [8, -5]]
❑ r2_score(y_true, y_pred,
multioutput='uniform_average')
❑ r2_score(y_true, y_pred, multioutput='raw_values')
❑ r2_score(y_true, y_pred, multioutput=[0.3, 0.7])
www.SunilOS.com 52

53. Clustering Metrics
❑Adjusted Rand index
❑Mutual Information based scores
❑Homogeneity, completeness and V-measure
❑Fowlkes-Mallows scores
❑Silhouette Coefficient
❑Contingency Matrix
www.SunilOS.com 53

54. Machine Learning In Brief
❑Tens of thousands of machine learning algorithms
❑Every ML algorithm has three components:
o Representation
o Optimization
o Evaluation
www.SunilOS.com 54

55. Tools For Machine Learning
❑ Machine Learning Libraries:
o Scikit Learn: Built on Numpy, Scipy and matplotlib. Used for Supervised
Learning and Unsupervised Learning.
o Keras: Built on Tensor flow
o Tensorflow: For NN and Deep Learning
❑ Programming:
o Python
❑ Machine Learning Tools
o Anaconda: Free distribution of Python, Package of scientific computing
o Jupyter: web Interface for Python Programming
❑ Data Visualization
o Matplotlib
o Numpy
o Pandas
www.SunilOS.com 55

56. What we will cover in this Course
❑Supervised Learning
❑Unsupervised Learning
❑Reinforcement Learning
❑Performance Measure
❑Applications of Machine Learning
❑Data Preprocessing
❑Data visualization
www.SunilOS.com 56

57. Disclaimer
❑This is an educational presentation to enhance the
skill of computer science students.
❑This presentation is available for free to computer
science students.
❑Some internet images from different URLs are used
in this presentation to simplify technical examples
and correlate examples with the real world.
❑We are grateful to owners of these URLs and
pictures.
www.SunilOS.com 57

58. Thank You!
www.SunilOS.com 58
www.SunilOS.com