- 2. www.SunilOS.com 2 What is Machine Learning? ❑ “Learning is any process by which a system improves performance from experience.”(Herbert Simon)
- 3. Concept in machine learning ❑Definition by Tom Mitchell (1998): ❑Machine Learning is the study of algorithms that ❑• improve their performance P. ❑• at some task T ❑• with experience E. ❑A well-defined learning task ❑ is given by <P,T,E> ❑ www.SunilOS.com 3
- 4. Traditional programming v/s machine learning www.SunilOS.com 4
- 5. Why Machine Learning? ❑Human expertise does not exist (Mission Mangal). www.SunilOS.com 5
- 6. Robotics ❑Human are not able to perform task with great expertise. www.SunilOS.com 6
- 7. Personalized Recommendation ❑Models must be customized (Netflix, Amazon) www.SunilOS.com 7
- 8. Big Data ❑Models are based on huge amounts of data. www.SunilOS.com 8
- 9. Why Now? ❑Flood of available data(especially with the use of Internet) ❑Increasing Computational power. ❑Growing progress in available algorithms and theory developed by researchers ❑Increasing support in Industries www.SunilOS.com 9
- 10. Types of Machine Learning www.SunilOS.com 10
- 11. Types of Learning ❑Supervised (inductive) learning: o Given: training data + desired outputs (labels). ❑Unsupervised learning o Given: training data (without desired outputs) ❑Reinforcement learning o Rewards from sequence of actions www.SunilOS.com 11
- 12. Supervised Learning www.SunilOS.com 12 WEIGHT = FEATURE CURRENCY= LABEL 03 Grams = 01 Rupee Coin
- 17. Applications of Machine Learning Model
- 18. Framing Learning Problem www.SunilOS.com 18 ❑Choose the data. ❑Choose exactly what is to be learned. o i.e. the target function ❑Choose the model ❑Train the model ❑Evaluate the model
- 20. Evaluation Criteria ❑Accuracy ❑Precision and recall ❑Squared error ❑Likelihood ❑Posterior probability ❑Cost/Utility ❑Margin ❑Entropy ❑etc. www.SunilOS.com 20
- 21. Performance Measure of Model ❑The metrics that you choose to evaluate your machine learning algorithms are very important. ❑Choice of metrics influences how the performance of machine learning algorithms is measured and compared. o Classification Metrics o Regression Metrics o Multilabel ranking metrics o Clustering metrics www.SunilOS.com 21
- 22. Classification Metrics ❑Classification accuracy is the number of correct predictions made as a ratio of all predictions made. o Correct Prediction/Total Prediction ❑This is the most common evaluation metric for classification problems. ❑It is really only suitable when o There are an equal number of observations in each class (which is rarely the case) and o That all predictions and prediction errors are equally important, which is often not the case. www.SunilOS.com 22
- 23. Classification Accuracy ❑ import numpy as np ❑ from sklearn.metrics import accuracy_score ❑ y_pred = [0, 2, 1, 3] ❑ y_true = [0, 1, 2, 3] ❑ print("Acurracy",accuracy_score(y_true, y_p red)) ❑ print("Accuracy:",accuracy_score(y_true, y_ pred, normalize=False)) ❑ #Output o Acurracy 0.5 o Accuracy: 2 www.SunilOS.com 23
- 24. Cohen’s kappa ❑The kappa score is a number between -1 and 1. Scores above .8 are generally considered good; zero or lower means not good (practically random labels). ❑Kappa scores can be computed for binary or multiclass problems, but not for multilabel problems www.SunilOS.com 24
- 25. Cohen’s kappa (cont.) ❑ from sklearn.metrics import cohen_kappa_sco re ❑ y_true = [2, 0, 2, 2, 0, 1] ❑ y_pred = [0, 0, 2, 2, 0, 2] ❑ print("Kappa Score=",cohen_kappa_score(y_tr ue, y_pred)) ❑ #Output o 0.4285714285714286 www.SunilOS.com 25
- 26. Precision and Recall ❑ Precision and Recall both are the important to evaluate the model. It is used for binary class labels. ❑ Precision can be said as a positive predictive value. The precision is the ratio tp / (tp + fp) where o tp is the number of true positives o fp the number of false positives. ❑ The precision is the ability of the classifier not to label as positive a sample that is negative. ❑ Recall is the ratio tp / (tp + fn) o where tp is the number of true positives o fn the number of false negatives. o The recall is the ability of the classifier to find all the positive samples. www.SunilOS.com 26
- 27. Precision and Recall (cont.) www.SunilOS.com 27
- 28. Precision and Recall (cont.) ❑ from sklearn import metrics ❑ y_pred = [0, 1, 0, 0] ❑ y_true = [0, 1, 0, 1] ❑ print("Precision:",metrics.precision_score(y_tr ue, y_pred)) ❑ # 1.0 ❑ print("Recall:,metrics.recall_score(y_true, y_p red)) ❑ # 0.5 www.SunilOS.com 28
- 29. Confusion Matrix ❑ The confusion matrix is a presentation of the accuracy of a model with two or more classes. ❑ The table presents predictions on the x-axis and accuracy outcomes on the y-axis. ❑ The cells of the table are the number of predictions made by a machine learning algorithm. ❑ For example, a machine learning algorithm can predict 0 or 1 and each prediction may actually have been a 0 or 1. o Predictions for 0 that were actually 0 , appear in the cell for prediction=0 and actual=0, o whereas predictions for 0 that were actually 1, appear in the cell for prediction = 0 and actual=1. and so on www.SunilOS.com 29
- 30. Confusion Matrix (cont.) ❑ The parameter normalize allows to report ratios instead of counts. The confusion matrix can be normalized in 3 different ways: 'pred', 'true', and 'all' which will divide the counts by the sum of each columns, rows, or the entire matrix, respectively. ❑ from sklearn.metrics import confusion_matrix ❑ y_true = [2, 0, 2, 2, 0, 1] ❑ y_pred = [0, 0, 2, 2, 0, 2] ❑ print(confusion_matrix(y_true, y_pred)) ❑ #Output ❑ [[2 1] ❑ [2 3]] www.SunilOS.com 30
- 31. Confusion Matrix with normalize parameter ❑ from sklearn.metrics import confusion_matrix ❑ y_true = [0, 0, 0, 1, 1, 1, 1, 1] ❑ y_pred = [0, 1, 0, 1, 0, 1, 0, 1] ❑ # print(confusion_matrix(y_true, y_pred, normal ize="true")) ❑ print(confusion_matrix(y_true, y_pred, normaliz e="all")) ❑ # print(confusion_matrix(y_true, y_pred, normal ize="pred")) ❑ #Output ❑ [[0.25 0.125] ❑ [0.25 0.375]] www.SunilOS.com 31
- 32. Get count of Values ❑For binary problems, we can get counts of true negatives, false positives, false negatives and true positives as follows: o tn, fn, fp, tp=confusion_matrix(y_true, y _pred).ravel()) www.SunilOS.com 32
- 33. Classification report ❑ The classification_report function builds a text report showing the main classification metrics. Here is a small example with custom target_names and inferred labels: o from sklearn.metrics import classification_re port o y_true = [0, 1, 2, 2, 0] o y_pred = [0, 0, 2, 1, 0] o target_names = ['class 0', 'class 1', 'class 2'] o print(classification_report(y_true, y_pred, t arget_names=target_names)) www.SunilOS.com 33
- 34. Classification Report Output ❑ precision recall f1-score support ❑ class 0 0.67 1.00 0.80 2 ❑ class 1 0.00 0.00 0.00 1 ❑ class 2 1.00 0.50 0.67 2 ❑ accuracy 0.60 5 ❑ macro avg 0.56 0.50 0.49 5 ❑ weighted avg 0.67 0.60 0.59 5 www.SunilOS.com 34
- 35. Hamming Loss ❑ The hamming loss computes the average Hamming loss or Hamming distance between two sets of samples. ❑ It is Used for multi class classification. ❑ Hamming Loss calculates loss generated in the bit string of class labels during prediction. ❑ It does that by XOR between the actual and predicted labels and then average across the dataset. o from sklearn.metrics import hamming_loss o y_pred = [1, 2, 3, 4,6,7,8] o y_true = [2, 2, 3, 4,5,6,7] o print(hamming_loss(y_true, y_pred)) www.SunilOS.com 35
- 36. F-measures ❑The F-Measure (Fβ and F1 measures) can be interpreted as a weighted harmonic mean of the precision and recall. A Fβ measure reaches its best value at 1 and its worst score at 0. o from sklearn import metrics o y_pred = [0, 1, 0, 0] o y_true = [0, 1, 0, 1] o metrics.f1_score(y_true, y_pred) www.SunilOS.com 36
- 37. Receiver operating characteristic ❑ The function roc_curve computes the receiver operating characteristic curve, or ROC curve ❑ ROC curve, is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied. ❑ It is created by plotting the fraction of true positives out of the positives (TPR = true positive rate) vs. the fraction of false positives out of the negatives (FPR = false positive rate), at various threshold settings. ❑ TPR is also known as sensitivity, and FPR is one minus the specificity or true negative rate. www.SunilOS.com 37
- 38. Area under ROC curve ❑ AUC ranges in value from 0 to 1. A model whose predictions are 100% wrong has an AUC of 0.0; one whose predictions are 100% correct has an AUC of 1.0. ❑ import numpy as np ❑ from sklearn.metrics import roc_auc_score ❑ y_true = np.array([0, 0, 1, 1]) ❑ y_scores = np.array([0.1, 0.4, 0.35, 0.8]) ❑ roc_auc_score(y_true, y_scores) www.SunilOS.com 38
- 40. Regression Metrics ❑ We will review 3 of the most common metrics for evaluating predictions on regression machine learning problems: ❑ Mean Absolute Error. o The Mean Absolute Error (or MAE) is the average of the absolute differences between predictions and actual values. It gives an idea of how wrong the predictions were. ❑ Mean Squared Error. o Same as the mean absolute error. o Taking the square root of the mean squared error converts the units back to the original units of the output variable and can be meaningful for description and presentation. This is called the Root Mean Squared Error (or RMSE). www.SunilOS.com 40
- 41. Regression Metrics ❑ R^2 o The R^2 (or R Squared) metric provides an indication of the goodness of fit of a set of predictions to the actual values. o In statistical literature, this measure is called the coefficient of determination. o This is a value between 0 and 1 for no-fit and perfect fit respectively. ❑These functions have an multioutput keyword argument which specifies the way the scores or losses for each individual target should be averaged. Values for this keyword o uniform_average o raw_values www.SunilOS.com 41
- 43. Mean absolute error (cont.) ❑ from sklearn.metrics import mean_absolute_error ❑ y_true = [3, -0.5, 2, 7] ❑ y_pred = [2.5, 0.0, 2, 8] ❑ mean_absolute_error(y_true, y_pred) ❑ y_true = [[0.5, 1], [-1, 1], [7, -6]] ❑ y_pred = [[0, 2], [-1, 2], [8, -5]] ❑ mean_absolute_error(y_true, y_pred)) ❑ mean_absolute_error(y_true, y_pred, multioutput='raw _values') ❑ mean_absolute_error(y_true, y_pred, multioutput=[0.3 , 0.7]) www.SunilOS.com 43
- 45. Mean squared error ❑ from sklearn.metrics import mean_squared_error ❑ y_true = [3, -0.5, 2, 7] ❑ y_pred = [2.5, 0.0, 2, 8] ❑ mean_squared_error(y_true, y_pred)) ❑ y_true = [[0.5, 1], [-1, 1], [7, -6]] ❑ y_pred = [[0, 2], [-1, 2], [8, -5]] ❑ mean_squared_error(y_true, y_pred)) ❑ Output: ❑ MSE: 0.375 ❑ MSE: 0.7083333333333334 www.SunilOS.com 45
- 46. Mean squared logarithmic error www.SunilOS.com 46
- 47. ❑ from sklearn.metrics import mean_squared_log_error ❑ y_true = [3, 5, 2.5, 7] ❑ y_pred = [2.5, 5, 4, 8] ❑ mean_squared_log_error(y_true, y_pred) ❑ y_true = [[0.5, 1], [1, 2], [7, 6]] ❑ y_pred = [[0.5, 2], [1, 2.5], [8, 8]] ❑ mean_squared_log_error(y_true, y_pred) ❑ Output: ❑ MSLE: 0.03973012298459379 ❑ MSLE: 0.044199361889160536 www.SunilOS.com 47
- 48. Median absolute error ❑ The median_absolute_error is particularly interesting because it is robust to outliers. ❑ The loss is calculated by taking the median of all absolute differences between the target and the prediction. ❑ If yj is the predicted value of the i-th sample and yi is the corresponding true value, then the median absolute error (MedAE) estimated over nsamples is defined as ❑ It does not support multioutput. o MeadAE(yi,yj)=median(|yi-yj|,……,|yi-yj|) www.SunilOS.com 48
- 49. ❑from sklearn.metrics import ❑median_absolute_error ❑y_true = [3, -0.5, 2,7] ❑y_pred = [2.5, 0.0, 2, 8] ❑median_absolute_error(y_true, y_pred) www.SunilOS.com 49
- 50. R² score, the coefficient of determination ❑The r2 function computes the coefficient of determination, usually denoted as R². ❑It represents the proportion of variance (of y) that has been explained by the independent variables in the model. ❑ It provides an indication of goodness of fit and therefore a measure of how well unseen samples are likely to be predicted by the model, through the proportion of explained variance. www.SunilOS.com 50
- 51. ❑As such variance is dataset dependent, R² may not be meaningfully comparable across different datasets. ❑Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). ❑A constant model that always predicts the expected value of y, disregarding the input features, would get a R² score of 0.0. ❑If yj is the predicted value of the i-th sample and yi is the corresponding true value for total n samples, the estimated R² is defined as: www.SunilOS.com 51
- 52. ❑ from sklearn.metrics import r2_score ❑ y_true = [3, -0.5, 2, 7] ❑ y_pred = [2.5, 0.0, 2, 8] ❑ r2_score(y_true, y_pred) ❑ y_true = [[0.5, 1], [-1, 1], [7, -6]] ❑ y_pred = [[0, 2], [-1, 2], [8, -5]] r2_score(y_true, y_pred, multioutput='variance_weighted') ❑ y_true = [[0.5, 1], [-1, 1], [7, -6]] ❑ y_pred = [[0, 2], [-1, 2], [8, -5]] ❑ r2_score(y_true, y_pred, multioutput='uniform_average') ❑ r2_score(y_true, y_pred, multioutput='raw_values') ❑ r2_score(y_true, y_pred, multioutput=[0.3, 0.7]) www.SunilOS.com 52
- 53. Clustering Metrics ❑Adjusted Rand index ❑Mutual Information based scores ❑Homogeneity, completeness and V-measure ❑Fowlkes-Mallows scores ❑Silhouette Coefficient ❑Contingency Matrix www.SunilOS.com 53
- 54. Machine Learning In Brief ❑Tens of thousands of machine learning algorithms ❑Every ML algorithm has three components: o Representation o Optimization o Evaluation www.SunilOS.com 54
- 55. Tools For Machine Learning ❑ Machine Learning Libraries: o Scikit Learn: Built on Numpy, Scipy and matplotlib. Used for Supervised Learning and Unsupervised Learning. o Keras: Built on Tensor flow o Tensorflow: For NN and Deep Learning ❑ Programming: o Python ❑ Machine Learning Tools o Anaconda: Free distribution of Python, Package of scientific computing o Jupyter: web Interface for Python Programming ❑ Data Visualization o Matplotlib o Numpy o Pandas www.SunilOS.com 55
- 56. What we will cover in this Course ❑Supervised Learning ❑Unsupervised Learning ❑Reinforcement Learning ❑Performance Measure ❑Applications of Machine Learning ❑Data Preprocessing ❑Data visualization www.SunilOS.com 56
- 57. Disclaimer ❑This is an educational presentation to enhance the skill of computer science students. ❑This presentation is available for free to computer science students. ❑Some internet images from different URLs are used in this presentation to simplify technical examples and correlate examples with the real world. ❑We are grateful to owners of these URLs and pictures. www.SunilOS.com 57