The document discusses the MNIST dataset and classification techniques. It contains: - An overview of the MNIST dataset containing 70,000 handwritten digit images - Preparation of the data for classification, including splitting into training and test sets - Binary classification to detect the digit 5 using a linear SGD classifier - Evaluation metrics like accuracy, precision, recall, F1 score, and confusion matrices - Tuning the classifier by adjusting decision thresholds to balance precision and recall

1. 3
Classification
Taka Wang
20210729

2. MNIST Dataset
2
70,000 images
28x28=784 pixels

3. from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1, as_frame=False)
mnist.keys() # 'data', 'target', 'frame' ...
X, y = mnist["data"], mnist["target"]
print(X.shape, y.shape) # (70000, 784) (70000,)
some_digit = X[0]
some_digit_image = some_digit.reshape(28, 28)
plt.imshow(some_digit_image, cmap=mpl.cm.binary)
y = y.astype(np.uint8) # y is `str` type
print(y[0], type(y[0])) # 5 <class 'numpy.uint8'>
X_train, X_test= X[:60000], X[60000:]
y_train, y_test = y[:60000], y[60000:]
print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
# (60000, 784) (10000, 784) (60000,) (10000,)
Data Preparation
60K
10K
784 1
X y

4. 4

5. 5

6. 6
Binary
Classification

7. Train a Linear 5-Detector
y_train_5 = (y_train == 5) # True for all 5s, False for all other digits.
y_test_5 = (y_test == 5)
from sklearn.linear_model import SGDClassifier
sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train_5)
sgd_clf.predict([some_digit])
# array([ True])

8. Linear Model in Scikit-Learn
8

9. from sklearn.linear_model import SGDClassifier
sgd_clf = SGDClassifier(random_state=42)
sgd_clf.fit(X_train, y_train_5)
sgd_clf.predict([some_digit])
# array([ True])
from sklearn.model_selection import cross_val_score
cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring="accuracy")
# array([0.95035, 0.96035, 0.9604 ]) <-- Awesome?!
Measuring Accuracy Using Cross-Validation
3-fold
cross-validation (cv)

10. Dumb classifier - Not-5 Detector
10
from sklearn.base import BaseEstimator
class Never5Classifier(BaseEstimator):
def fit(self, X, y=None):
pass
def predict(self, X):
return np.zeros((len(X), 1), dtype=bool)
never_5_clf = Never5Classifier()
cross_val_score(never_5_clf, X_train, y_train_5, cv=3, scoring="accuracy")
# array([0.91125, 0.90855, 0.90915])
This demonstrates why accuracy is generally not the
preferred performance measure for classifiers,
especially when you are dealing with skewed datasets.

11. Confusion Matrix
# pretend we reached perfection
y_train_perfect_predictions = y_train_5
confusion_matrix(y_train_5, y_train_perfect_predictions)
# array([[54579, 0],
# [ 0, 5421]])
from sklearn.model_selection import cross_val_predict
y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)
from sklearn.metrics import confusion_matrix
confusion_matrix(y_train_5, y_train_pred)
# array([[53892, 687],
# [ 1891, 3530]])
直觀：計算A類別被誤認為B類別的個數
Negative Postive
TN FP(偽陽)
FN(偽陰) TP
Negative
Positive
Predicted
Ground-Truth
output is label

12. 12
30x30
Confusion Matrix

13. Measurement Metrics: Precision and Recall
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_score, recall_score
precision_score(y_train_5, y_train_pred)
# 0.7290850836596654
cm = confusion_matrix(y_train_5, y_train_pred)
cm[1, 1] / (cm[0, 1] + cm[1, 1])
# 0.7290850836596654
recall_score(y_train_5, y_train_pred)
# 0.6511713705958311
cm[1, 1] / (cm[1, 0] + cm[1, 1])
# 0.6511713705958311
Negative Postive
TN FP(偽陽)
FN(偽陰) TP
Negative
Positive
Predicted
Ground-Truth
Precision 若預測為真，而且確實為真的比例
(謹慎出手，只出手一次，而且確保是對的)
Recall 在確實為真的中，可以正確找出多少
(若亂槍打鳥，全部都預測為真呢?)
sensitivity or
true positive rate
宣稱是5，只有七成三真的是5
所有的5中，只找出六成五

14. Confusion Matrix
14
Negative Postive
TN FP(偽陽)
FN(偽陰) TP
Negative
Positive
Predicted
Ground-Truth

15. Precision + Recall
15

16. F1 Score
16
Harmonic mean
Arithmetic mean
Harmonic mean
● The F1 score favors classifiers that
have similar precision and recall.
● the harmonic mean gives much more
weight to low values.

17. Precision/Recall Tradeoff
17
If you trained a classifier to detect videos that are safe for kids, you would probably
prefer a classifier that rejects many good videos (low recall) but keeps only safe ones
(high precision), rather than a classifier that has a much higher recall but lets a few
really bad videos show up in your product. <<寧可錯過好片，一出手只有好影片>>
Suppose you train a classifier to detect shoplifters on surveillance images: it is
probably fine if your classifier has only 30% precision as long as it has 99% recall (sure,
the security guards will get a few false alerts, but almost all shoplifters will get
caught). <<寧可錯殺，不可放過>>
Precision 若預測為真，而且確實為真的比例
(謹慎出手，只出手一次，而且確保是好影片 )
Recall 在確實為真的中，可以正確找出多少
(亂槍打鳥，全部都預測為小偷?)
Precision
Recall

18. Precision/Recall Tradeoff
Negative Postive
TN FP(偽陽)
FN(偽陰) TP
Negative
Positive
Predicted
Ground-Truth
6
predicted as `5`
decision_function

19. Adjust Thresholds
19
y_scores = sgd_clf.decision_function([some_digit])
# array([2412.53175101])
threshold = 0
y_some_digit_pred = (y_scores > threshold)
# array([ True])
threshold = 8000
y_some_digit_pred = (y_scores > threshold)
# array([False])
y_scores = cross_val_predict(
sgd_clf, X_train, y_train_5, cv=3,
method="decision_function")
from sklearn.metrics import precision_recall_curve
precisions, recalls, thresholds =
precision_recall_curve(y_train_5, y_scores)

20. Preceison-Recall Curve (PR Curve)
20
threshold_90_precision = thresholds[np.argmax(precisions >= 0.90)]
# ~7816
y_train_pred_90 = (y_scores >= threshold_90_precision)
# [False False False ... True False False]
precision_score(y_train_5, y_train_pred_90)
# 0.9000380083618396
recall_score(y_train_5, y_train_pred_90)
# 0.4368197749492714
y_scores = cross_val_predict(
sgd_clf, X_train, y_train_5, cv=3,
method="decision_function")
precisions, recalls, thresholds =
precision_recall_curve(y_train_5, y_scores)

21. If someone says let’s reach
99% precision.
you should ask, at what recall?
21

22. The ROC Curve (receiver operating characteristic)
Negative Postive
TN FP(偽陽)
FN(偽陰) TP
Negative
Positive
Predicted
Ground-Truth
FPR = 1-TNR
TPR = Recall
from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)
plt.plot(fpr, tpr, linewidth=2, label=label)
plt.plot([0, 1], [0, 1], 'k--') 22

23. Supplement
23
/ˌspɛsɪˈfɪsɪti
/
TPR = Recall TNR
predicted as
Positive

24. ROC Curve and AUC (Area under Curve)
24
from sklearn.metrics import roc_auc_score
roc_auc_score(y_train_5, y_scores)
# 0.9611778893101814
A good classifier stays as far away from
that line as possible (toward the top-left
corner).

25. Tips
25
Prefer the PR curve whenever the positive
class is rare or when you care more about the
false positives than the false negatives.

26. 26
Multiclass
Classification

27. One-versus-All (OvA) vs One-versus-One (OvO)
27
0-detector
1-detector
8-detector
9-detector
0/1 CLF
0/2 CLF
0/3 CLF
9/8 CLF
9/9 CLF
For most
binary classification algorithm
Support Vector Machines
(Scale Poorly)
.
.
.
.
.
.
one-versus-the rest

28. Error Analysis (How to Debug your model)
28
y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv=3)
conf_mx = confusion_matrix(y_train, y_train_pred)
```
array([[5578, 0, 22, 7, 8, 45, 35, 5, 222, 1],
[ 0, 6410, 35, 26, 4, 44, 4, 8, 198, 13],
[ 28, 27, 5232, 100, 74, 27, 68, 37, 354, 11],
[ 23, 18, 115, 5254, 2, 209, 26, 38, 373, 73],
[ 11, 14, 45, 12, 5219, 11, 33, 26, 299, 172],
[ 26, 16, 31, 173, 54, 4484, 76, 14, 482, 65],
[ 31, 17, 45, 2, 42, 98, 5556, 3, 123, 1],
[ 20, 10, 53, 27, 50, 13, 3, 5696, 173, 220],
[ 17, 64, 47, 91, 3, 125, 24, 11, 5421, 48],
[ 24, 18, 29, 67, 116, 39, 1, 174, 329, 5152]])
```

29. Error Analysis
29
plt.matshow(conf_mx, cmap=plt.cm.gray)
plt.show()
row_sums = conf_mx.sum(axis=1, keepdims=True)
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()

30. 30
Predict: 3 Predict: 5

31. Summary
● Accuracy is terrible metric for skewed datasets
● Confusion Matrix
● Precision, Recall, F1 Score
● Precision/Recall Tradeoff
● PR Curve
● ROC Curve
● AUC
● Error Analysis
31

32. 遺珠之憾
32

33. Implementing Cross-Validation
33
from sklearn.model_selection import StratifiedKFold
from sklearn.base import clone
skfolds = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
for train_index, test_index in skfolds.split(X_train, y_train_5):
clone_clf = clone(sgd_clf)
X_train_folds = X_train[train_index]
y_train_folds = y_train_5[train_index]
X_test_fold = X_train[test_index]
y_test_fold = y_train_5[test_index]
clone_clf.fit(X_train_folds, y_train_folds)
y_pred = clone_clf.predict(X_test_fold)
n_correct = sum(y_pred == y_test_fold)
print(n_correct / len(y_pred))

34. ● Multilabel Classification
Say the classifier has been trained to recognize three faces, Alice, Bob, and Charlie;
then when it is shown a picture of Alice and Charlie, it should output [1, 0, 1]
(meaning “Alice yes, Bob no, Charlie yes”).
34
● Multioutput Classification (Multioutput Multiclass
Classification)

35. 謝謝
35