This document provides an overview of classification techniques in machine learning, including: 1. It reviews binary classification problems and how logistic regression can be used to predict the probability that an input belongs to a particular class. 2. It then discusses multi-class classification problems with more than two classes and how logistic regression can be extended to provide probabilities for each class. 3. Examples are given of classifying tumors as malignant or benign and predicting whether a user has a flip phone based on age and income. Visualizations of logistic regression are also presented.

1. Introduction to
Classification
Class 2
1
Nandita Naik, Proof School
Oliver Zhang, Proof School
02/24/2018

2. Lesson Plan:
1. Homework Survey and Announcements
2. Review
a. Loss Function
b. Gradient Descent
c. Classification vs Regression
3. Binary Classification
4. Multi Class Classification

3. Check in: Fill out the Homework Survey!
Announcements:
- Practical AI Syllabus

4. Machine Learning Model:
Predicting and Training

5. 5
Grade
Homework
hours per
week
f(grade) = homework
hours

6. 6
Grade
Homework
hours per
week
f(grade) = homework
hours

7. 7
Grade
Homework
hours per
week
f(grade) = homework
hours

8. 8
Grade
Homework
hours per
week
f(grade) = homework
hours

9. 9
Grade
Homework
hours per
week
f(grade) = homework
hours

10. 10
Grade
Homework
hours per
week
f(grade) = homework
hours

11. Loss: A measure of how bad your function is doing
Which Picture Has More Loss?

12. Loss is based on your line’s slope and its y-intercept
Grade
Homework
hours per
week
w = 1/3
b = 5

13. Loss Function
We then build a loss function that takes in w
and b and spits out the loss associated with
the line y = wx + b.
For y = wx + b, loss(w, b).
(y = wx + b)

14. B -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0
W
2.0
1.5
1.0
0.5
0.0
-0.5
-1.0
-1.5
-2.0
(y = wx + b)

15. B -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0
W
2.0 4.03 4.53 5.53 7.03 9.03 11.5 14.5 18.0 22.0
1.5 5.80 4.05 2.80 2.05 1.80 2.05 2.80 4.05 5.80
1.0 21.8 17.8 14.3 11.3 8.82 6.83 5.33 4.33 3.83
0.5 52.1 45.8 30.1 34.8 30.1 25.8 22.1 18.8 16.1
0.0 96.6 88.1 80.1 72.6 65.5 59.1 53.1 47.6 42.6
-0.5 155 144 134 124 115 106 98 90 83
-1.0 228 215 202 190 179 168 157 147 138
-1.5 315 300 285 271 257 244 231 219 207
-2.0 417 399 382 366 350 334 319 305 291
(y = wx + b)

16. 16
Loss Function
W
b
Loss
(y = wx + b)

17. Gradient Descent:
Method of Minimizing Loss

18. Loss
b
w
(y = wx + b)

19. Loss
b
w
(y = wx + b)

20. Gradient Descent
20
(y = wx + b)

21. Regression vs. Classification?

22. Example:

23. Regression:
The Outputs are a spectrum

24. Example:

25. Classification:
The Outputs are certain categories

26. Example:

27. Binary Classification

28. Binary Classification
Cat or not?

29. Logistic regression: want model to predict a
probability of belonging to a class
Probability of being a cat = 0.6

30. Logistic regression is a regression problem
because it predicts a numerical value (i.e. the
probability between 0 and 1) for an input.
If we “round” the probability to the nearest class
(i.e. 0.97 -> 1, and 0.24 -> 0), then we get binary
classification.

31. Classifying a tumor as malignant or benign
Tumor Size
example from Andrew Ng’s lecture 6
Probability
it’s Malignant

32. Let’s have the y-axis be the probability that the
tumor is malignant.
Tumor Size
0
1
Probability
it’s Malignant

33. What happens if we try linear regression?
Tumor Size
Probability
it’s Malignant
0
1

34. What happens if we try linear regression?
Tumor Size
0
1
thresholdProbability
it’s Malignant

35. Tumor Size
0
1
Problem: we may get values < 0 or > 1
Probability
it’s Malignant

36. Using the sigmoid function, we can squash the
output into the range [0,1].
y = mx + b
0
1
(Predicting Malignancy
Based on Tumor Size)
Malignancy0
Probability
it’s Malignant

37. Probability
it’s Malignant
0
1
Malignancy0
S(0)?

38. 0
1
Malignancy0
½
S(0)?
Probability
it’s Malignant

39. 0
1
Malignancy0
S(1)?
1
¾
Probability
it’s Malignant

40. 0
1
Malignancy0
S(-1)?
1-1
¼
Probability
it’s Malignant

41. Using the sigmoid function, we can squash the
output into the range [0,1].
y = mx + b
Probability
it’s Malignant
0
1
Malignancy0
(Predicting Malignancy
Based on Tumor Size)

42. Tumor Size
Probability
it’s Malignant
0
1
Recap: Linear Regression vs. Logistic Regression
Tumor Size
Probability
it’s Malignant
0
1

43. Tumor Size
Probability
it’s Malignant
0
1
Recap: Linear Regression vs. Logistic Regression
Tumor Size
Probability
it’s Malignant
0
1
0.5

44. Recap: Linear Regression vs. Logistic Regression
Tumor Size
Probability
it’s Malignant
0
1
0.5

45. Logistic Regression Visualization
Age
Income
Circle : Flip Phone User
Plus : Not a Flip Phone User.

46. Logistic Regression Visualization
Age
Income
Circle : Flip Phone User
Plus : Not a Flip Phone User.

47. Logistic Regression Visualization
Age
Income

48. Logistic Regression Visualization
Age
Income

49. Logistic Regression Visualization
Age
Income

50. Logistic Regression Visualization
Age
Income

51. Logistic Regression Visualization
Age
Income

53. Multi Class Classification

54. Multi Class Classification

55. Multi Class Classification Probability it’s a:
- dog = 0.3
- cat = 0.7
- guinea pig = 0.5

56. Probability it’s a:
- dog = 0.8
- cat = 0.5
- guinea pig = 0.4

57. We will have two implementation problems for
you.
1. Straightforward Logistic Regression
2. Predicting Titanic Survival on Kaggle

58. Homework:
- Read Binary and Multi Class Classification Notes
- Watch Logsitic Regression
- Ask Questions to make sure you understand material!