The document discusses binary classification techniques including Naive Bayes classifiers and support vector machines (SVM). It explains that Naive Bayes classifiers use Bayes' theorem to calculate conditional probabilities for classification. Specifically, it describes how to calculate priors and fit probability distributions like binomial, multinomial, and Gaussian distributions for the Naive Bayes model. The document also outlines how SVMs find the optimal separating hyperplane between two classes by maximizing the margin between them. It provides code examples for implementing Naive Bayes and SVM classifiers and evaluating their performance.

1. Programming for Data
Analysis
Week 11
Dr. Ferdin Joe John Joseph
Faculty of Information Technology
Thai – Nichi Institute of Technology, Bangkok

2. Today’s lesson
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
2
• Binary Classification
• Naïve Bayes Classifier
• Support Vector Machine

3. Naïve Bayes Classifier
• Conditional Probability Model of Classification
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
3

4. Conditional Probability Model of Classification
• The conditional probability can be calculated using the joint
probability, although it would be intractable.
• Bayes Theorem provides a principled way for calculating the
conditional probability.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
4

5. Bayes Theorem
• P(A|B) = P(B|A) * P(A) / P(B)
• We can frame classification as a conditional classification problem
with Bayes Theorem as follows:
P(yi | x1, x2, …, xn) = P(x1, x2, …, xn | yi) * P(yi) / P(x1, x2, …, xn)
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
5

6. Naïve Bayes
• For simplifying the calculation
• The Bayes Theorem assumes that each input variable is dependent
upon all other variables.
• This is a cause of complexity in the calculation.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
6

7. Calculation of Prior and Conditional
Probabilities
• P(yi) = examples with yi / total examples
• In the case of categorical variables, such as counts or labels, a
multinomial distribution can be used.
• If the variables are binary, such as yes/no or true/false, a binomial
distribution can be used.
• If a variable is numerical, such as a measurement, often a Gaussian
distribution is used.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
7

8. Naïve Bayes Distribution
• Binomial Naïve Bayes: Naïve Bayes that uses a binomial distribution.
• Multinomial Naïve Bayes: Naïve Bayes that uses a multinomial
distribution.
• Gaussian Naïve Bayes: Naïve Bayes that uses a Gaussian distribution.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
8

9. Preparation of dataset
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
9

10. Fitting Probability Distribution
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
10

11. Sorting Data
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
11

12. Calculate Priors
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
12

13. Fit Distribution
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
13

14. Summary
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
14

15. Independent Conditional Probability
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
15

16. Taking a sample
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
16

17. Probability Scores
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
17

18. Wrap Up
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
18

19. Gaussian Naïve Bayes
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
19

20. Lab Exercise
• Use this source code and make a classification report which gives
accuracy, precision, recall and F1-score
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
20

21. Support Vector Machine
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
21

22. SVM Objective
• A training set, S, for an SVM is comprised of m samples.
• The features, x, consist of real numbers and the classifications, y,
must be -1 or 1.
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
22

23. SVM Hyperplane
• The SVM hyperlane is defined by the weight vector, w, and the bias, b,
and is defined as:
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
23

24. Example for 2 Feature
• Hyperplane for two features can be written as:
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
24

25. Libraries
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
25

26. Data Clusters
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
26

27. Data Clusters
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
27

28. Prepare Datasets
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
28

29. Learning Rate
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
29

30. Learning Rate
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
30

31. Draw Decision Boundary
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
31

32. Plot Data
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
32

33. Plot Data
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
33

34. Test Classifier
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
34

35. Test Classifier
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
35

36. Output
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
36

37. Lab Exercise
• Create Confusion Matrix and classification report for SVM
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
37

38. DSA 207 – Binary Classifier
Faculty of Information Technology, Thai - Nichi Institute of
Technology, Bangkok
38