This document provides an overview of support vector machines (SVMs). It discusses how SVMs can be used to perform classification tasks by finding optimal separating hyperplanes that maximize the margin between different classes. The document outlines how SVMs solve an optimization problem to find these optimal hyperplanes using techniques like Lagrange duality, kernels, and soft margins. It also covers model selection methods like cross-validation and discusses extensions of SVMs to multi-class classification problems.

1. Support Vector Machine Shao-Chuan Wang 1

2. Support Vector Machine 1D Classification Problem: how will you separate these data?(H1, H2, H3?) 2 H1 H2 H3 x 0

3. Support Vector Machine 2D Classification Problem: which H is better? 3

4. Max-Margin Classifier Functional Margin Geometric Margin 4 We feel more confident when functional margin is larger Note that scaling on w, b won’t change the plane. Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

5. Maximize margins Optimization problem: maximize minimal geometric margin under constraints. Introduce scaling factor such that 5 Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

6. Optimization problem subject to constraints Maximize f(x, y), subject to constraint g(x, y) = c 6 -> Lagrange multiplier method

7. Lagrange duality Primal optimization problem: GeneralizedLagrangian method Primal optimization problem (equivalent form) Dual optimization problem: 7 Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

8. Dual Problem The necessary conditions that equality holds: f, giare convex, and hi are affine. KKT conditions. 8 Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

9. Optimal margin classifiers Its Lagrangian Its dual problem 9 Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

10. Support Vector Machine (cont’d) If not linearly separable, we can Find a nonlinear solution Technically, it’s a linear solution in higher-order space Kernel Trick 26

11. Kernel and feature mapping Kernel: Positive semi-definite Symmetric For example: Loose Intuition “similarity” between features 11 Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

12. Soft Margin (L1 regularization) 12 C = ∞ leads to hard margin SVM, Rychetsky (2001) Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

13. Why doesn’t my model fit well on test data ? 13

14. Bias/variance tradeoff underfitting(high bias) overfitting(high variance) Training Error = Generalization Error = 14 In-sample error Out-of-sample error Andrew Ng. Part V Support Vector Machines. CS229 Lecture Notes (2008).

15. Bias/variance tradeoff 15 T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer series in statistics. Springer, New York, 2001.

16. Is training error a good estimator of generalization error? 16

17. Chernoff bound (|H|=finite) Lemma: Assume Z1, Z2, …, Zmare drawn iid from Bernoulli(φ), and and let γ > 0 be fixed. Then, based on this lemma, one can find, with probability 1-δ (k = # of hypotheses) 17 Andrew Ng. Part VI Learning Theory. CS229 Lecture Notes (2008).

18. Chernoff bound (|H|=infinite) VC Dimension d : The size of largest set that H can shatter. e.g. H = linear classifiers in 2-D VC(H) = 3 With probability at least 1-δ, 18 Andrew Ng. Part VI Learning Theory. CS229 Lecture Notes (2008).

21. Model Selection Loop possible parameters: Pick one set of parameter, e.g. C = 2.0 Do cross validation, get a error estimation Pick the Cbest (with minimal error estimation) as the parameter 20

22. Multiclass SVM One against one There are binary SVMs. (1v2, 1v3, …) To predict, each SVM can vote between 2 classes. One against all There are k binary SVMs. (1 v rest, 2 v rest, …) To predict, evaluate , pick the largest. Multiclass SVM by solving ONE optimization problem 21 K = 1 3 5 3 2 1 1 2 3 4 5 6 K = 3 poll Crammer, K., & Singer, Y. (2001). On the algorithmic implementation of multiclass kernel-based vector machines. JMLR, 2, 265-292.

23. Multiclass SVM (2/2) DAGSVM (Directed Acyclic Graph SVM) 22

24. An Example: image classification Process 23 K = 6 1/4 3/4 1 0:49 1:25 … 1 0:49 1:25 … ： ： 2 0:49 1:25 … ： Test Data Accuracy

25. An Example: image classification Results Run Multi-class SVM 100 times for both (linear/Gaussian). Accuracy Histogram 24