This document introduces support vector machines (SVMs), including their use of hyperplanes to create classifiers with maximal marginal widths. It discusses how SVMs solve convex optimization problems to find the optimal hyperplane. Kernels are introduced to project data into higher dimensional spaces to allow for nonlinear classification. The document concludes by applying an SVM to a gender classification problem based on mobile app usage, using a custom kernel to account for apps and app categories.

1. Introduction to Support Vector Machine
Lucas Xu
September 4, 2012
Lucas Xu Introduction to Support Vector Machine September 4, 2012 1 / 20

2. 1 Classifier
2 Hyper-Plane
3 Convex Optimization
4 Kernel
5 Application
Lucas Xu Introduction to Support Vector Machine September 4, 2012 2 / 20

3. Classifier
Attributes and Class Labels
Training Data
S = (x(1) , y (1) ), · · · , (x(m) , y (m) ) , x(i) ∈ Rd , y (i) ∈ {−1, 1}
Lucas Xu Introduction to Support Vector Machine September 4, 2012 3 / 20

4. Classifier
Umeng Gender Classification Data
user app1 app2 ··· appd gender
user1 1 0 ··· 0 male
user2 0 1 ··· 1 f emale
.
. .
. .
. .. .
. .
.
. . . . . .
usern 1 1 ··· 1 f emale
Each App belongs to one category, ≈ 20 categories.
Categories are mutual exclusive.
Lucas Xu Introduction to Support Vector Machine September 4, 2012 4 / 20

5. Classifier
Umeng Gender Classification Data
S = (x(1) , y (1) ), · · · , (x(m) , y (m) ) , x(i) ∈ Rd , y (i) ∈ {−1, 1}
(i)
xk ∈ {0, 1}, 0 means not installed, 1 means installed on the device
1 ≤ k ≤ d, d 30, 000, about 30,000 apps
y (i) ∈ {male, f emale}
Lucas Xu Introduction to Support Vector Machine September 4, 2012 5 / 20

6. Hyper-Plane
Figure : Hyper Plane
The hyper-plane: wT x + b = 0
Classification function: hw,b (x) = g(wT x + b)
1 if z ≥ 0
g(z) =
−1 otherwise
Lucas Xu Introduction to Support Vector Machine September 4, 2012 6 / 20

7. Hyper-Plane
Functional Margin:
γ (i) = y (i) (wT x(i) + b)
ˆ
Scaling: set constraint normalization condition : w = 1
Geometric Margin:
w T b
γ (i) = y (i) x(i) +
w w
γ (i) should be a large positive number to increase the prediction
confidence.
Lucas Xu Introduction to Support Vector Machine September 4, 2012 7 / 20

8. Hyper-Plane
Definition
The geometry margin of (w, b) with respect to training dataset S:
γ = min γ (i)
i=1,...,m
Lucas Xu Introduction to Support Vector Machine September 4, 2012 8 / 20

9. Hyper-Plane
The optimal margin classifier: (Intuitive)
find a decision boundary that maximizes the margin.
maxγ,w,b γ
s.t. y (i) (wT x(i) + b) ≥ γ, i = 1, ..., m
w = 1.
Lucas Xu Introduction to Support Vector Machine September 4, 2012 9 / 20

10. Hyper-Plane
Normalization Constraint: let function margin γ = 1
ˆ
⇓
1
maxγ,w,b
w
s.t. y (i) (wT x(i) + b) ≥ γ, i = 1, ..., m
⇓
1
maxw,b w 2
2
s.t. y (i) (wT x(i) + b) ≥ 1, i = 1, ..., m
Lucas Xu Introduction to Support Vector Machine September 4, 2012 10 / 20

11. Hyper-Plane
Convex function
Lucas Xu Introduction to Support Vector Machine September 4, 2012 11 / 20

12. Hyper-Plane
Convex function
Convex set
Lucas Xu Introduction to Support Vector Machine September 4, 2012 11 / 20

13. Hyper-Plane
Convex function
Convex set
So-called Quadratic Programming. Their are many software
packages to solve the problem.
Lucas Xu Introduction to Support Vector Machine September 4, 2012 11 / 20

14. Hyper-Plane
Convex function
Convex set
So-called Quadratic Programming. Their are many software
packages to solve the problem.
Basic Ideas for Support Vector Machine DONE !
Lucas Xu Introduction to Support Vector Machine September 4, 2012 11 / 20

15. Hyper-Plane
Convex function
Convex set
So-called Quadratic Programming. Their are many software
packages to solve the problem.
Basic Ideas for Support Vector Machine DONE !
More efficient solution ?
Lucas Xu Introduction to Support Vector Machine September 4, 2012 11 / 20

16. Convex Optimization
Primal Problem:
1
maxw,b w 2
2
s.t. y (i) (wT x(i) + b) ≥ 1, i = 1, ..., m
Lucas Xu Introduction to Support Vector Machine September 4, 2012 12 / 20

17. Convex Optimization
Lagrangian for the original problem:
m
1 2
min max L(w, b, α) = w − αi y (i) (wT x(i) + b) − 1
w,b α:αi ≥0 2
i=1
⇓
Under K.K.T condition, transforms to its Dual problem:
m m
1
max W (α) = αi − y (i) y (j) αi αj x(i) , x(j)
α 2
i=1 i,j=1
s.t. αi ≥ 0, i = 1, ..., m
m
αi y (i) = 0
i=1
Lucas Xu Introduction to Support Vector Machine September 4, 2012 13 / 20

18. Convex Optimization
Solutions:
m
∗
w = αi y (i) x(i)
i=1
maxi:y(i) =−1 w∗T x(i) + mini:y(i) =1 w∗T x(i)
b∗ = −
2
Predict:
g(x) = wT x + b
m T
= αi y (i) x(i) x+b
i=1
m
= αi y (i) x(i) , x + b
i=1
Lucas Xu Introduction to Support Vector Machine September 4, 2012 14 / 20

19. Kernel
For most of αi , αi = 0.
For those αi > 0, (x(i) , y (i) ) are called support vectors
Only needs to compute x(i) , x
(i) (i) (i)
if we can map feature space (x1 , x2 , ...xk ) to another high
(i) (i) (i)
dimension space (z1 , z2 , ...zl ), z = φ(x)
i.e. φ(x(i) , φ(x)
we can easily compute z (i) , z = K(φ( x(i) , x ))
Use a slightly different notation:
K(x, y) = φ(x), φ(y)
Intuitive Explanation: Measure of Similarities
Lucas Xu Introduction to Support Vector Machine September 4, 2012 15 / 20

20. Kernel
Definition
Mercer Kernel: K is positive semi-definite
Lucas Xu Introduction to Support Vector Machine September 4, 2012 16 / 20

21. Kernel
Primitive x, y
Lucas Xu Introduction to Support Vector Machine September 4, 2012 17 / 20

22. Kernel
Primitive x, y
Polynomial ( x, y + 1)d
Lucas Xu Introduction to Support Vector Machine September 4, 2012 17 / 20

23. Kernel
Primitive x, y
Polynomial ( x, y + 1)d
RBF exp(−γ||x − y||2 )
Lucas Xu Introduction to Support Vector Machine September 4, 2012 17 / 20

24. Kernel
Primitive x, y
Polynomial ( x, y + 1)d
RBF exp(−γ||x − y||2 )
Sigmoid tanh(κ x, y + c).
Lucas Xu Introduction to Support Vector Machine September 4, 2012 17 / 20

25. Kernel
Primitive x, y
Polynomial ( x, y + 1)d
RBF exp(−γ||x − y||2 )
String
Lucas Xu Introduction to Support Vector Machine September 4, 2012 17 / 20

26. Kernel
Primitive x, y
Polynomial ( x, y + 1)d
RBF exp(−γ||x − y||2 )
String
Tree
Lucas Xu Introduction to Support Vector Machine September 4, 2012 17 / 20

27. Apply to Umeng Gender Classification
Problem Description
Classify the gender of a user based on apps (s)he installed and
categories of apps.
Kernel Design
m
K(x, y) = φ(xi , yj )
i,j=0

 (1 + w)xi yj if i = j
φ(xi , yj ) = xi yj if i = j but the same category
0 if not the same category

w ≥ 0 , the extra weight if two users have installed the same app.
default to 1.0
Experiment Result
Lucas Xu Introduction to Support Vector Machine September 4, 2012 18 / 20

28. Apply to Umeng Gender Classification
 
x1
 x2 
 
 . 
 . 
.
xm
⇓
 
w · x1
 w · x2 

 . 

 . 
. 

w · xm 
 
 c1 
 
 c2 
 
 . 
 . . 
c20
ci counts the number of apps belonging to category i
Lucas Xu Introduction to Support Vector Machine September 4, 2012 19 / 20

29. references
Book: Christopher Bishop – PRML Chapter 7: Section 7.1
Slides: Andrew Moore – Support Vector Machines
Video: Bernhard Scholkopf – Kernel Methods
Video: Liva Ralaivola – Introduction to Kernel Methods
Video: Colin Campbell – Introduction to Support Vector Machines
Video: Alex Smola – Kernel Methods and Support Vector
Machines
Video: Partha Niyogi – Introduction to Kernel Methods
Many more videos on kernel-related topics here
http://www.seas.harvard.edu/courses/cs281/
Lucas Xu Introduction to Support Vector Machine September 4, 2012 20 / 20