Support Vector Machine Classifiers, Classification and Prediction

1. Classification and Prediction

2. › Learning Task
– Given: Expression profiles of leukemia patients and healthy
persons.
– Compute: A model distinguishing if a person has leukemia from
expression data.
› Classification Task
– Given: Expression profile of a new patient + a learned model
– Determine: If a patient has leukemia or not.

3.  Often high dimension of data.
 Hard to put up simple rules.
 Amount of data.
 Need automated ways to deal with the data.
 Use computers – data processing, statistical analysis, try to learn
patterns from the data (Machine Learning)

4. o Binary Classification problem
o The data above the red line
belongs to class ‘x’
o The data below the red line
belongs to class ‘o’
 Examples: SVM, Probabilistic
Classifier
x
x
x
x
xx
x
x
x
x
o
o
o
o
o
o
o
o
o o
o
o
o

5. o Classification of high-dimensional
data sets
o No need for feature selection
o Quadratic programming problem
o Finds an optimal solution.
o Most successful current text
classification method

6. o Lots of possible linear separator
o Select one that maximizes the margin!
o Separator depends only on a small
number of training examples.
f(x) =-1
=+1

7. Var1
Var2
Margin
Width
Margin
Width

8. Var1
Var2
Margin
Width
Support Vectors

9. Var1
Var2

10. 0
x
 Datasets that are linearly separable work out
great
 But what are we going to do if the dataset is just too
hard?
0
x

11. 0
x
 How about … mapping data to a higher-dimensional
space:
0
x

12. › Mapping to transform training data into a higher
dimension
› With the new dimension, it searches for the linear
optimal separating hyperplane.
› With an appropriate nonlinear mapping, data from two
classes can always be separated by a hyperplane.
› SVM finds this hyperplane using support vectors.

13. Φ: x → φ(x)

14. 𝑥2
2
𝑥1
2
√2𝑥1 𝑥2
𝑥1
𝑥2

15. › Impossible to perfectly separates the two classes

16. Space
𝑤 𝑤 → 𝑝𝑒𝑟𝑝𝑒𝑛𝑑𝑖𝑐𝑢𝑙𝑎𝑟 𝑡𝑜 𝑟𝑜𝑎𝑑
𝑢 𝑢 → 𝑢𝑛𝑘𝑛𝑜𝑤𝑛 𝑣𝑒𝑐𝑡𝑜𝑟
𝑤. 𝑢 ≥ 𝑐 𝑤. 𝑢 → 𝑝𝑟𝑜𝑗𝑒𝑐𝑡𝑖𝑜𝑛 𝑜𝑓 𝑢 𝑡𝑜 𝑤
𝑤. 𝑢 + 𝑏 ≥ 0 then +

17. 𝑤
𝑢
𝑤. 𝑥+ + 𝑏 ≥ 1
𝑤. 𝑥− + 𝑏 ≤ −1
𝑤. 𝑢 + 𝑏 ≥ 0 --- (1)
𝐼𝑛𝑡𝑟𝑜𝑑𝑢𝑐𝑒 𝑦𝑖 such that,
𝑦𝑖 = +1 𝐹𝑜𝑟 + 𝑠𝑎𝑚𝑝𝑙𝑒𝑠
𝑦𝑖 = −1 𝐹𝑜𝑟 − 𝑠𝑎𝑚𝑝𝑙𝑒𝑠

18. 𝑤
𝑢
𝑦𝑖 ( 𝑤. 𝑥𝑖 + 𝑏) ≥ 1
𝑦𝑖( 𝑤. 𝑥𝑖 + 𝑏) ≤ (−1) 𝑦𝑖
⇒ 𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 ≥ 1
𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1 ≥ 0
𝑁𝑜𝑤,
𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1 = 0, for Gutter --- (2)

19. 𝑥+𝑥−
𝑥+ − 𝑥−
𝑤𝑖𝑑𝑡ℎ = 𝑥+ − 𝑥− ∗
𝑤
𝑤
=
2
𝑤
𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1 = 0
𝑤. 𝑥+ = 1 − 𝑏
𝑦𝑖 𝑤. 𝑥𝑖 + 𝑏 − 1 = 0
𝑤. 𝑥− = 1 + 𝑏
𝑀𝐴𝑋
2
𝑤
= 𝑀𝐴𝑋
1
𝑤
= 𝑀𝐼𝑁 𝑤 = 𝑀𝐼𝑁
1
2
𝑤 2