Soft margin derivation

1. SVM – Soft margin hyperplanes
Sarith Divakar M
LBS College of Engineering, Kasaragod
sarith@lbscek.ac.in

2. RECAP OF SESSION 1
2
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

3. Width of the street
𝑊𝑖𝑑𝑡ℎ = (𝑥+ - 𝑥− ) .
𝑤
||𝑤||
For +ve samples 𝑦𝑖=1 and –ve samples 𝑦𝑖 = -1
𝑦𝑖(𝑤. 𝑥𝑖 + b) −1 = 0 3
1.(𝑤. 𝑥𝑖 + b) −1 = 0
𝑤. 𝑥𝑖 = 1-b
i
-1.(𝑤. 𝑥𝑖 + b) −1 = 0
𝑤. 𝑥𝑖 = -1-b
ii
𝑊𝑖𝑑𝑡ℎ = (1-b – (-1-b)) .
1
||𝑤||
𝑥+
𝑥−
𝑥+ - 𝑥−
𝑊𝑖𝑑𝑡ℎ =
2
||𝑤||
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
3

4. Maximize Width of the street
𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 =
2
||𝑤||
𝑥+
𝑥−
𝑥+ + 𝑥−
𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 =
1
||𝑤||
𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 = ||𝑤||
𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 =
1
2
||𝑤||
2
4
𝑦𝑖(𝑤. 𝑥𝑖 + b) −1 = 0
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
4

5. Optimization using Lagrange multipliers
Expression: 𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 =
1
2
||𝑤||
2
Constraint: 𝑦𝑖(𝑤. 𝑥𝑖 + b) −1 = 0
Primal problem:
L=
1
2
||𝑤||
2
- 𝛼𝑖 [𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1]
𝜕𝐿
𝜕𝑤
= 𝑤- 𝛼𝑖 𝑦𝑖 𝑥𝑖 = 0
𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 6
𝜕𝐿
𝜕𝑏
= - 𝛼𝑖 𝑦𝑖 = 0
𝛼𝑖 𝑦𝑖 = 0 7
5
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
5
𝑥+
𝑥−
𝑥+ + 𝑥−

6. Optimization using Lagrange multipliers
7
L=
1
2
||𝑤||
2
- 𝛼𝑖 [𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1]
𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖
5
6
𝛼𝑖 𝑦𝑖 = 0
L=
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- 𝛼𝑖 [𝑦𝑖( 𝛼𝑗 𝑦𝑗 𝑥𝑗 . 𝑥𝑖 + b) − 1]
L=
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗) − 𝛼𝑖 𝑦𝑖 b + 𝛼𝑖
= 0
L= 𝛼𝑖
+
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗)
L= 𝛼𝑖 −
1
2
𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗)
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
6

7. SVM Classifier
𝜙 𝛼 = 𝛼
𝑖
−
1
2
𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗)
Compute 𝑤 and b
Dual Problem: Find vector 𝛼 which maximizes
Subject to
𝛼𝑖 𝑦𝑖 = 0
SVM Classifier Function:
𝑏 =
1
2
(𝑚𝑖𝑛𝑖:𝑦 𝑖=+1(𝑤. 𝑥𝑖) + 𝑚𝑎𝑥𝑖:𝑦 𝑖=−1(𝑤. 𝑥𝑖))
𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖
f( 𝑥) =(𝑤. 𝑥) - b
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
7
𝛼𝑖 ≥ 0

8. 8
SVM Classifier Function: f( 𝑥) =(𝑤. 𝑥) - b
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
= −
1
2
, −
1
2
. (𝑥1, 𝑥2) +
5
2
= −
1
2
𝑥1 −
1
2
𝑥2 +
5
2
= −
1
2
[𝑥1 + 𝑥2−5]
Equation of maximal margin line
f( 𝑥) = 0 𝑥1 + 𝑥2 = 5
0 1 2 3 4 5 6
1
2
3
4
5
6
Sampl
e
F1 F2 Class
1 2 1 +1
2 4 3 -1

9. Soft margin hyperplanes
Department of Computer Science and Engineering, LBS College
of Engineering, Kasaragod
9

10. 10
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

11. Soft margin hyperplanes
11
Outside the margin or on the margin
Inside the margin correctly classified
Inside the margin misclassified
Soft Error 𝐶 𝜉𝑖
𝜉𝑖 = 0
0 < 𝜉𝑖 < 1
𝜉𝑖 > 1
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
𝑤
𝑢
0 < 𝜉𝑖 < 1
𝜉𝑖 > 1
𝜉𝑖 = 0
𝜉𝑖 = 0

12. Optimization using Lagrange multipliers
12
Primal problem:
Expression: 𝑀𝑖𝑛𝑖𝑚𝑖𝑧𝑒 =
1
2
||𝑤||
2
+ C 𝜉𝑖
Constraint: 𝑦𝑖(𝑤. 𝑥𝑖 + b)≥ 1 − 𝜉𝑖
𝜉𝑖 ≥ 0
L=
1
2
||𝑤||
2
+ C 𝜉𝑖 - 𝛼𝑖 𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1 + 𝜉𝑖 − 𝜇𝑖 𝜉𝑖
𝛼𝑖 ≥ 0, 𝜇𝑖 ≥ 0
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
𝑤
𝑢
0 < 𝜉𝑖 < 1
𝜉𝑖 > 1
𝜉𝑖 = 0
𝜉𝑖 = 0

13. 13
𝜕𝐿
𝜕𝑤
= 𝑤- 𝛼𝑖 𝑦𝑖 𝑥𝑖 = 0 𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖
𝜕𝐿
𝜕𝑏
= - 𝛼𝑖 𝑦𝑖 = 0 𝛼𝑖 𝑦𝑖 = 0
𝜕𝐿
𝜕𝜉 𝑖
= C - 𝛼𝑖 - 𝜇𝑖= 0 𝜇𝑖= C -𝛼𝑖 𝜇𝑖≥ 0 ⇒ 𝛼𝑖 ≤ 𝐶
L=
1
2
||𝑤||
2
+ C 𝜉𝑖 - 𝛼𝑖 𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1 + 𝜉𝑖 − 𝜇𝑖 𝜉𝑖
Optimization using Lagrange multipliers
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

14. Optimization using Lagrange multipliers
L=
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 ) + C 𝜉𝑖 − 𝛼𝑖 𝑦𝑖( 𝛼𝑗 𝑦𝑗 𝑥𝑗 . 𝑥𝑖 + b) − 1 + 𝜉𝑖 - 𝐶𝜉𝑖 + 𝛼𝑖 𝜉𝑖
L=
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 ) + C 𝜉𝑖- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗) − 𝛼𝑖 𝑦𝑖 b + 𝛼𝑖
− 𝛼𝑖 𝜉𝑖 −C 𝜉𝑖 + 𝛼𝑖 𝜉𝑖
= 0
L= 𝛼𝑖
+
1
2
( 𝛼𝑖 𝑦𝑖 𝑥𝑖 ).( 𝛼𝑗 𝑦𝑗 𝑥𝑗 )- ( 𝛼𝑖 𝑦𝑖 𝑥𝑖). ( 𝛼𝑗 𝑦𝑗 𝑥𝑗)
L= 𝛼𝑖
−
1
2
𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗)
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
14
L=
1
2
||𝑤||
2
+ C 𝜉𝑖 - 𝛼𝑖 𝑦𝑖(𝑤. 𝑥𝑖 + b) − 1 + 𝜉𝑖 − 𝜇𝑖 𝜉𝑖
𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖 𝛼𝑖 𝑦𝑖 = 0 𝜇𝑖= C -𝛼𝑖

15. SVM Soft Margin Classifier
𝜙 𝛼 = 𝛼
𝑖
−
1
2
𝛼𝑖 𝑦𝑖 𝛼𝑗 𝑦𝑗 (𝑥𝑖. 𝑥𝑗)
Compute 𝑤 and b
Dual Problem: Find vector 𝛼 which maximizes
Subject to
𝛼𝑖 𝑦𝑖 = 0
SVM Classifier Function:
𝑏 =
1
2
(𝑚𝑖𝑛𝑖:𝑦 𝑖=+1(𝑤. 𝑥𝑖) + 𝑚𝑎𝑥𝑖:𝑦 𝑖=−1(𝑤. 𝑥𝑖))
𝑤 = 𝛼𝑖 𝑦𝑖 𝑥𝑖
f( 𝑥) =(𝑤. 𝑥) - b
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
15
0 ≤ 𝛼𝑖 ≤ 𝐶

16. Hyperparamter Tuning
16
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

17. C=1
17
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

18. C=2000
18
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

19. C=78
19
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

20. Hyperparameter Tuning
Parameter Range Step Length
C [20, 211] 21
20
Parameter Cherkassky et. al. [11]
C max( 𝑦 + 3𝜎 , | 𝑦 − 3𝜎|)
Heuristic Methods
Search Space
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod

21. Reference
1. Sudheep Elayidom, M. Data mining and warehousing, Cengage.
2. V. Vapnik, The Nature of Statistical Learning Theory, Springer, 1995
3. Vapnik, V. Statistical Learning Theory. John Wiley & Sons. Inc., New York, 1998
4. B. Schölkopf and A. J. Smola, Learning with Kernels, MIT Press, Cambridge, MA, 2002
5. Davide, M. and Simon, H. Advances in Kernel Methods, 1999, 226-227.
6. Jaiwei Han, Micheline Kamber, “Data Mining Concepts and Techniques”, Elsevier, 2006.
7. Pang-Ning Tan, Michael Steinbach, “Introduction to Data Mining”, Addison Wesley, 2006.
8. Dunham M H, “Data Mining: Introductory and Advanced Topics”, Pearson Education, New
Delhi, 2003.
9. Mehmed Kantardzic, “Data Mining Concepts, Methods and Algorithms”, John Wiley and Sons,
USA, 2003.
10. Sarith Divakar M, Sudheep Elayidom M, Rajesh R, “An efficient approach for crop yield
forecasting using machine learning techniques based on normalized difference vegetation
index and climatic indices”, JARDCS, Vol. 10, 15-Special Issue, 2018
11. Cherkassky, V. and Ma, Y. Selection of meta-parameters for support vector regression.
International Con-ference on Artificial Neural Networks, 2002, 687-693
Department of Computer Science and Engineering, LBS College of Engineering, Kasaragod
21