Lecture 1

Seminar in Advanced Machine Learning Rong Jin

Course Description ,[object Object],[object Object],[object Object],[object Object]

Course Organization ,[object Object],[object Object],[object Object],[object Object],[object Object]

Course Organization ,[object Object],[object Object],[object Object],[object Object],Let’s have fun!

Convex Programming and Classification Problems Rong Jin

Outline ,[object Object],[object Object],[object Object]

Support Vector Machine (SVM) ,[object Object],[object Object],D = f ( x 1 ; y 1 ) ; ( x 2 ; y 2 ) ; : : : ; ( x n ; y n ) g w h e r e x i 2 R d ; y i 2 f ¡ 1 ; + 1 g m i n k w k 2 2 s . t . y i ( w > x i ¡ b ) ¸ 1 ; i = 1 ; 2 ; : : : ; n

SVM: Robust Optimization ,[object Object],[object Object],[object Object],½

SVM: Robust Optimization ,[object Object],m a x ½ s . t . 8 i = 1 ; 2 ; : : : ; n j x ¡ x i j 2 · ½ ! y i ( w > x ¡ b ) ¸ 1

SVM: Robust Optimization j x ¡ x i j 2 · ½ ! y i ( w > x ¡ b ) ¸ 1 y i ( w > x i ¡ b ) ¡ ½ k w k 2 ¸ 1 m a x ½ s . t . 8 i = 1 ; 2 ; : : : ; n y i ( w > x i ¡ b ) ¡ ½ k w k 2 ¸ 1

Robust Optimization ,[object Object],[object Object],[object Object],m i n c > x ; : a > i x · b i ; i = 1 ; 2 ; : : : ; n m i n c > x ; : 8 a i 2 E i ; a > i x · b i ; i = 1 ; 2 ; : : : ; n E i = f a i j ( a i ¡ ^ a i ) > § ¡ 1 i ( a i ¡ ^ a i ) · 1 g

Minimax Probability Machine (MPM) ,[object Object],Negative Class Positive Class x > a · b N e g a t i v e c l a s s : x ¡ » N ( ¹ x ¡ ; § ¡ ) x > a ¸ b P o s i t i v e c l a s s : x + » N ( ¹ x + ; § + ) x > a = b

Minimax Probability Machine (MPM) w h e r e x ¡ » N ( ¹ x ¡ ; § ¡ ) a n d x + » N ( ¹ x + ; § + ) x > a · b x > a ¸ b Negative Class Positive Class m i n m a x ( ² + ; ² ¡ ) s . t . P r ( x > + a · b ) = 1 ¡ ² + P r ( x > ¡ a ¸ b ) = 1 ¡ ² ¡

Minimax Probability Machine (MPM) w h e r e x ¡ » N ( ¹ x ¡ ; § ¡ ) a n d x + » N ( ¹ x + ; § + ) x > a · b x > a ¸ b Negative Class Positive Class m i n ² s . t . P r ( x > + a · b ) ¸ 1 ¡ ² P r ( x > ¡ a ¸ b ) ¸ 1 ¡ ²

Minimax Probability Machine (MPM) ,[object Object],Negative Class Positive Class N ( ¹ x ; § ) P r ( x > a · b ) · 1 ¡ ² ¹ x > a + · k § a k 2 · b w h e r e · = © ¡ 1 ( 1 ¡ ² ) x > a · b x > a ¸ b

Minimax Probability Machine (MPM) m a x · s . t . x > + a + · k § + a k 2 2 · b x > ¡ a ¡ · k § ¡ a k 2 ¸ b Second order cone constraints m i n ® + ¯ s . t . a > ( x ¡ ¡ x + ) = 1 ® ¸ k § + a k 2 2 ; ¯ ¸ k § ¡ a k 2 2

Second Order Cone Programming (SOCP) x o ¸ p x 2 1 + x 2 2 ® ¸ k § ¡ a k 2 y = § ¡ a ® ¸ k y k 2 z 2 Q Ã ! z º Q 0 C o n e : Q = f z j z 0 ¸ k ¹ z k 2 g

Second Order Cone Programming (SOCP) SOCP LP Generalize the inequality definition

Minimax Probability Machine (MPM) m i n ² s . t . P r ( x > + a · b ) ¸ 1 ¡ ² P r ( x > ¡ a ¸ b ) ¸ 1 ¡ ² w h e r e x ¡ » N ( ¹ x ¡ ; § ¡ ) a n d x + » N ( ¹ x + ; § + ) m i n ² s . t . i n f x + » ( ¹ x + ; § + ) P r ( x > + a · b ) ¸ 1 ¡ ² i n f x ¡ » ( ¹ x ¡ ; § ¡ ) P r ( x > ¡ a ¸ b ) ¸ 1 ¡ ²

MPM ,[object Object],i n f x » ( ¹ x ; § ) P r ( x > a · b ) = ( b ¡ ¹ x > a ) 2 + ( b ¡ ¹ x > a ) 2 + + a > § a w h e r e [ x ] + o u t p u t s 0 w h e n x < 0 a n d x w h e n x ¸ 0 . m i n ® + ¯ s . t . a > ( x ¡ ¡ x + ) = 1 ® ¸ k § + a k 2 2 ; ¯ ¸ k § ¡ a k 2 2

Pattern Invariance In Images Translation Rotation Shear

Learning from Invariance Trans. Á 1 Á 2

Incorporating Invariance Trans. ,[object Object],[object Object],Infinite number of examples T ( x ; µ = 0 ) = x x ( µ ) = T ( x ; µ ) : R d £ R ! R d m i n k w k 2 2 s . t . 8 µ 2 R ; i = 1 ; 2 ; : : : ; n y i ( w > x i ( µ ) ¡ b ) ¸ 1

Taylor Approximation of Invariance ,[object Object]

Polynomial Approximation ,[object Object],8 µ 2 R : y w > x ( µ ) ¡ 1 ¸ 0

Non-Negative Polynomials (I) ,[object Object],[object Object],9 P º 0 ; s . t . p ( µ ) = µ > P µ

Semidefinite Programming Machines A j := g 1, j g i , j g m , j B := 1 1 1 1 G 1, j G i , j G m , j Semi-definite programming 1 0 0 0 1 0 0 0 1 0 0 0

Semidefinite Programming (SDP) SDP LP Generalize the inequality definition

Beyond Convex Programming ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Example: MAXCUT Problem ,[object Object],m i n x > Q x s . t . x i 2 f ¡ 1 ; + 1 g m i n x > Q x s . t . x 2 i = 1

LMI Relaxation x 2 i = 1 X = x x > X i ; i = 1 m i n x > Q x s . t . x i 2 f ¡ 1 ; + 1 g m i n X i ; j Q i ; j X i ; j s . t . X i ; i = 1 X º 0 ; r a n k ( X ) = 1 m i n X i ; j Q i ; j X i ; j s . t . X i ; i = 1 ; X º 0

How Good is the Approximation? ,[object Object],d ¤ = m i n x > Q x s . t . x i 2 f ¡ 1 ; + 1 g g ¤ = m i n X i ; j Q i ; j X i ; j s . t . X i ; i = 1 ; X º 0 1 ¸ g ¤ d ¤ ¸ 2 ¼ = 0 : 6 3 6 6

What you should learn ? ,[object Object],[object Object],[object Object],[object Object],[object Object]

Lecture 1

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Lecture 1

Similar to Lecture 1 (20)

More from butest

More from butest (20)

Lecture 1

Editor's Notes