1. Classification Michael I. Jordan University of California, Berkeley

16. Intuitive Picture of the Problem Class1 Class2

42. Linearly Separable Data Class1 Class2 Linear Decision boundary

43. Nonlinearly Separable Data Non Linear Classifier Class1 Class2

44. Which Separating Hyperplane to Use? x 1 x 2

45. Maximizing the Margin x 1 x 2 Margin Width Margin Width Select the separating hyperplane that maximizes the margin

46. Support Vectors x 1 x 2 Margin Width Support Vectors

47. Setting Up the Optimization Problem x 1 x 2 The maximum margin can be characterized as a solution to an optimization problem:

50. Nonlinearly Separable Data Var 1 Var 2 Introduce slack variables Allow some instances to fall within the margin, but penalize them

51. Formulating the Optimization Problem Var 1 Var 2 Constraints becomes : Objective function penalizes for misclassified instances and those within the margin C trades-off margin width and misclassifications

53. Robustness of Soft vs Hard Margin SVMs Var 1 Var 2 Soft Margin SVN Hard Margin SVN Var 1 Var 2  i

54. Disadvantages of Linear Decision Surfaces Var 1 Var 2

55. Advantages of Nonlinear Surfaces Var 1 Var 2

56. Linear Classifiers in High-Dimensional Spaces Var 1 Var 2 Constructed Feature 1 Find function  (x) to map to a different space Constructed Feature 2

60. Example X=[x z]  (X)=[x 2 z 2 xz] f(x) = sign(w 1 x 2 +w 2 z 2 +w 3 xz + b) w T  (x)+b=0

61. Example  (X 1 ) T  (X 2 )= [x 1 2 z 1 2 2 1/2 x 1 z 1 ] [x 2 2 z 2 2 2 1/2 x 2 z 2 ] T = x 1 2 z 1 2 + x 2 2 z 2 2 + 2 x 1 z 1 x 2 z 2 = (x 1 z 1 + x 2 z 2 ) 2 = (X 1 T X 2 ) 2 X 1 =[x 1 z 1 ] X 2 =[x 2 z 2 ]  (X 1 )=[x 1 2 z 1 2 2 1/2 x 1 z 1 ]  (X 2 )=[x 2 2 z 2 2 2 1/2 x 2 z 2 ] Expensive! O(d 2 ) Efficient! O(d)

66. Gaussian Kernel: Example A hyperplane in some space

69. Kernel-Based Learning K Data Embedding Linear algorithm Kernel design Kernel algorithm

73. [From Tom Mitchell’s slides]

74. Spatial example: recursive binary splits + + + + + + + + + + + +

75. Spatial example: recursive binary splits + + + + + + + + + + + +

76. Spatial example: recursive binary splits + + + + + + + + + + + +

77. Spatial example: recursive binary splits + + + + + + + + + + + +

78. Spatial example: recursive binary splits + + + + + + + + + + + + p m =5/6 Once regions are chosen class probabilities are easy to calculate

80. Overfitting and pruning + + + + + + + + + + + + L: 0-1 loss min T  i  L(x i ) +  |T| then choose  with CV increase 