Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Mining Exercise Course, Non-Linear Classification

28 views

Published on

A lecture to the bachelor students of Data Mining about non-linear classification using Support Vector Machine (SVM)

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Data Mining Exercise Course, Non-Linear Classification

  1. 1. Research Group for Pattern Recognition Institute for Vision and Graphics University of Siegen, Germany Data Mining Exercise Course Non-Linear Classification Zeyd Boukhers Katowice, June 2nd, 2016
  2. 2.  An Intuitive Understanding  Example of Non-Linear Problems  Possible Solutions  Examples implemented in R Outline
  3. 3. An Intuitive Understanding Linearly Separable Non-Linearly Separable We have seen that for an SVM learning a linear classifier: , Then the SVM can be formulated to learn a linear classifier: by solving an optimization problem over
  4. 4. An Intuitive Understanding Linearly Separable Non-Linearly Separable
  5. 5. An Intuitive Understanding Linearly Separable Non-Linearly Separable
  6. 6. An Intuitive Understanding Linearly Separable Non-Linearly Separable
  7. 7. An Intuitive Understanding
  8. 8. Example of Non-Linear Problems Interference Inclusion Multiple-Inclusion Indistinctive
  9. 9. Solution 1: Use polar coordinats 2 / 30 < 0 > 0 θ r Data is linearly separable in polar coordinates
  10. 10. Solution 2: Map data into higher dimension 2 / 30 Data is linearly separable in 3D  The problem can still be solved by a linear classifier
  11. 11. Solutions Learn classifier linear in for : Where is a feature map.
  12. 12. Solutions Learn classifier linear in for : Where is a feature map.
  13. 13. Dual Classifier in transformed feature space Classifier:  Learning:  Subject to:
  14. 14. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2)
  15. 15. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into a new feature space, where a seperating hyperplan can be found.
  16. 16. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into a new feature space, where a seperating hyperplan can be found. Let consider the following mapping function:
  17. 17. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into a new feature space, where a seperating hyperplan can be found. Let consider the following mapping function: Blue class vectors become: (1,1), (-1,1), (-1,-1), (1,-1) No change Red class vectors become: (8,10), (10,8), (12,10), (10,12)
  18. 18. Example 1 Linearly Sperable S1 S2 S3Bias = 1
  19. 19. Example 1 We need to find 2 parameters and based on the following 3 linear equations:
  20. 20. Example 1 Then, we get: The hyper plan is given by:
  21. 21. Example 1: Implementation in R Learning: > library("e1071") > x <- array(data = c(-2,-1,-1,0,0,1,1,2,0,-1,1,-2,2,- 1,1,0),dim=c(8,2)) > y <- factor(c(-1,1,1,-1,-1,1,1,-1)) > for ( i in 1:8) { + if(sqrt(x[i,1]^2+ x[i,2]^2) >= 2){ + x[i,1] <- (6-x[i,1]+(x[i,1]-x[i,2])^2) + x[i,2] <- (6-x[i,2]+(x[i,1]-x[i,2])^2) + } else { + x[i,1] <- x[i,1] + x[i,2] <- x[i,2] + } + } > model <- svm(x,y,kernel="linear",type="C-classification")
  22. 22. Example 1: Implementation in R Prediction: > z <- array(data = c(0,0),dim=c(1,2)) + if(sqrt(z[1,1]^2+ z[1,2]^2) >= 2){ + z[1,1] <- (6-z[1,1]+(z[1,1]-z[1,2])^2) + z[1,2] <- (6-z[1,2]+(z[1,1]-z[1,2])^2) + } else { + z[1,1] <- z[1,1] + z[1,2] <- z[1,2] + } > predict(model,z)
  23. 23. Example 2 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into higher dimension feature space, where a seperating hyperplan can be found Hint:
  24. 24. Example 2: Implementation in R Learning: > library("e1071") > x <- array(data = c(-2,-1,-1,0,0,1,1,2,0,-1,1,-2,2,- 1,1,0),dim=c(8,2)) > y <- factor(c(-1,1,1,-1,-1,1,1,-1)) > x2 <- array(0,dim=c(8,3)) > for ( i in 1:8) { + x2[i,1] <- x[i,1]^2 + x2[i,2] <- x[i,2]^2 + x2[i,3] <- sqrt(2)*x[i,1]*x[i,2] + } > model <- svm(x2,y,kernel="linear",type="C-classification")
  25. 25. Example 1: Implementation in R Prediction: > z <- array(data = c(0,0),dim=c(1,2)) > z2 <- array(0,dim=c(1,3)) > z2[1,1] <- z[1,1]^2 > z2[1,2] <- z[1,2]^2 > z2[1,3] <- sqrt(2)*z[1,1]*z[1,2] > predict(model,z2)
  26. 26. Kernel Trick this is known as a Kernel Then the classifier becomes: And the learning becomes: Subject to The classifier can be learnt and applied without explicitly computing It requires More about SVM-Kernels: http://www.robots.ox.ac.uk/~az/lectures/ml/lect3.pdf
  27. 27. Example Kernels •Linear Kernel: •Polynomial Kernel: •Gaussian Kernel (RBF):
  28. 28. Data Mining Exercise Course Non-Linear Classification Questions?

×