Successfully reported this slideshow.
Upcoming SlideShare
×

# Data Mining Exercise Course, Non-Linear Classification

28 views

Published on

A lecture to the bachelor students of Data Mining about non-linear classification using Support Vector Machine (SVM)

Published in: Data & Analytics
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### Data Mining Exercise Course, Non-Linear Classification

1. 1. Research Group for Pattern Recognition Institute for Vision and Graphics University of Siegen, Germany Data Mining Exercise Course Non-Linear Classification Zeyd Boukhers Katowice, June 2nd, 2016
2. 2.  An Intuitive Understanding  Example of Non-Linear Problems  Possible Solutions  Examples implemented in R Outline
3. 3. An Intuitive Understanding Linearly Separable Non-Linearly Separable We have seen that for an SVM learning a linear classifier: , Then the SVM can be formulated to learn a linear classifier: by solving an optimization problem over
4. 4. An Intuitive Understanding Linearly Separable Non-Linearly Separable
5. 5. An Intuitive Understanding Linearly Separable Non-Linearly Separable
6. 6. An Intuitive Understanding Linearly Separable Non-Linearly Separable
7. 7. An Intuitive Understanding
8. 8. Example of Non-Linear Problems Interference Inclusion Multiple-Inclusion Indistinctive
9. 9. Solution 1: Use polar coordinats 2 / 30 < 0 > 0 θ r Data is linearly separable in polar coordinates
10. 10. Solution 2: Map data into higher dimension 2 / 30 Data is linearly separable in 3D  The problem can still be solved by a linear classifier
11. 11. Solutions Learn classifier linear in for : Where is a feature map.
12. 12. Solutions Learn classifier linear in for : Where is a feature map.
13. 13. Dual Classifier in transformed feature space Classifier:  Learning:  Subject to:
14. 14. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2)
15. 15. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into a new feature space, where a seperating hyperplan can be found.
16. 16. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into a new feature space, where a seperating hyperplan can be found. Let consider the following mapping function:
17. 17. Example 1 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into a new feature space, where a seperating hyperplan can be found. Let consider the following mapping function: Blue class vectors become: (1,1), (-1,1), (-1,-1), (1,-1) No change Red class vectors become: (8,10), (10,8), (12,10), (10,12)
18. 18. Example 1 Linearly Sperable S1 S2 S3Bias = 1
19. 19. Example 1 We need to find 2 parameters and based on the following 3 linear equations:
20. 20. Example 1 Then, we get: The hyper plan is given by:
21. 21. Example 1: Implementation in R Learning: > library("e1071") > x <- array(data = c(-2,-1,-1,0,0,1,1,2,0,-1,1,-2,2,- 1,1,0),dim=c(8,2)) > y <- factor(c(-1,1,1,-1,-1,1,1,-1)) > for ( i in 1:8) { + if(sqrt(x[i,1]^2+ x[i,2]^2) >= 2){ + x[i,1] <- (6-x[i,1]+(x[i,1]-x[i,2])^2) + x[i,2] <- (6-x[i,2]+(x[i,1]-x[i,2])^2) + } else { + x[i,1] <- x[i,1] + x[i,2] <- x[i,2] + } + } > model <- svm(x,y,kernel="linear",type="C-classification")
22. 22. Example 1: Implementation in R Prediction: > z <- array(data = c(0,0),dim=c(1,2)) + if(sqrt(z[1,1]^2+ z[1,2]^2) >= 2){ + z[1,1] <- (6-z[1,1]+(z[1,1]-z[1,2])^2) + z[1,2] <- (6-z[1,2]+(z[1,1]-z[1,2])^2) + } else { + z[1,1] <- z[1,1] + z[1,2] <- z[1,2] + } > predict(model,z)
23. 23. Example 2 Blue class vectors: (1,1), (-1,1), (-1,-1), (1,-1) Red class vectors: (2,0), (0,2), (-2,0), (0,-2) Transform these data into higher dimension feature space, where a seperating hyperplan can be found Hint:
24. 24. Example 2: Implementation in R Learning: > library("e1071") > x <- array(data = c(-2,-1,-1,0,0,1,1,2,0,-1,1,-2,2,- 1,1,0),dim=c(8,2)) > y <- factor(c(-1,1,1,-1,-1,1,1,-1)) > x2 <- array(0,dim=c(8,3)) > for ( i in 1:8) { + x2[i,1] <- x[i,1]^2 + x2[i,2] <- x[i,2]^2 + x2[i,3] <- sqrt(2)*x[i,1]*x[i,2] + } > model <- svm(x2,y,kernel="linear",type="C-classification")
25. 25. Example 1: Implementation in R Prediction: > z <- array(data = c(0,0),dim=c(1,2)) > z2 <- array(0,dim=c(1,3)) > z2[1,1] <- z[1,1]^2 > z2[1,2] <- z[1,2]^2 > z2[1,3] <- sqrt(2)*z[1,1]*z[1,2] > predict(model,z2)
26. 26. Kernel Trick this is known as a Kernel Then the classifier becomes: And the learning becomes: Subject to The classifier can be learnt and applied without explicitly computing It requires More about SVM-Kernels: http://www.robots.ox.ac.uk/~az/lectures/ml/lect3.pdf
27. 27. Example Kernels •Linear Kernel: •Polynomial Kernel: •Gaussian Kernel (RBF):
28. 28. Data Mining Exercise Course Non-Linear Classification Questions?