Support Vector Machines

Advanced Computing Seminar Data Mining and Its Industrial Applications — Chapter 8 — Support Vector Machines

Zhongzhi Shi, Markus Stumptner, Yalei Hao, Gerald Quirchmayr

Knowledge and Software Engineering Lab

Advanced Computing Research Centre

School of Computer and Information Science

University of South Australia

Outline

Introduction

Support Vector Machine

Non-linear Classification

SVM and PAC

Applications

Summary

History

SVM is a classifier derived from statistical learning theory by Vapnik and Chervonenkis

SVMs introduced by Boser, Guyon, Vapnik in COLT-92

Initially popularized in the NIPS community, now an important and active field of all Machine Learning research.

Special issues of Machine Learning Journal, and Journal of Machine Learning Research.

What is SVM?

SVMs are learning systems that

use a hypothesis space of linear functions

in a high dimensional feature space — Kernel function

trained with a learning algorithm from optimization theory — Lagrange

Implements a learning bias derived from statistical learning theory — Generalisation SVM is a classifier derived from statistical learning theory by Vapnik and Chervonenkis

Maximum Margin f x  y est denotes +1 denotes -1 f ( x , w ,b ) = sign( w . x - b ) The maximum margin linear classifier is the linear classifier with the maximum margin. This is the simplest kind of SVM (Called an LSVM) Linear SVM Copyright © 2001, 2003, Andrew W. Moore

Model of Linear Classification

Binary classification is frequently performed by using a real-valued hypothesis function:

The input x is assigned to the positive class, if

Otherwise to the negative class.

The concept of Hyperplane

For a binary linear separable training set, we can find at least a hyperplane (w,b) which divides the space into two half spaces.

The definition of hyperplane

Tuning the Hyperplane (w,b)

The Perceptron Algorithm

Proposed by Frank Rosenblatt in 1956

Preliminary definition

The functional margin of an example (x i ,y i )

implies correct classification of (x i ,y i )

The Perceptron Algorithm

The number of mistakes is at most

The Geometric margin ->

The Euclidean distance of an example (x i ,y i ) from the decision boundary

The Geometric margin

The margin of a training set S

Maximal Margin Hyperplane

A hyperplane realising the maximun geometric margin

The optimal linear classifier

If it can form the Maximal Margin Hyperplane.

How to Find the optimal solution ？

The drawback of the perceptron algorithm

The algorithm may give a different solution depending on the order in which the examples are processed.

The superiority of SVM

The kind of learning machines tune the solution based on the optimization theory .

The Maximal Margin Classifier

The simplest model of SVM

Finds the maximal margin hyperplane in an chosen kernel-induced feature space.

A convex optimization problem

Minimizing a quadratic function under linear inequality constrains

Support Vector Classifiers

Support vector machines

Cortes and Vapnik (1995)

well suited for high-dimensional data

binary classification

Training set

D = {( x i ,y i ), i=1,…,n}, x i  R m and y i  {-1,1}

Linear discriminant classifier

Separating hyperplane

{ x : g( x ) = w T x + w 0 = 0 }

model parameters: w  R m and w 0  R

Formalizi the geometric margin

Assumes that

The geometric margin

In order to find the maximum ,we must find the minimum

Minimizing the norm ->

Because

We can re-formalize the optimization problem

Minimizing the norm ->

Uses the Lagrangian function

Obtained

Resubstituting into the primal to obtain

Minimizing the norm

Finds the minimum is equivalent to find the maximum

The strategies for minimizing differentiable function

Decomposition

Sequential Minimal Optimization (SMO)

The Support Vector

The condition of the optimization problem states that

This implies that only for input xi for which the functional margin is one

This implies that it lies closest to the hyperplane

The corresponding

The optimal hypothesis (w,b)

The two parameters can be obtained from

The hypothesis is

Soft Margin Optimization

The main problem with the maximal margin classifier is that it always products perfectly a consistent hypothesis

a hypothesis with no training error

Relax the boundary

The problem

The maximal margin classifier is an important concept, but it cannot be used in many real-world problems

There will in general be no linear separation in the feature space.

The solution

Maps the data into another space that can be separated linearly.

A learning machine

A learning machine f takes an input x and transforms it, somehow using weights  , into a predicted output y est = +/- 1

f x  y est  is some vector of adjustable parameters

Some definitions

Given some machine f

And under the assumption that all training points (x k ,y k ) were drawn i.i.d from some distribution.

And under the assumption that future test points will be drawn from the same distribution

Define

Official terminology

Some definitions

Given some machine f

And under the assumption that all training points (x k ,y k ) were drawn i.i.d from some distribution.

And under the assumption that future test points will be drawn from the same distribution

Define

Official terminology R = #training set data points

Vapnik-Chervonenkis Dimension

Given some machine f , let h be its VC dimension.

h is a measure of f ’s power ( h does not depend on the choice of training set)

Vapnik showed that with probability 1- 

This gives us a way to estimate the error on future data based only on the training error and the VC-dimension of f

Structural Risk Minimization

Let  (f) = the set of functions representable by f.

Suppose

Then

We’re trying to decide which machine to use.

We train each machine and make a table…

f 4 4 f 5 5 f 6 6  f 3 3 f 2 2 f 1 1 Choice Probable upper bound on TESTERR VC-Conf TRAINERR f i i

Kernel-Induced Feature Space

Mapping the data of space X into space F

Implicit Mapping into Feature Space

For the non-linear separable data set, we can modify the hypothesis to map implicitly the data to another feature space

Kernel Function

A Kernel is a function K , such that for all

The benefits

Solve the computational problem of working with many dimensions

Kernel function

The Polynomial Kernel

The kind of kernel represents the inner product of two vector(point) in a feature space of dimension.

For example

Text Categorization Inductive learning Inpute : Output : f(x) = confidence(class) In the case of text classification ,the attribute are words in the document ,and the classes are the categories.

PROPERTIES OF TEXT-CLASSIFICATION TASKS

High-Dimensional Feature Space.

Sparse Document Vectors.

High Level of Redundancy.

Text representation and feature selection

Binary feature

term frequency

Inverse document frequency

n is the total number of documents

DF(w) is the number of documents the word occurs in

Learning SVMS

To learn the vector of feature weights

Linear SVMS

Polynomial classifiers

Radial basis functions

Processing

Text files are processed to produce a vector of words

Select 300 words with highest mutual information with each category(remove stopwords)

A separate classifier is learned for each category.

An example - Reuters ( trends & controversies)

Category : interest

Weight vector

large positive weights : prime (.70), rate (.67), interest (.63), rates (.60), and discount (.46)

large negative weights: group (–.24),year (–.25), sees (–.33) world (–.35), and dlrs (–.71)

Text Categorization Results Dumais et al. (1998)

Apply to the Linear Classifier

Substitutes to the hypothesis

Substitutes to the margin optimization

SVMs and PAC Learning

Theorems connect PAC theory to the size of the margin

Basically, the larger the margin, the better the expected accuracy

See, for example, Chapter 4 of Support Vector Machines by Christianini and Shawe-Taylor, Cambridge University Press, 2002

PAC and the Number of Support Vectors

The fewer the support vectors, the better the generalization will be

Recall, non-support vectors are

Correctly classified

Don’t change the learned model if left out of the training set

So

VC-dimension of an SVM

Very loosely speaking there is some theory which under some different assumptions puts an upper bound on the VC dimension as

where

Diameter is the diameter of the smallest sphere that can enclose all the high-dimensional term-vectors derived from the training set.

Margin is the smallest margin we’ll let the SVM use

This can be used in SRM (Structural Risk Minimization) for choosing the polynomial degree, RBF  , etc.

But most people just use Cross-Validation

Finding Non-Linear Separating Surfaces

Map inputs into new space

Example: features x 1 x 2

5 4

Example: features x 1 x 2 x 1 2 x 2 2 x 1 *x 2

5 4 25 16 20

Solve SVM program in this new space

Computationally complex if many features

But a clever trick exists

Summary

Maximize the margin between positive and negative examples (connects to PAC theory)

The support vectors contribute to the solution

Kernels map examples into a new, usually non-linear space

References

Vladimir Vapnik. The Nature of Statistical Learning Theory , Springer, 1995

Andrew W. Moore. cmsc726: SVMs. http:// www.cs.cmu.edu/~awm/tutorials

C. Burges. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery, 2(2):955-974, 1998. http://citeseer.nj.nec.com/burges98tutorial.html