Machine Learning in Speech and Language Processing




                 Jeff A. Bilmes                   Patrick Haffner
 ...
2nd Half: Kernel and Margin Classifiers
  « Good classifiers for speech and language applications?
  « Huge range of approac...
2nd Half: Outline
  « Statistical learning theory
  « Kernel Classifiers
     SVMs: Maximize the margin
     Making kernels...
Binary Classifiers


 Using training data (x1 , y1 ), . . . , (xm , ym ).
  « Associate output y to input x ∈ X
  « Most of...
Statistical learning theory



  « Purpose:
     Techniques to predict the generalization error.
     Methods to minimize ...
Empirical risk minimization

 Training examples (x1 , y1 ), . . . , (xm , ym ) drawn from P (x, y).
  « Learn f : X → {−1,...
Uniform convergence bounds

 Law of large numbers:Rm [f ] → Rexp [f ] when m → ∞.
 But convergence must be Uniform over al...
Putting bounds into practice


  « Worst case bounds are over-pessimistic:
     Huge number of examples required to guaran...
2nd Half: Kernel Classifiers
  « Statistical learning theory
  « Kernel Classifiers
     SVMs: Maximize the margin
     Maki...
Large-Margin Classifiers, Support Vector Machines


                      = sign(w · x + b).
 Classifiers of type f (x)
    ...
Finding the maximum margin separation


  « Find weight vector w that
     maximizes the margin ρ
     with constraints w ...
Non-Linear SVMs

 Mapping from input space X to a higher-dimensional Hilbert space.
                                      ...
The Kernel Trick

  « Idea
     Mapping Φ from input space X to high-dimensional Hilbert
       space F : x → Φ(x) = (φ1 (...
Positive Definite Symmetric (PDS) Kernels

 Define Kernel Matrix K
                                            K1,1       K1...
Example with Polynomial Kernels

 Imagine a linear classifier over all pairs of input features:
  « d(d + 1)/2 features.
  ...
Polynomial Kernel Success Story: MNIST



 Optical Character Recognition
  « Reference dataset.
  « 60,000 training sample...
Making kernels from distances

 We want to imagine what kernels do
  « PDS kernels can be interpreted as measures of simil...
Distance Kernels for Distributions

 Gaussian kernel based on Euclidean distance is standard:

                      K(x, ...
Distance Kernel Success Story: WebKB



  « Home pages gathered from university
      computer science departments
  « Cla...
Complexity of runtime implementation



  « Classifier with set of support vectors SV .
  « Classify example x
     f (x) =...
Classifying complex objects

  « Classification algorithms on flat fixed-size input vectors.
  « However, the world is more c...
How to project objects into a Hilbert Space?



                          Structure Kernel


                             ...
Kernels for structured data


  « Model Driven: knowledge about objects encoded in models or
      relationships.
       F...
Tree Kernels

                                                                                A
 Subtree tree kernels:(Vis...
Kernels for strings: a general framework


 What features can be extracted from strings?
  « Bag of Words feature sets:
  ...
String Kernels


 Find similarities between CAT and BAT strings.
 Explicit extraction of all subsequences:
               ...
Examples of String Kernels

  « Convolution kernels (Haussler, 1999)
     Syntax-driven kernels constructed with recursive...
A structured object from Speech recognition

                                                                             ...
A generic generalization of String kernels

  « Each string kernel:
     Specific, usually recursive, implementation
     D...
Spoken-Language Classification – Task

                                                                           R
 Call-c...
Spoken-Language Classification – Experimental Result

 Start from Bag-of-Word features from one-best sentence            Er...
Kernel vs. Preprocessing Interpretation


                                                            Direct inner product...
Kernel vs. Preprocessing Alternative


 Structured kernels such as string, rational or tree kernels provide
  « Frameworks...
2nd Half: Margin Classifiers
  « Statistical learning theory
  « Kernel Classifiers
     SVMs: Maximize the margin
     Maki...
Classification for Natural Language Applications



  « Large number of training examples and classes.
  « Large number of ...
Margin-based linear classifiers


 Linear classifier fw (x)   = w · x + b with trainable weight w and
 bias b.

  « The tota...
Sample vs. Separation margins


                                        Loss enforces a large sample margin:
             ...
Example of soft margin support vectors

                                          One spring/outlier




    Linear cost 1...
Another interpretation of SVMs




                  Sample Loss                              Regularizer
  Hinge loss for...
Feature-based learning


  « Adaboost: Features as weak classifiers.
     incrementally refines a weighted combination of we...
Boosting for dummies

 Examples of calls to classify as Pay Bill:
  « I’d like to make a payment
  « I wanna pay my bill
 ...
The boosting family of algorithms

  « Training examples (x1 , y1 ), . . . , (xm , ym ) ∈ X × {−1, 1}.
  « for t = 1, . . ...
Adaboost as a Margin Classifier


 Adaboost shown to minimize the exponential loss

     i exp(−yi f (xi ))
               ...
Conditional Maximum Entropy

 Exponential models for + and - examples (with normalization Z(x))
                         e...
Optimization space


             Primal- features                          Dual - vectors
  Weight vector:1 parameter per...
Multiclass learning - Previous work


  « Large Margin Classifiers are by essence binary.
  « Many combination of binary cl...
Specific Multiclass Interpretations


 Definition of a multiclass sample margin:
  « Target class c: margin is fc (x) − supd...
Large Scale Experiments


                      R
 ATT VoiceTone           natural dialog system:
  « Collect together dat...
Scaling up from 18K to 580K samples: Test Error


                         33
                         32
                ...
Scaling up from 18K to 580K samples: Training time


                               Boost, underfit
                  64  ...
Comparative Summary


                                SVM                   Adaboost          Maxent
   Combine           ...
2nd half: Concluding remarks


 Choosing the right approach:
  « Kernels give you more representation power:
     Fold you...
Acknowledgments




 This half of the tutorial relies on contributions and help of:
 Srinivas Bangalore, Leon Bottou, Cori...
Online References




  « SVM: http://www.kernel-machines.org/
  « Boosting:
     Website: http://www.boosting.org/
     S...
Software

  « FSM library
    http://www.research.att.com/sw/tools/fsm
  « SVM light http://svmlight.joachims.org/
     Re...
Books

  « Trevor Hastie, Robert Tibshirani and Jerome Friedman. The Elements
      of Statistical Learning: Data Mining, ...
References
    C OLLINS , M ICHAEL AND N IGEL D UFFY. 2002. N EW R ANKING A LGORITHMS FOR PARSING
    AND TAGGING : K ERNE...
S CHAPIRE , R OBERT E.
                       AND YORAM S INGER . 2000. B OOSTEXTER : A BOOSTING - BASED
   SYSTEM FOR TEX...
Upcoming SlideShare
Loading in …5
×

Machine Learning in Speech and Language Processing

1,417 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,417
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
45
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Machine Learning in Speech and Language Processing

  1. 1. Machine Learning in Speech and Language Processing Jeff A. Bilmes Patrick Haffner Trutorial presented on March 19, 2005 ICASSP’05, PHILADEPHIA 2nd Half: Kernel and Margin Classifiers Presented by Patrick Haffner Bilmes/Haffner Kernel and Margin-based Classifiers 0 1
  2. 2. 2nd Half: Kernel and Margin Classifiers « Good classifiers for speech and language applications? « Huge range of approaches: focus on recent advances. Stronger emphasis on natural language applications: received a lot of recent attention from the Machine Learning community. Applications such as Neural Networks for Speech processing: Considered as an established technique « Key challenges: Beat the curse of dimensionality: good generalization to test data. Classifying structured objects. Scaling learning to very large corpora. Bilmes/Haffner Kernel and Margin-based Classifiers Statistical learning theory 2
  3. 3. 2nd Half: Outline « Statistical learning theory « Kernel Classifiers SVMs: Maximize the margin Making kernels, kernels for Structured Data String and Weighted Automata Kernels « Margin Classifiers Margin-based loss functions Generalize the concept of margin to a large class of classifier, including Boosting and Maxent. Large Scale Experiments « Software and References Bilmes/Haffner Kernel and Margin-based Classifiers Statistical learning theory 3
  4. 4. Binary Classifiers Using training data (x1 , y1 ), . . . , (xm , ym ). « Associate output y to input x ∈ X « Most of our presentation assume binary classification: y ∈ {−1, 1} or {0, 1}. The following generalizations are usually possible: « Probability estimator y ∈ [0, 1]. « Regression y ∈ R. « Multi-category classification y ∈ {1, . . . , l}. « Output codes y ∈ {−1, 1}l . « Graphical Models x is the set of variables y depends upon. Bilmes/Haffner Kernel and Margin-based Classifiers Statistical learning theory 4
  5. 5. Statistical learning theory « Purpose: Techniques to predict the generalization error. Methods to minimize this error. « Model: we observe data generated by an unknown IID (Independent Identically Distributed) distribution. « Analysis of the learning problem What functions can the machine implement? Capacity of this class of function? Bilmes/Haffner Kernel and Margin-based Classifiers Statistical learning theory 5
  6. 6. Empirical risk minimization Training examples (x1 , y1 ), . . . , (xm , ym ) drawn from P (x, y). « Learn f : X → {−1, +1}. « Minimize the expected risk on a test set drawn from the same P (x, y). Rexp [f ] = L(f (x), y)dP (x, y) where L(f (x), y) = 2|f (x)−y| Problem: P is unknown Need an induction principle Find f ∈ F minimizing the Empirical Risk: 1 Rm [f ] = L(f (xi ), yi ) m m Bilmes/Haffner Kernel and Margin-based Classifiers Statistical learning theory 6
  7. 7. Uniform convergence bounds Law of large numbers:Rm [f ] → Rexp [f ] when m → ∞. But convergence must be Uniform over all functions f ∈ F . lim P sup (Rexp [f ] − Rm [f ]) = 0 for all 0 m→∞ f ∈F Bounds to guarantee this convergence established by Vapnik (Vapnik, 1998) and others. They typically establish that with probability 1 − δ ˜ O(h) Rexp [f ] ≤ Rm [f ] + mα h is a measure of representation power of the function class F Example: VC dimension. Bilmes/Haffner Kernel and Margin-based Classifiers Statistical learning theory 7
  8. 8. Putting bounds into practice « Worst case bounds are over-pessimistic: Huge number of examples required to guarantee good generalization « Most bounds are dimension free: they do not depend on the dimension of the sample space. « While they cannot be used for accurate predictions They suggest how to constrain the class of function to beat the curse of dimensionality « First success story: Support Vector Machines and Kernel methods. Bilmes/Haffner Kernel and Margin-based Classifiers Statistical learning theory 8
  9. 9. 2nd Half: Kernel Classifiers « Statistical learning theory « Kernel Classifiers SVMs: Maximize the margin Making kernels, kernels for Structured Data String and Weighted Automata Kernels « Margin Classifiers Margin-based loss functions Generalize the concept of margin to a large class of classifier, including Boosting and Maxent. Large Scale Experiments « Software and References Bilmes/Haffner Kernel and Margin-based Classifiers Maximize the margin 9
  10. 10. Large-Margin Classifiers, Support Vector Machines = sign(w · x + b). Classifiers of type f (x) 1 ˜ D2 « Bound Rexp [f ] ≤ Rm [f ] + m O( M 2 ) Separate classes with the maximum margin M : Support Vectors Diameter D of smallest sphere cont. data. Margin M Soft margin to allow training errors: described later. Bilmes/Haffner Kernel and Margin-based Classifiers Maximize the margin 10
  11. 11. Finding the maximum margin separation « Find weight vector w that maximizes the margin ρ with constraints w = 1 and yi (w · xi + b) ≥ M . « Laplace optimization with choice of parameters: Primal Coefficients of the weight vector w. Dual Coefficient of combination of training vectors N w= i=1 αi xi where αi = 0 are Support Vectors. « With dual representation, all computations use inner products N with support vectors only: f (x) = i=1 αi (xi · x) + b Bilmes/Haffner Kernel and Margin-based Classifiers Maximize the margin 11
  12. 12. Non-Linear SVMs Mapping from input space X to a higher-dimensional Hilbert space. Linear Classifier Preprocessing φ Expensive φ(ο) φ(ο) ο ο φ(+) + φ(ο) ο + ο ο ο φ(+) φ(+) φ(ο) φ(ο) ο ο + + + φ(ο) ο φ(+) φ(+) φ(ο) + ++ + φ(+) + φ(+) φ(+) Input Space Hilbert Space: large dimension Avoid preprocessing: Work in input space? Bilmes/Haffner Kernel and Margin-based Classifiers Maximize the margin 12
  13. 13. The Kernel Trick « Idea Mapping Φ from input space X to high-dimensional Hilbert space F : x → Φ(x) = (φ1 (x), . . . , φd (x)). Define kernel K such that: Φ(x) · Φ(z) = K(x, z). « Benefits No need to define or compute Φ explicitly. Efficiency: K may be much more efficient to compute than Φ(x) · Φ(z) when Hilbert dimension is large. Flexibility: K can be chosen arbitrarily, so long as the existence of Φ is guaranteed. Condition: Positive Definite Symmetric (PDS) Kernels Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 13
  14. 14. Positive Definite Symmetric (PDS) Kernels Define Kernel Matrix K K1,1 K1,2 ... K1,m Ki,j = K(xi , xj ) K2,1 K2,2 ... K2,m ... ... ... ... K must be symmetric with positive eigenvalues. Km,1 Km,2 ... Km,m Property must be true for all m ≥ 1, {x1 , . . . , xm } ⊆ X and also writes: m ∀ {c1 , . . . , cm } ⊆ R : ci cj K(xi , xj ) ≥ 0 i,j=1 Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 14
  15. 15. Example with Polynomial Kernels Imagine a linear classifier over all pairs of input features: « d(d + 1)/2 features. « Dot product in Hilbert space: O(d2 ) operations. K(x, z) = Φ(x) · Φ(z) with Φ(x) = (xi xj ; 1 ≤ i, j ≤ d) = (xi xj )(zi zj ) 1≤i,j≤d 2 = xi zi = (x · z)2 i « Kernel in input space: O(d) operations. « Generalize to any degree polynomial. « Control over number of features in Hilbert space: capacity. Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 15
  16. 16. Polynomial Kernel Success Story: MNIST Optical Character Recognition « Reference dataset. « 60,000 training samples « 28x28 images: 784 pixels « 9th degree polynomial kernel: Kernel Error Use all 3 × 1020 pixel 9-uple as Linear 8% features « SVMs beat all other methods Poly N=9 0.6% Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 16
  17. 17. Making kernels from distances We want to imagine what kernels do « PDS kernels can be interpreted as measures of similarity, but... « Distances are much easier to visualize. We have tools to turn distances into kernels provided that they are ... « Symmetric and negative definite. For all x1 , ..., xn and c1 , ..., cn ∈ Rn satisfying i ci =0 ci cj Kd (xi , xj ) ≤ 0 i j « Key property: exp(−tKd ) positive definite for all t 0. Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 17
  18. 18. Distance Kernels for Distributions Gaussian kernel based on Euclidean distance is standard: K(x, z) = exp −ρ(x − z)2 But is it the best distance for a distribution? « Input vectors x represent distributions: i xi = 1. « Measure of divergence for Distribution:Kullback-Liebler. Not a distance, but can be approximated by: √ √ 2 The Hellinger/Bhattachryya distance d = i | xi − zi | . the L1 distance d = i |xi − zi | (Laplacian Kernel) These distances are Negative Definite Symmetric: → K = e−ρd is a valid kernel. Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 18
  19. 19. Distance Kernel Success Story: WebKB « Home pages gathered from university computer science departments « Classify them as student or faculty. Kernel Error Features: Bag of Words Gaussian 10% « Put all the words from a web page in a Laplacian 7% bag. Hellinger 6.5% « Distribution: P (w) is the probability to take word w out of the bag. Good distances can be turned into good kernels Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 19
  20. 20. Complexity of runtime implementation « Classifier with set of support vectors SV . « Classify example x f (x) = i∈SV αi K(xi , x) + b Complexity |SV | × TK , where TK is the time to compute the kernel. « Suppose Kernel is linear K(xi , x) = xi · x : f (x) = w · x + b with w = i∈SV αi xi . Complexity: number of non-zero features in x. Bilmes/Haffner Kernel and Margin-based Classifiers Making kernels 20
  21. 21. Classifying complex objects « Classification algorithms on flat fixed-size input vectors. « However, the world is more complex in areas such as: Spoken-dialog classification (strings, graphs) Text Categorization (strings) Parsing/reranking (trees) Part-of-speech tagging, named entity extraction (strings, trees) « Traditional method: feature extraction. « Kernels are a measure of similarity between objects, or a distance that: Avoid explicit computation a large-dimension feature space. By factorizing these computations. Bilmes/Haffner Kernel and Margin-based Classifiers Kernels for Structured Data 21
  22. 22. How to project objects into a Hilbert Space? Structure Kernel F(O) F(O) F(X) F(O) F(X) F(X) F(O) F(O) F(O) F(X) F(X) F(O) F(X) F(X) F(X) Structured Objects Implicit Hilbert Space Bilmes/Haffner Kernel and Margin-based Classifiers Kernels for Structured Data 22
  23. 23. Kernels for structured data « Model Driven: knowledge about objects encoded in models or relationships. Fisher Kernels: built from a generative model or a graphical model. Diffusion Kernels: local relationships encoded in a graph. « Syntax Driven: objects generated by syntactic rules. Trees Variable-length sequences. Distributions of variable-length sequences. Bilmes/Haffner Kernel and Margin-based Classifiers Kernels for Structured Data 23
  24. 24. Tree Kernels A Subtree tree kernels:(Viswanathan and Smola, 2002) Encode tree as a string by traversing it B E s = (A(B(C)(D))(E)) Use kernel for strings.. C D Subset tree kernels:(Collins and Duffy, A 2002) More general: match any subset of B E connected nodes. Subset, not subtree Recursive construction Applications: reranking of parsing trees, shallow semantic parsing. Bilmes/Haffner Kernel and Margin-based Classifiers Kernels for Structured Data 24
  25. 25. Kernels for strings: a general framework What features can be extracted from strings? « Bag of Words feature sets: text categorization (Reuters) (Joachims, 1998) « Grammar Fragments or N-grams: spoken dialog classification (ATT How May I Help You). « Variable length N-grams in computational biology. Kernels based on matching subsequences? Must generalize to: « sequences with gaps or mismatches « distributions over sequences Bilmes/Haffner Kernel and Margin-based Classifiers String Kernels 25
  26. 26. String Kernels Find similarities between CAT and BAT strings. Explicit extraction of all subsequences: Equivalent Recursive Time(string length): exponential computation: CAT Time: linear A C T CA CT AT CA CAT A B T BA BT AT Kt−2 Kt−1 Kt BAT 3 subseq. matches Recursive computation Possible generalization with penalized gaps and mismatches. BA BAT Bilmes/Haffner Kernel and Margin-based Classifiers String Kernels 26
  27. 27. Examples of String Kernels « Convolution kernels (Haussler, 1999) Syntax-driven kernels constructed with recursive convolutions. « String Kernels for text mining applications (Lodhi et al., 2001) Count common substrings (using recursion). « Mismatch Kernels in computational biology (Leslie et al., 2003) Allows mismatches between common subsequences. « Lattice Kernels for spoken language understanding Lattice: weighted alternative of sequences represented as a Finite State Automata (Cortes, Haffner, and Mohri, 2004). Average count of common word N-grams in lattice. Bilmes/Haffner Kernel and Margin-based Classifiers String Kernels 27
  28. 28. A structured object from Speech recognition around/81 27 at/19.1 around/83 32 nine/21.8 saint/51.2 petersburg/83.6 around/107 nine/5.61 9 14 19 23 36 arriving/45.2 saint/51.9 petersburg/85.9 nine/21.7 12 17 21 around/96.5 a/15.3 in/16.3 25 ninety/18 at/16.1 arrive/17.4 at/23 7 11 petersburg/80 a/15.3 saint/49.4 18 22 29 33 m/13.9 and/49.7 5 at/16.1 26 ninety/34.1 arrives/23.6 8 at/20.9 m/12.5 which/69.9 flights/53.1 leave/61.2 detroit/105 13 saint/43 nine/21.7 0 1 2 3 4 that/60.1 a/9.34 petersburg/85.6 around/97.1 30 34 6 arrive/12.8 saint/43 16 20 at/21.9 a/14.1 10 15 nine/21.7 around/97.1 28 ninety/34.1 35 and/18.7 nine/10.7 and/18.7 31 24 ninety/34.1 and/18.7 « Output of Speech Recognition Systems : word or phone lattice. « Very large number of paths, or alternative sequences. « Represented as Weighted Finite State Machines (FSM). « Weights used to rank alternative sequences. Bilmes/Haffner Kernel and Margin-based Classifiers String Kernels 28
  29. 29. A generic generalization of String kernels « Each string kernel: Specific, usually recursive, implementation Does not generalize to FSMs « Rational kernels: General framework covering all these kernels: Weighted FSM: represent both object and kernel. Key operation: composition of two FSMs. Factorization: distributivity property of the semiring used in FSMs. « FSM Software libraries available: e.g. ATT FSM library. Kernel operations implemented with a few library functions. Existing algorithms to optimize the computation: determinization, minimization. Bilmes/Haffner Kernel and Margin-based Classifiers String Kernels 29
  30. 30. Spoken-Language Classification – Task R Call-classification in ATT How May I Help You and VoiceTone spoken dialog applications. « A customer calls a service and speaks naturally. « Speech recognition outputs: acyclic weighted automata (word lattices). « Goal: route the call or send it to a dialog manager. « Categories: call-types and named entities such as Billing Services, or Calling Plans. 3 deployed customer-care applications for Health-care and Telecommunication industries: 64 to 97 categories. « Data: word lattices for 10 to 35 thousand training utterances. « Vocabulary of 5 to 8 thousand words. Bilmes/Haffner Kernel and Margin-based Classifiers Rational Kernels 30
  31. 31. Spoken-Language Classification – Experimental Result Start from Bag-of-Word features from one-best sentence Err=29% « Use word bigrams Err 24.5% 18 bigram/sentence on average! « Use word trigrams Err→ 24.5% « Use lattice instead of one-best. Err 23% « Fold preprocessing into Rational kernel based on FSM computations. Equivalent models Err→ 23% Faster implementation. « Combine rational with polynomial kernel. Err 22% But slow runtime performance. Bilmes/Haffner Kernel and Margin-based Classifiers Rational Kernels 31
  32. 32. Kernel vs. Preprocessing Interpretation Direct inner product Preprocessing= F(X).F(Y) Explicit transform F(O) F(O) F(X) F(O) F(X) F(X) F(O) F(O) F(O) F(X) F(X) F(O) F(X) F(X) F(X) Structured Objects Implicit Hilbert Space Implicit transform Structured Kernel K(X,Y)=F(X).F(Y) Bilmes/Haffner Kernel and Margin-based Classifiers When to use Kernels? 32
  33. 33. Kernel vs. Preprocessing Alternative Structured kernels such as string, rational or tree kernels provide « Frameworks to understand what type of PDS kernels can be built « Efficient and generic implementations They avoid computing features whose number: « in theory, grow exponentially « in practice, number of non-zero features surprisingly small. When to prefer a combination of preprocessing and linear classifiers? « Sparsity of Features: attractive in Natural Language applications. « Much faster runtime implementation. Bilmes/Haffner Kernel and Margin-based Classifiers When to use Kernels? 33
  34. 34. 2nd Half: Margin Classifiers « Statistical learning theory « Kernel Classifiers SVMs: Maximize the margin Making kernels, kernels for Structured Data String and Weighted Automata Kernels « Margin Classifiers Margin-based linear classifiers Generalize the concept of margin to a large class of classfier, including Boosting and Maxent. Large Scale Experiments « Software and References Bilmes/Haffner Kernel and Margin-based Classifiers When to use Kernels? 34
  35. 35. Classification for Natural Language Applications « Large number of training examples and classes. « Large number of possible features « Each example: small number of non zero features. Examples « Spoken Language Understanding « Text categorization. « Part-of-speech tagging: millions of examples. « Machine Translation: 100, 000 classes. Bilmes/Haffner Kernel and Margin-based Classifiers When to use Kernels? 35
  36. 36. Margin-based linear classifiers Linear classifier fw (x) = w · x + b with trainable weight w and bias b. « The total loss function is a combination of 2 terms Ltotal = L(f (xi ), yi ) + R(w) i « The sample loss L(f (x), y) is computed from the sample margin y.f (x), hence the name margin-based. « Regularization term R measures the complexity of the model, and encourages simple models which cannot over-train. Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 36
  37. 37. Sample vs. Separation margins Loss enforces a large sample margin: Will we have a large separation margin? Sample Margin: Separation − y (w.x) margin − Distance from decison boundary Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 37
  38. 38. Example of soft margin support vectors One spring/outlier Linear cost 1−y.f(x) Margin Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 38
  39. 39. Another interpretation of SVMs Sample Loss Regularizer Hinge loss for soft-margin SVMs: L2-norm of the weight vector C max(0, 1 − yi f (xi )) R= w2 k i k The cost parameter C: → inverse of the SVM margin. ∞ for hard-margin SVMs L1-norm: R = k |w k | Small C allow more errors → no Kernel trick possible. Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 39
  40. 40. Feature-based learning « Adaboost: Features as weak classifiers. incrementally refines a weighted combination of weak classifiers. importance of examples that remain erroneous is “adaptively boosted”. « Maximum Entropy: Features as constraints. looks for a distribution over training samples with maximum entropy satisfying a set of user-defined constraints. Regularization: relax constraints or use Gaussian prior. Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 40
  41. 41. Boosting for dummies Examples of calls to classify as Pay Bill: « I’d like to make a payment « I wanna pay my bill « I would like to pay the bill « I would like to pay my mothers long distance bill « I am making a payment with a late payment « I wanna see if I can pay my bill over the phone rules of thumbs such as the detection of pay or bill are easy to find. Boosting: method for converting rules of thumb or weak classifiers into accurate prediction rules Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 41
  42. 42. The boosting family of algorithms « Training examples (x1 , y1 ), . . . , (xm , ym ) ∈ X × {−1, 1}. « for t = 1, . . . , T : update distribution Dt on examples {1, . . . , m} find weak hypotheses ht : X → {−1, 1} with small error t = PrDt [ht (xi ) = yi ] « Output final hypotheses Hf inal . Most famous member of the family: Adaboost (Freund and Schapire, 1996) 1 1− « Using weight-update factor α = 2 log t t Dt (i) « Dt+1 (i) = Zt exp(−αt yi ht (xi )) « Hf inal (x) = sign( t αt ht (x)) Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 42
  43. 43. Adaboost as a Margin Classifier Adaboost shown to minimize the exponential loss i exp(−yi f (xi )) Sample Loss Regularizer « No explicit regularization term. L(f (x), y) = exp(−yi f (x)) « Learning concentrates on Logistic loss also possible: examples with the smallest −2yf (x) margin. L = log(1 + e ) « Generalization error bound based on L1 margin. Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 43
  44. 44. Conditional Maximum Entropy Exponential models for + and - examples (with normalization Z(x)) exp(yf + (x)) exp(yf − (x)) P (y = +1|x) = Z(x) P (y = −1|x) = Z(x) Many other models/distributions possible. Sample Loss Regularizer Log likelihood Relax constraints: parameterized L1-norm L(f (x), y) = − log(P (y|x)) −2y(f + (x)−f − (x)) R= βk |wk | L = log(1+e ) k Same as logistic regression. Log of Gaussian prior: L2-norm Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 44
  45. 45. Optimization space Primal- features Dual - vectors Weight vector:1 parameter per SVMs allows decomposition: feature f (x) = αi K(xi , x) « Adaboost i « MaxEnt Optimization over α. Sequential algorithms: Sequential algorithms: add 1 feature/iteration Most popular: SMO O(N ) complexity. add 2 vectors/iteration Faster than Iterative Scaling. O(N 2 ) complexity. Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 45
  46. 46. Multiclass learning - Previous work « Large Margin Classifiers are by essence binary. « Many combination of binary classifiers are possible: 1-vs-other: separate examples belonging to one class from all others (Rifkin and Klautau, 2004). 1-vs-1: each classifier is trained to separate a pair of classes. ECOC: error correcting output codes. (Crammer and Singer, 2000) Hierarchical approaches have only been studied recently. « Problem with methods based on recombination of binary classifiers: Each classifier corresponds to a different optimization problem. Do not guarantee a global optimality. « Global optimization procedure possible but expensive: Often requires C times more time and memory. Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 46
  47. 47. Specific Multiclass Interpretations Definition of a multiclass sample margin: « Target class c: margin is fc (x) − supd=c fd (x). « Costly global optimization required, without improved results Adaboost.MR (Schapire and Singer, 2000). Several SVM formalizations. Probabilistic framework « Maxent: many ways to define distributions over multiple classes Beyond conditional: joint or class conditional. Assumptions about how classes correlate. « SVMs ≈ probability with univariate logistic regression (Platt, 1999). Bilmes/Haffner Kernel and Margin-based Classifiers Margin Classifiers 47
  48. 48. Large Scale Experiments R ATT VoiceTone natural dialog system: « Collect together datasets from different applications. « Merge call-types when possible (for instance, all the “Request(customer care)”). 600 total. « Very large and noisy hand-labeled data with 580,000 training examples. Methods experimented « Linear: Adaboost « Linear, regularized: Maxent, Linear SVM. « Kernel, Regularized: SVM with 2nd degree polynomial. Bilmes/Haffner Kernel and Margin-based Classifiers Large Scale Experiments 48
  49. 49. Scaling up from 18K to 580K samples: Test Error 33 32 31 Top Class Error Rate 30 29 28 27 26 Boost, underfit 25 Boost, overfit Maxent 24 Linear SVM Poly2 SVM 18 36 72 144 288 576 Set size (thousands) Bilmes/Haffner Kernel and Margin-based Classifiers Large Scale Experiments 49
  50. 50. Scaling up from 18K to 580K samples: Training time Boost, underfit 64 Boost, overfit Maxent 32 Linear SVM Poly2 SVM 16 O(N) Reference O(N2) Reference 8 Time (hours) 4 2 1 0.5 0.25 18 36 72 144 288 576 Set size (thousands) Bilmes/Haffner Kernel and Margin-based Classifiers Large Scale Experiments 50
  51. 51. Comparative Summary SVM Adaboost Maxent Combine vectors features features Regularization L2 or L1 implicit L2 or L1 Capacity f(kernel) linear linear Test error f(kernel,regul) hard to improve f(regul) Training time O(m2 ) O(m) O(m) Sparsity/runtime poor excellent very good Prob. interpret. with remapping with remapping natural Color codes: best performer, worst performer Bilmes/Haffner Kernel and Margin-based Classifiers Large Scale Experiments 51
  52. 52. 2nd half: Concluding remarks Choosing the right approach: « Kernels give you more representation power: Fold your preprocessing into a kernel. Slower learning and runtime... « Many sparse features: Linear classifiers often sufficient. Use margin classifiers: generalize well. Use feature-based sequential learning: very fast « Control your regularization. « Excellent software available. Bilmes/Haffner Kernel and Margin-based Classifiers Large Scale Experiments 52
  53. 53. Acknowledgments This half of the tutorial relies on contributions and help of: Srinivas Bangalore, Leon Bottou, Corinna Cortes, Dilek Hakkani-Tur, Mazin Gilbert, Yann LeCun, Mehryar Mohri, Steven Phillips, Rob Schapire, Juergen Schroeter, Gokhan Tur, Vladimir Vapnik. Bilmes/Haffner Kernel and Margin-based Classifiers Large Scale Experiments 53
  54. 54. Online References « SVM: http://www.kernel-machines.org/ « Boosting: Website: http://www.boosting.org/ Schapire’s publications: http://www.cs.princeton.edu/˜schapire/boost.ht Freund’s applet: http://www1.cs.columbia.edu/˜freund/adaboost/ « Maxent: http://www-2.cs.cmu.edu/˜aberger/maxent.html Bilmes/Haffner Kernel and Margin-based Classifiers Software and References 54
  55. 55. Software « FSM library http://www.research.att.com/sw/tools/fsm « SVM light http://svmlight.joachims.org/ Ready to use implementation optimized for Natural Language applications. Can handle structured data. « LIBSVM http://www.csie.ntu.edu.tw/˜cjlin/libsvm/ An integrated tool implementing support vector classification, regression, and distribution estimation. C++ and Java sources available. « SVM torch http://www.torch.ch/ a modular library supporting SVMs, Adaboost and gradient based learning « BoosTexter www.cs.princeton.edu/˜schapire/BoosTexter/ a general purpose machine-learning program based on boosting for building a classifier from text and/or attribute-value data. Bilmes/Haffner Kernel and Margin-based Classifiers Software and References 55
  56. 56. Books « Trevor Hastie, Robert Tibshirani and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2004 Covers a broad range, from supervised learning, to unsupervised learning, including classification trees, neural networks, and support vector machines. « John Shawe-Taylor and Nello Cristianini Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge, UK, 2004. An introduction to Kernel methods which is accessible in its description of the theoretical foundations of large margin algorithms (467 pages). « ¨ Bernhard Scholkopf and Alex Smola Learning with Kernels. MIT Press, Cambridge, MA, 2002 An introduction and overview over SVMs. (650 pages) « Vladimir Vapnik Statistical Learning Theory. Wiley, NY, 1998. The comprehensive treatment of statistical learning theory, including a large amount of material on SVMs (768 pages,) Bilmes/Haffner Kernel and Margin-based Classifiers Software and References 56
  57. 57. References C OLLINS , M ICHAEL AND N IGEL D UFFY. 2002. N EW R ANKING A LGORITHMS FOR PARSING AND TAGGING : K ERNELS OVER D ISCRETE S TRUCTURES , AND THE VOTED P ERCEPTRON . I N ACL. C ORTES , C ORINNA , PATRICK H AFFNER , AND M EHRYAR M OHRI . 2004. R ATIONAL K ERNELS : T HEORY AND A LGORITHMS . Journal of Machine Learning Research, 5:1035–1062, AUGUST. C RAMMER , KOBY AND YORAM S INGER . 2000. O N THE L EARNABILITY AND D ESIGN OF O UTPUT C ODES FOR M ULTICLASS P ROBLEMS . I N Proceedings of COLT’00, PAGES 35–46. F REUND, YOAV AND R OBERT E. S CHAPIRE . 1996. E XPERIMENTS WITH A N EW B OOSTING A LGORITHM . I N International Conference on Machine Learning, PAGES 148–156. H AUSSLER , DAVID. 1999. C ONVOLUTION K ERNELS ON D ISCRETE S TRUCTURES . T ECHNICAL R EPORT UCSC-CRL-99-10, U NIVERSITY OF C ALIFORNIA AT S ANTA C RUZ . J OACHIMS , T HORSTEN . 1998. T EXT CATEGORIZATION WITH SUPPORT VECTOR MACHINES : LEARNING WITH MANY RELEVANT FEATURES . I N Proc. of ECML-98. S PRINGER V ERLAG . L ESLIE , C HRISTINA , E LEAZAR E SKIN , J ASON W ESTON , AND W ILLIAM S TAFFORD N OBLE . 2003. M ISMATCH S TRING K ERNELS FOR SVM P ROTEIN C LASSIFICATION . I N Advances in Neural Information Processing Systems 15 (NIPS 2002), VANCOUVER , C ANADA , M ARCH . MIT P RESS . L ODHI , H UMA , J OHN S HAWE -TAYLOR , N ELLO C RISTIANINI , AND C HRIS WATKINS . 2001. T EXT CLASSIFICATION USING STRING KERNELS . I N Advances in Neural Information Processing Systems 13 (NIPS 2000), PAGES 563–569. MIT P RESS . P LATT, J. 1999. P ROBABILISTIC OUTPUTS FOR SUPPORT VECTOR MACHINES AND COMPARISON TO REGULARIZED LIKELIHOOD METHODS . I N NIPS. MIT P RESS . R IFKIN , RYAN AND A LDEBARO K LAUTAU. 2004. I N DEFENSE OF ONE - VS - ALL CLASSIFICATION . Journal of Machine Learning Research, PAGES 101–141. Bilmes/Haffner Kernel and Margin-based Classifiers Software and References 57
  58. 58. S CHAPIRE , R OBERT E. AND YORAM S INGER . 2000. B OOSTEXTER : A BOOSTING - BASED SYSTEM FOR TEXT CATEGORIZATION . Machine Learning, 39(2/3):135–168. VAPNIK , V LADIMIR N. 1998. Statistical Learning Theory. J OHN W ILEY S ONS , N EW-YORK . V ISWANATHAN , S. V. N. AND A LEXANDER J. S MOLA . 2002. FAST KERNELS FOR STRING AND TREE MATCHING . I N NIPS, PAGES 569–576. Bilmes/Haffner Kernel and Margin-based Classifiers Software and References 58

×