The document provides a practical overview of linear classifiers, discussing the advantages and disadvantages of generative and discriminative models, their combination, and key concepts like perceptron, margin, and kernel methods. It covers various applications and algorithms, including online learning and ensemble methods, as well as practical implementations using libraries like sklearn and Weka. Additionally, it raises questions related to non-linear separability, kernel choices, and relationships to structured models such as CRF and maximum margin Markov networks.

1. A
Prac'cal
Overview
of
Linear
Classifier

2. Discrimina've/Genera've
Pair

3. Ques'ons
• Advantages/disadvantages
of
genera've/
discrimina've
model?
• Can
we
combine
genera've
and
discrimina've
model?

4. Outline
• Perceptron
• Margin
• Kernel
• Structure
• Online
• Ensemble

7. Generic
Perceptron

12. Margin

13. Max
Margin
Helps

14. Percptron
VS
SVM

15. Theory
• Theore'cal
jus'fica'on:
– Large
margin
implies
small
VC
dimension
• Sta's'cal
Learning
Theory
(SLT)
– VC-­‐dimension
(Vapnik-­‐Chervonenkis)
• Probably
Approximately
Correct
(PAC)
Theory

16. Parameter
Es'ma'on
• SMO
algorithm
• Online
algorithm

17. Classifier
Comparison
on
Sen'ment
Classifica'on
Accuracy

18. Ques'ons
• What
if
not
linear
separable?
• Can
SVM
be
used
when
some
data
do
not
have
labels?

19. Kernel

20. Why
Linear
Classifier
• Simple
• Applicable
• Fast

22. Kernel:
Nonlinear
-­‐>
Linear

24. Kernel
SVM

25. Ques'ons
• What
is
the
best
kernel?
• Can
we
use
mul'ple
kernels?
• kernels
for
NLP?

26. Structure

28. Feature
Func'ons

30. Applica'on1:
POS
Tagging

31. Applica'on2:
NP
Chunking

32. Ques'ons
• Rela'onship
to
CRF?
• Structured
SVM?
• Maximum
Margin
Markov
Network?

33. Ensemble
Perceptron
• Many
classifiers
are
beXer
than
one
• Vo'ng
perceptron
• Average
perceptron

35. Online
Learning
• Batch
• Online
• MiniBatch

36. Implementa'ons
Algorithm Implemena/on
Perceptron Sklearn(Python),
Weka(Java)
SVM LibSVM,
LibLinear(Linear
kernel)
Logis'c
regression/Maximum
Entropy LibLinear
CRF CRF++,
CRFSuite
Naïve
Bayes Sklean,
Weka

37. Perceptron
and
Beyond
Kernel Online
Kernel
SVM Online
SVM
Kernel
perceptron Online
CRF
Perceptron
Margin Vo'ng
perceptron
Linear
SVM Ensemble
Averaged
perceptron
Margin
perceptron
Structured
Perceptron
Boos'ng
Structured
SVM
CRF
M3N
Structure

38. Examples
• hXp://www.mathworks.com/matlabcentral/
fileexchange/28302-­‐svm-­‐demo
• hXp://www.eee.metu.edu.tr/~alatan/
Courses/Demo/AppletSVM.html