This document provides an overview of pattern recognition and classification. It discusses key concepts such as features, feature vectors, classifiers, supervised vs. unsupervised learning, Bayes classifiers, discriminant functions, decision surfaces, and the Gaussian distribution. The Gaussian distribution is described as being useful for modeling feature vectors, with the multivariate Gaussian defined using a mean vector and covariance matrix. Bayes classifiers are highlighted as being optimal for minimizing classification error probability when assigning patterns to classes based on feature vectors.
2. 2
PATTERN RECOGNITION
Typical application areas
Machine vision
Character recognition (OCR)
Computer aided diagnosis
Speech recognition
Face recognition
Biometrics
Image Data Base retrieval
Data mining
Bionformatics
The task: Assign unknown objects – patterns – into the correct
class. This is known as classification.
3. 3
Features: These are measurable quantities obtained from
the patterns, and the classification task is based on their
respective values.
Feature vectors: A number of features
constitute the feature vector
Feature vectors are treated as random vectors.
,
,...,
1 l
x
x
l
T
l R
x
x
x
,...,
1
5. 5
The classifier consists of a set of functions, whose values,
computed at , determine the class to which the
corresponding pattern belongs
Classification system overview
x
sensor
feature
generation
feature
selection
classifier
design
system
evaluation
Patterns
6. 6
Supervised – unsupervised – semisupervised pattern
recognition:
The major directions of learning are:
Supervised: Patterns whose class is known a-priori
are used for training.
Unsupervised: The number of classes/groups is (in
general) unknown and no training patterns are
available.
Semisupervised: A mixed type of patterns is
available. For some of them, their corresponding class
is known and for the rest is not.
7. 7
CLASSIFIERS BASED ON BAYES DECISION
THEORY
Statistical nature of feature vectors
Assign the pattern represented by feature vector
to the most probable of the available classes
That is
maximum
T
l
2
1 x
,...,
x
,
x
x
x
M
,...,
, 2
1
)
(
: x
P
x i
i
8. 8
Computation of a-posteriori probabilities
Assume known
• a-priori probabilities
•
This is also known as the likelihood of
)
(
)...,
(
),
( 2
1 M
P
P
P
M
i
x
p i ,...,
2
,
1
,
)
(
.
.
. i
to
r
w
x
10. 10
The Bayes classification rule (for two classes M=2)
Given classify it according to the rule
Equivalently: classify according to the rule
For equiprobable classes the test becomes
x
2
1
2
1
2
1
)
(
)
(
)
(
)
(
x
x
P
x
P
x
x
P
x
P
If
If
)
(
)
(
)
(
)
(
)
( 2
2
1
1
P
x
p
P
x
p
)
(
)
(
)
( 2
1
x
p
x
p
x
12. 12
Equivalently in words: Divide space in two regions
Probability of error
Total shaded area
Bayesian classifier is OPTIMAL with respect to
minimising the classification error probability!!!!
2
2
1
1
in
If
in
If
x
R
x
x
R
x
0
0
)
(
2
1
)
(
2
1
1
2
x
x
e dx
x
p
dx
x
p
P
13. 13
Indeed: Moving the threshold the total shaded
area INCREASES by the extra “grey” area.
14. 14
The Bayes classification rule for many (M>2) classes:
Given classify it to if:
Such a choice also minimizes the classification error
probability
Minimizing the average risk
For each wrong decision, a penalty term is assigned since
some decisions are more sensitive than others
i
j
x
P
x
P j
i
)
(
)
(
x i
15. 15
For M=2
• Define the loss matrix
• penalty term for deciding class ,
although the pattern belongs to , etc.
Risk with respect to
)
(
22
21
12
11
L
12
1
1
x
d
x
p
x
d
x
p
r
R
R
)
(
)
( 1
12
1
11
1
2
1
2
16. 16
Risk with respect to
Average risk
2
x
d
x
p
x
d
x
p
r
R
R
)
(
)
( 2
22
2
21
2
2
1
)
(
)
( 2
2
1
1
P
r
P
r
r
Probabilities of wrong decisions,
weighted by the penalty terms
17. 17
Choose and so that r is minimized
Then assign to if
Equivalently:
assign x in if
: likelihood ratio
1
R 2
R
x i
)
( 2
1
11
12
22
21
1
2
2
1
12
)
(
)
(
)
(
)
(
)
(
P
P
x
p
x
p
12
)
(
)
(
)
(
)
(
)
(
)
(
)
(
)
(
2
2
22
1
1
12
2
2
2
21
1
1
11
1
P
x
p
P
x
p
P
x
p
P
x
p
20. 20
Then the threshold value is:
Threshold for minimum r
2
1
)
)
1
(
exp(
)
exp(
:
:
minimum
for
0
2
2
0
0
x
x
x
x
P
x e
2
1
2
)
2
1
(
ˆ
)
)
1
(
(
exp
2
)
(
exp
:
ˆ
0
2
2
0
n
x
x
x
x
0
x̂
22. 22
DISCRIMINANT FUNCTIONS
DECISION SURFACES
If are contiguous:
is the surface separating the regions. On the one
side is positive (+), on the other is negative (-). It is
known as Decision Surface.
)
(
)
(
:
)
(
)
(
:
x
P
x
P
R
x
P
x
P
R
i
j
j
j
i
i
j
i R
R , 0
)
(
)
(
)
(
x
P
x
P
x
g j
i
+
- 0
)
(
x
g
23. 23
If f (.) monotonically increasing, the rule remains the same if we use:
is a discriminant function.
In general, discriminant functions can be defined independent of the
Bayesian rule. They lead to suboptimal solutions, yet, if chosen
appropriately, they can be computationally more tractable.
Moreover, in practice, they may also lead to better solutions. This,
for example, may be case if the nature of the underlying pdf’s are
unknown.
j
i
x
P
f
x
P
f
x j
i
i
))
(
(
))
(
(
:
if
))
(
(
)
( x
P
f
x
g i
i
24. THE GAUSSIAN DISTRIBUTION
The one-dimensional case
where
is the mean value, i.e.:
is the variance,
24
2
2
2
)
(
exp
2
1
)
(
x
x
p
dx
x
xp
x
E )
(
2
dx
x
p
x
x
E
x
E )
(
)
(
)
( 2
2
2
25. The Multivariate (Multidimensional) case:
where is the mean value,
and is known s the covariance matrix and it is defined as:
An example: The two-dimensional case:
,
where 25
)
(
)
(
2
1
exp
)
2
(
1
)
( 1
2
1
2
x
x
x
p T
x
E
]
)
)(
[( T
x
x
E
2
2
1
1
1
2
2
1
1
2
1
2
1 ,
2
1
exp
2
1
,
)
(
x
x
x
x
x
x
p
x
p
2
1
2
1
x
E
x
E
2
2
2
1
2
2
1
1
2
2
1
1
,
x
x
x
x
E
)
)(
( 2
2
1
1
x
x
E
26. 26
BAYESIAN CLASSIFIER FOR NORMAL
DISTRIBUTIONS
Multivariate Gaussian pdf
is the covariance matrix.
)
)(
(
for
vector,
1
an
is
)
(
)
(
2
1
exp
)
2
(
1
)
( 1
2
1
2
i
i
i
i
i
i
i
i
i
i
x
x
E
x
x
E
x
x
x
p
27. 27
is monotonic. Define:
Example:
)
ln(
)
(
ln
)
(
ln
))
(
)
(
ln(
)
(
i
i
i
i
i
P
x
p
P
x
p
x
g
i
i
i
i
i
i
T
i
i
C
C
P
x
x
x
g
ln
)
2
1
(
2
ln
)
2
(
)
(
ln
)
(
)
(
2
1
)
( 1
2
2
i
0
0
28. 28
That is, is quadratic and the surfaces
quadrics, ellipsoids, parabolas, hyperbolas,
pairs of lines.
i
i
i
i
i
i
i
C
P
x
x
x
x
x
g
)
ln(
)
(
2
1
)
(
1
)
(
2
1
)
(
2
2
2
1
2
2
2
1
1
2
2
2
2
1
2
)
(x
gi
0
)
(
)
(
x
g
x
g j
i
30. 30
Decision Hyperplanes
Quadratic terms:
If ALL (the same) the quadratic
terms are not of interest. They are not
involved in comparisons. Then, equivalently,
we can write:
Discriminant functions are LINEAR.
x
x 1
i
T
Σ
Σi
i
i
Τ
i
i
i
i
io
T
i
i
Σ
P
w
Σ
w
w
x
w
x
g
1
0
1
2
1
)
(
ln
)
(
31. 31
Let in addition:
•
•
•
• 2
2
0
2
2
)
(
)
(
ln
)
(
2
1
,
)
(
0
)
(
)
(
)
(
1
)
(
Then
.
j
i
j
i
j
i
j
i
o
j
i
o
T
j
i
ij
i
T
i
i
P
P
x
w
x
x
w
x
g
x
g
x
g
w
x
x
g
I
Σ
32. Remark :
• If , then
32
)
(
)
( 2
1
P
P
2
1
0
2
1
x
33. • If , the linear classifier moves towards the
class with the smaller probability
33
2
1
P
P
34. 34
Nondiagonal:
•
•
•
Decision hyperplane
2
0
)
(
)
( 0
x
x
w
x
g
T
ij
)
(
1
j
i
w
2
1
1
2
0
)
(
where
)
)
(
)
(
(
n
)
(
2
1
1
1
x
x
x
P
P
x
T
j
i
j
i
j
i
j
i
)
(
to
normal
to
normal
not
1
j
i
j
i
35. 35
Minimum Distance Classifiers
equiprobable
Euclidean Distance:
smaller
Mahalanobis Distance:
smaller
M
P i
1
)
(
)
(
)
(
2
1
)
( 1
i
T
i
i x
x
x
g
:
Assign
:
2
i
x
I
i
E x
d
:
Assign
:
2
i
x
I
2
1
1
))
(
)
(( i
T
i
m x
x
d
38. 38
Maximum Likelihood
:
method
The
to
w.r.
of
Likelihood
the
as
known
is
which
)
;
(
)
;
,...
,
(
)
;
(
,...
,
)
;
(
)
(
:
parameter
ctor
unknown ve
an
in
known with
)
(
Let
t
independen
and
known
,....,
,
Let
1
2
1
2
1
2
1
X
x
p
x
x
x
p
X
p
x
x
x
X
x
p
x
p
x
p
x
x
x
k
N
k
N
N
N
ESTIMATION OF UNKNOWN PROBABILITY
DENSITY FUNCTIONS
41. 41
Asymptotically unbiased and consistent
0
ˆ
lim
]
ˆ
[
lim
then
,
)
;
(
)
(
such that
a
is
there
indeed,
If,
2
0
N
0
N
0
0
ML
ML
E
E
x
p
x
p
43. 43
Maximum Aposteriori Probability Estimation
In ML method, θ was considered as a parameter
Here we shall look at θ as a random vector
described by a pdf p(θ), assumed to be known
Given
Compute the maximum of
From Bayes theorem
N
2
1 x
,...,
x
,
x
X
)
( X
p
)
(
)
(
)
(
)
(
or
)
(
)
(
)
(
)
(
X
p
X
p
p
X
p
X
p
X
p
X
p
p
49. 49
The previous formulae correspond to a sequence
of Gaussians for different values of N.
Example : Prior information : , ,
True mean .
0
0
8
2
0
2
50. 50
Maximum Entropy Method
Compute the pdf so that to be maximally non-
committal to the unavailable information and
constrained to respect the available information.
The above is equivalent with maximizing uncertainty,
i.e., entropy, subject to the available constraints.
Entropy
x
d
x
p
x
p
H )
(
ln
)
(
s
constraint
available
the
subject to
maximum
:
)
(
ˆ H
x
p
51. 51
Example: x is nonzero in the interval
and zero otherwise. Compute the ME pdf
• The constraint:
• Lagrange Multipliers
•
2
1 x
x
x
2
1
1
)
(
x
x
dx
x
p
2
1
)
1
)
(
(
x
x
L dx
x
p
H
H
otherwise
0
1
)
(
ˆ
)
1
exp(
)
(
ˆ
2
1
1
2
x
x
x
x
x
x
p
x
p
52. • This is most natural. The “most random”
pdf is the uniform one. This complies
with the Maximum Entropy rationale.
It turns out that, if the constraints are the:
mean value and the variance
then the Maximum Entropy estimate is the
Gaussian pdf.
That is, the Gaussian pdf is the “most
random” one, among all pdfs with the
same mean and variance.
52
53. 53
Mixture Models
Assume parametric modeling, i.e.,
The goal is to estimate
given a set
Why not ML? As before?
M
j x
j
J
j
j
x
d
j
x
p
P
P
j
x
p
x
p
1
1
1
)
(
,
1
)
(
)
(
)
;
(
j
x
p
J
P
P
P ,...,
,
and 2
1
N
2
1 x
,...,
x
,
x
X
)
,...,
,
;
(
max 1
1
,...,
, 1
J
k
N
k
P
P
P
P
x
p
J
54. 54
This is a nonlinear problem due to the missing
label information. This is a typical problem with
an incomplete data set.
The Expectation-Maximisation (EM) algorithm.
• General formulation
–
which are not observed directly.
We observe
a many to one transformation
,
;
y
p
R
Y
y
y y
m
)
(
with
,
set
data
complete
the
),
;
(
with
,
)
(
x
p
m
l
R
X
y
g
x x
l
ob
55. 55
• Let
• What we need is to compute
• But are not observed. Here comes the EM.
Maximize the expectation of the loglikelihood
conditioned on the observed samples and the
current iteration estimate of
0
))
;
(
ln(
:
ˆ
k
k
y
ML
y
p
s
yk '
.
)
(
)
;
(
)
;
(
specific
a
to
'
all
)
(
x
Y
y
x y
d
y
p
x
p
x
s
y
Y
x
Y
56. 56
The algorithm:
• E-step:
• M-step:
Application to the mixture modeling problem
• Complete data
• Observed data
•
• Assuming mutual independence
k
k
y t
X
y
p
E
t
Q ))]
(
;
;
(
ln(
[
))
(
;
(
0
))
(
;
(
:
)
1
(
t
Q
t
N
k
j
x k
k ,...,
2
,
1
),
,
(
N
k
xk ,...,
2
,
1
,
k
j
k
k
k
k P
j
x
p
j
x
p )
;
(
)
;
,
(
)
)
;
(
ln(
)
(
1
N
k
jk
k
k P
j
x
p
L
57. 57
• Unknown parameters
• E-step
• M-step
T
J
T
T
T
T
P
P
P
P
P ]
,...,
,
[
,
]
,
[ 2
1
J
j
P
Q
Q
k
jk
,...,
2
,
1
,
0
0
J
j
j
k
k
k
j
k
k P
t
j
x
p
t
x
p
t
x
p
P
t
j
x
p
t
x
j
P
1
))
(
;
(
))
(
;
(
))
(
;
(
))
(
;
(
))
(
;
(
)
)
(
(
ln
))
(
;
(
]
[
)]
)
;
(
ln(
[
))
(
;
(
;
1
1
1
1
k
k
k
j
k
k
k
k
J
j
N
k
N
k
N
k
j
k
k
P
j
x
p
t
x
j
P
E
P
j
x
p
E
t
Q
58. 58
Nonparametric Estimation
In words : Place a segment of length h at
and count points inside it.
If is continuous: as , if
2
2
h
x̂
x̂
h
x̂
)
(
)
( x
p
x
p
2
ˆ
-
,
1
)
ˆ
(
ˆ
)
(
ˆ
total
in
h
x
x
N
k
h
x
p
x
p
N
h
k
N
k
P
N
N
N
x̂
0
,
,
0
N
k
k
h N
N
N
)
(x
p
N
60. 60
Define
• That is, it is 1 inside a unit side hypercube centered
at 0
•
•
• The problem:
• Parzen windows-kernels-potential functions
))
(
1
(
1
)
(
ˆ
1
N
i
i
l
h
x
x
N
h
x
p
x
N
at
centered
hypercube
side
-
h
an
inside
points
of
number
*
1
*
volume
1
ous
discontinu
(.)
continuous
)
(
x
p
x
x
d
x
x
x
1
)
(
,
0
)
(
smooth
is
)
(
otherwise
2
1
0
1
)
( ij
i
x
x
61. 61
Mean value
•
•
•
•
Hence unbiased in the limit
'
1
'
)
'
(
)
'
(
1
)])
(
[
1
(
1
)]
(
ˆ
[
x
l
N
i
i
l
x
d
x
p
h
x
x
h
h
x
x
E
N
h
x
p
E
l
h
1
,
0
h
0
)
'
(
of
width
the
0
h
x
x
h
1
)
'
(
1
x
d
h
x
x
hl
)
(
)
(
1
0 x
h
x
h
h l
'
)
(
'
)
'
(
)
'
(
)]
(
ˆ
[
x
x
p
x
d
x
p
x
x
x
p
E
64. 64
If
•
•
•
asymptotically unbiased
The method
• Remember:
•
Ν
h
N
h 0
11
12
22
21
1
2
2
1
12
)
(
)
(
)
(
)
(
)
(
P
P
x
p
x
p
l
)
(
)
(
1
)
(
1
2
1
1
2
1
1
N
i
i
l
N
i
i
l
h
x
x
h
N
h
x
x
h
N
65. 65
CURSE OF DIMENSIONALITY
In all the methods, so far, we saw that the higher
the number of points, N, the better the resulting
estimate.
If in the one-dimensional space an interval, filled
with N points, is adequate (for good estimation), in
the two-dimensional space the corresponding square
will require N2 and in the ℓ-dimensional space the ℓ-
dimensional cube will require Nℓ points.
The exponential increase in the number of necessary
points in known as the curse of dimensionality. This
is a major problem one is confronted with in high
dimensional spaces.
67. 67
NAIVE – BAYES CLASSIFIER
Let and the goal is to estimate
i = 1, 2, …, M. For a “good” estimate of the pdf
one would need, say, Nℓ points.
Assume x1, x2 ,…, xℓ mutually independent. Then:
In this case, one would require, roughly, N points
for each pdf. Thus, a number of points of the
order N·ℓ would suffice.
It turns out that the Naïve – Bayes classifier
works reasonably well even in cases that violate
the independence assumption.
x
i
x
p
|
1
|
|
j
i
j
i x
p
x
p
68. 68
K Nearest Neighbor Density Estimation
In Parzen:
• The volume is constant
• The number of points in the volume is varying
Now:
• Keep the number of points
constant
• Leave the volume to be varying
•
k
kN
)
(
)
(
ˆ
x
NV
k
x
p
70. 70
The Nearest Neighbor Rule
Choose k out of the N training vectors, identify the k
nearest ones to x
Out of these k identify ki that belong to class ωi
The simplest version
k=1 !!!
For large N this is not bad. It can be shown that:
if PB is the optimal Bayesian error probability, then:
j
i
k
k
:
x j
i
i
Assign
2
)
1
2
( B
B
B
NN
B P
P
M
M
P
P
P
71. 71
For small PB:
An example:
k
P
2
P
P
P NN
B
kNN
B
B
kNN P
P
,
k
2
3 )
(
3
2
B
B
NN
B
NN
P
P
P
P
P
73. 73
Bayes Probability Chain Rule
Assume now that the conditional dependence for
each xi is limited to a subset of the features
appearing in each of the product terms. That is:
where
BAYESIAN NETWORKS
)
(
)
|
(
...
...
)
,...,
|
(
)
,...,
|
(
)
,...,
,
(
1
1
2
1
2
1
1
1
2
1
x
p
x
x
p
x
x
x
p
x
x
x
p
x
x
x
p
2
1
2
1 )
|
(
)
(
)
,...,
,
(
i
i
i A
x
p
x
p
x
x
x
p
1
2
1 ,...,
, x
x
x
A i
i
i
74. 74
For example, if ℓ=6, then we could assume:
Then:
The above is a generalization of the Naïve – Bayes.
For the Naïve – Bayes the assumption is:
Ai = Ø, for i=1, 2, …, ℓ
)
,
|
(
)
,...,
|
( 4
5
6
1
5
6 x
x
x
p
x
x
x
p
1
5
4
5
6 ,...,
, x
x
x
x
A
75. 75
A graphical way to portray conditional dependencies
is given below
According to this figure we
have that:
• x6 is conditionally dependent on
x4, x5
• x5 on x4
• x4 on x1, x2
• x3 on x2
• x1, x2 are conditionally
independent on other variables.
For this case:
)
(
)
(
)
|
(
)
|
(
)
,
|
(
)
,...,
,
( 1
2
2
3
4
5
4
5
6
6
2
1 x
p
x
p
x
x
p
x
x
p
x
x
x
p
x
x
x
p
76. 76
Bayesian Networks
Definition: A Bayesian Network is a directed acyclic
graph (DAG) where the nodes correspond to random
variables. Each node is associated with a set of
conditional probabilities (densities), p(xi|Ai), where xi
is the variable associated with the node and Ai is the
set of its parents in the graph.
A Bayesian Network is specified by:
• The marginal probabilities of its root nodes.
• The conditional probabilities of the non-root nodes,
given their parents, for ALL possible values of the
involved variables.
77. 77
The figure below is an example of a Bayesian
Network corresponding to a paradigm from the
medical applications field.
This Bayesian network
models conditional
dependencies for an
example concerning
smokers (S),
tendencies to develop
cancer (C) and heart
disease (H), together
with variables
corresponding to heart
(H1, H2) and cancer
(C1, C2) medical tests.
78. 78
Once a DAG has been constructed, the joint
probability can be obtained by multiplying the
marginal (root nodes) and the conditional (non-root
nodes) probabilities.
Training: Once a topology is given, probabilities are
estimated via the training data set. There are also
methods that learn the topology.
Probability Inference: This is the most common task
that Bayesian networks help us to solve efficiently.
Given the values of some of the variables in the
graph, known as evidence, the goal is to compute
the conditional probabilities for some of the other
variables, given the evidence.
79. 79
Example: Consider the Bayesian network of the
figure:
a) If x is measured to be x=1 (x1), compute
P(w=0|x=1) [P(w0|x1)].
b) If w is measured to be w=1 (w1) compute
P(x=0|w=1) [ P(x0|w1)].
80. 80
For a), a set of calculations are required that
propagate from node x to node w. It turns out that
P(w0|x1) = 0.63.
For b), the propagation is reversed in direction. It
turns out that P(x0|w1) = 0.4.
In general, the required inference information is
computed via a combined process of “message
passing” among the nodes of the DAG.
Complexity:
For singly connected graphs, message passing
algorithms amount to a complexity linear in the
number of nodes.