This document provides an overview of pattern classification and clustering algorithms. It defines key concepts like pattern recognition, supervised and unsupervised learning. For pattern classification, it discusses algorithms like decision trees, kernel estimation, K-nearest neighbors, linear discriminant analysis, quadratic discriminant analysis, naive Bayes classifier and artificial neural networks. It provides examples to illustrate decision tree classification and information gain calculation. For clustering, it mentions hierarchical, K-means and KPCA clustering algorithms. The document is a guide to pattern recognition models and algorithms for classification and clustering.

1. Pattern
Classification
Prepared By:
Ranjan Ganguli
Master Of Engineering
UIT, Burdwan

2. Contents
• Introduction
• Pattern Recognition Models
• Pattern Recognition Algorithms
* Classification
• Clustering Algorithm
Pattern Classification……By Ranjan Ganguli

3. Pattern Classification……By Ranjan Ganguli
3
What is a Pattern Recognition?
• The study of how machines can observe the
environment,
• learn to distinguish patterns of interest from
their background, and
• make sound and reasonable decisions about
the categories of the patterns.
• What is a pattern?
• What kinds of category we have?

4. What is a Pattern?
• “A pattern is essentially an arrangement” ………..(Definition 1)
• “It can also be defined by the common denominator among the
multiple instances of an entity”.
e.g., commonality in all fingerprint images defines the
fingerprint pattern; thus, a pattern could be a fingerprint
image, a handwritten cursive word, a human face, a
speech signal ……………………………...(Definition II)
•For example, a pattern could be
• A fingerprint images
• A handwritten cursive word
• A human face
A speech signal …..etc Pattern Classification……By Ranjan Ganguli

5. What is Pattern Category?
• It is a collection of similar, not necessarily
identical objects. Often, individual patterns may
be grouped into a category based on their
common properties; the resultant is also a
pattern and is often called a pattern category.
Pattern Classification……By Ranjan Ganguli

6. Pattern Recognition System
•The design model of a pattern recognition
system essentially involves the following three
steps:
I. Data acquisition and pre-processing
II. Feature extraction
III. Decision making
Pattern Classification……By Ranjan Ganguli

7. Block diagram of a pattern recognition system:
7
Pattern Classification……By Ranjan Ganguli

8. Pattern Recognition Models
•The four best known approaches
• template matching
• statistical classification
• syntactic or structural matching
• neural networks
Pattern Classification……By Ranjan Ganguli

9. Important characteristics of the pattern
recognition models.
Pattern Classification……By Ranjan Ganguli

10. PATTERN RECOGNITION ALGORITHMS
•The design pattern of algorithms consists of three
basic elements, i.e., data perception, feature
extraction and classification.
•Algorithms for pattern recognition depend on the type of
label output, on whether learning is supervised or
unsupervised
Pattern Classification……By Ranjan Ganguli

11. What is a Supervised Learning?
•In supervised learning, there is a teacher
who provides a category label or cost for
each pattern in the training set which is used
as a classifier.
• So basically a supervised learning method
is used for classification purpose.
Pattern Classification……By Ranjan Ganguli

12. • In the given figure, the input image consist of
mixture of two alphabets, i.e., A and B. Then the
classification algorithm classifies the input to two
different categories
Here a set of combined input is classified using
supervised learning approach.
Pattern Classification……By Ranjan Ganguli

13. What is Unsupervised learning?
The system forms clusters or “natural
groupings” of the input patterns
Pattern Classification……By Ranjan Ganguli

14. •Here the input consists of some unlabeled values
whose distinguishing feature is initially not known.
The following input consists of such a combination
with all values technically same but still its clusters
are formed using some metric which is different for
each algorithm
Pattern Classification……By Ranjan Ganguli

15. 15
An Example
• “Sorting incoming Fish on a conveyor according to
species using optical sensing”
Sea bass
Species
Salmon
Pattern Classification……By Ranjan Ganguli

16. 16
• Problem Analysis
• Set up a camera and take some sample images to extract
features
• Length
• Lightness
• Width
• Number and shape of fins
• Position of the mouth, etc…
• This is the set of all suggested features to explore for use in our
classifier!
Pattern Classification……By Ranjan Ganguli

17. 17
• Preprocessing
• Use a segmentation operation to isolate fishes from one
another and from the background
• Information from a single fish is sent to a feature
extractor whose purpose is to reduce the data by
measuring certain features
• The features are passed to a classifier
Pattern Classification……By Ranjan Ganguli

18. 18
Pattern Classification……By Ranjan Ganguli

19. 19
• Classification
• Select the length of the fish as a possible feature for
discrimination
Pattern Classification……By Ranjan Ganguli

20. 20
Pattern Classification……By Ranjan Ganguli

21. 21
The length is a poor feature alone!
Select the lightness as a possible feature.
Pattern Classification……By Ranjan Ganguli

22. 22
Pattern Classification……By Ranjan Ganguli

23. 23
• Threshold decision boundary and cost relationship
• Move our decision boundary toward smaller values of
lightness in order to minimize the cost (reduce the number
of sea bass that are classified salmon!)
Task of decision theory
Pattern Classification……By Ranjan Ganguli

24. 24
• Adopt the lightness and add the width of the fish
Fish xT
= [x1, x2]
Lightness Width
Pattern Classification……By Ranjan Ganguli

25. 25
Pattern Classification……By Ranjan Ganguli

26. 26
• We might add other features that are not correlated
with the ones we already have. A precaution should
be taken not to reduce the performance by adding
such “noisy features”
• Ideally, the best decision boundary should be the one
which provides an optimal performance such as in the
following figure:
Pattern Classification……By Ranjan Ganguli

27. 27
Pattern Classification……By Ranjan Ganguli

28. 28
• However, our satisfaction is premature because
the central aim of designing a classifier is to
correctly classify novel input
Issue of generalization!
Pattern Classification……By Ranjan Ganguli

29. 29
Pattern Classification……By Ranjan Ganguli

30. 30
Pattern Classification……By Ranjan Ganguli

31. CLASSIFICATION ALGORITHMS
(Supervised Learning)
•Decision trees
•Kernel Estimation & K-nearest neighbour(KNn)
•Linear discriminate analysis (LDA)
• Quadratic Discriminate Analysis (QDA)
•Maximum entropy classifier (multinomial logistic regression)
•Naive Bayes classifier
•Artificial Neural Networks
•Support Vector Machine
Pattern Classification……By Ranjan Ganguli

32. Decision Trees
“Splitting datasets one feature at a time”
The decision tree is one of the most commonly used
classification techniques; recent surveys claim that it’s
the most commonly used technique.

34. Advantages:
“Major focus on insights about the data”.

35. Decision tree–building algorithm use
information theory to split the data-set
based on some decisions

36. 1. To build a decision tree, we need to make a
first decision on the dataset to dictate which feature is
used to split the data.
2. To determine this, we try every feature and measure
which split will give you the best results.
3. After that, we’ll split the dataset into subsets.
4. The subsets will then traverse down the branches of
the first decision node. If the data on the branches is
the same class, then you’ve properly classified it and
Steps:

37. 5. If the data isn’t the same, then we need
to repeat the splitting process on this
subset. The decision on how to split this
subset is done the same way as
the original dataset, and we repeat this
process until we’ve classified all the data.

38. Information gain
•We choose to split our dataset in a way
that makes our unorganized data more
organized. One way to organize this is to
measure the information.
•Using information theory, we can measure
the information before and after the split
•The change in information before and after
the split is known as the information
gain.

39. Note:
Highest information gain helps to
split the data set
The attribute with the highest
information gain is chosen as the
splitting

40. Information gain = Entropy
What is entropy?
Entropy is defined as the expected value
of the information.
(Here, it is measured on each attribute)

41. For entropy to calculate, we need the
expected value of all the information
of all possible values of our class.
This is given by:

42. Example to calculate information gain

44. •Next, we need to calculate expected
information gain for each attribute

49. CLUSTERING ALGORITHMS
(Un-supervised Learning)
•Hierarchical Clustering
•K-means Clustering
•KPCA (Kernel Principle Component Analysis)
Pattern Classification……By Ranjan Ganguli

50. Questions?