Design principles of pattern recognition system.....

1. Pattern Recognition
UNIT 5
BY:SURBHI SAROHA

2. SYLLABUS
• Introduction
• Design principles of pattern recognition system
• Statistical Pattern recognition
• Parameter estimation methods
• Principle Component Analysis(PCA)
• Linear Discriminant Analysis(LDA)
• Classification Techniques
• Nearest Neighbor(NN) Rule

3. Cont…
• Bayes Classifier
• Support Vector Machine(SVM)
• K-means clustering

4. Pattern Recognition
• Pattern recognition is the automated recognition of patterns and
regularities in data.
• It has applications in statistical data analysis, signal processing, image
analysis, information retrieval, bioinformatics, data
compression, computer graphics and machine learning.
• Pattern recognition has its origins in statistics and engineering; some
modern approaches to pattern recognition include the use of machine
learning, due to the increased availability of big data and a new
abundance of processing power.
• However, these activities can be viewed as two facets of the same field of
application, and together they have undergone substantial development
over the past few decades.
• The field of pattern recognition is concerned with the automatic discovery
of regularities in data through the use of computer algorithms and with
the use of these regularities to take actions such as classifying the data
into different categories.[1]

5. Design principles of pattern
recognition system
• Pattern Recognition System
Pattern is everything around in this digital world.
• A pattern can either be seen physically or it can
be observed mathematically by applying
algorithms.
• In Pattern Recognition, pattern is comprises of
the following two fundamental things:
• Collection of observations
• The concept behind the observation

7. Cont…
• Design Principles of Pattern Recognition
In pattern recognition system, for recognizing the pattern or structure two
basic approaches are used which can be implemented in diferrent
techniques. These are –
– Statistical Approach and
– Structural Approach
• Statistical Approach:
Statistical methods are mathematical formulas, models, and techniques
that are used in the statistical analysis of raw research data.
• The application of statistical methods extracts information from research
data and provides different ways to assess the robustness of research
outputs.
– Two main statistical methods are used :Descriptive Statistics: It summarizes
data from a sample using indexes such as the mean or standard deviation.
– Inferential Statistics: It draw conclusions from data that are subject to random
variation.

8. Statistical Pattern recognition
• Structural Approach:
The Structural Approach is a technique wherein
the learner masters the pattern of sentence.
Structures are the different arrangements of
words in one accepted style or the other.
– Types of structures:Sentence Patterns
– Phrase Patterns
– Formulas
– Idioms

9. Difference Between Statistical
Approach and Structural Approach:

10. Parameter estimation methods
• The term parameter estimation refers to the process of
using sample data (in reliability engineering, usually
times-to-failure or success data) to estimate the
parameters of the selected distribution.
• Several parameter estimation methods are available.
• More specifically, we start with the relatively simple
method of Probability Plotting and continue with the
more sophisticated methods of Rank Regression (or
Least Squares), Maximum Likelihood Estimation and
Bayesian Estimation Methods.

11. Probability Plotting
• The least mathematically intensive method for parameter estimation is
the method of probability plotting.
• As the term implies, probability plotting involves a physical plot of the
data on specially constructed probability plotting paper. This method is
easily implemented by hand, given that one can obtain the appropriate
probability plotting paper.
• The method of probability plotting takes the cdf of the distribution and
attempts to linearize it by employing a specially constructed paper. The
following sections illustrate the steps in this method using the 2-
parameter Weibull distribution as an example. This includes:
• Linearize the unreliability function
• Construct the probability plotting paper
• Determine the X and Y positions of the plot points
• And then using the plot to read any particular time or
reliability/unreliability value of interest.

12. Methods of Parameter Estimation
• The techniques used for parameter estimation are called estimators.
• Some estimators are:
• Probability Plotting: A method of finding parameter values where the data
is plotted on special plotting paper and parameters are derived from the
visual plot
• Rank Regression (Least Squares): A method of finding parameter values
that minimizes the sum of the squares of the residuals.
• Maximum Likelihood Estimation: A method of finding parameter values
that, given a set of observations, will maximize the likelihood function.
• Bayesian Estimation Methods: A family of estimation methods that tries
to minimize the posterior expectation of what is called the utility function.
In practice, what this means is that existing knowledge about a situation is
formulated, data is gathered, and then posterior knowledge is used to
update our beliefs.

13. Principle Component Analysis(PCA)
• Principal component analysis (PCA) is a technique used to
emphasize variation and bring out strong patterns in a dataset.
• It's often used to make data easy to explore and visualize.
• 2D example
• First, consider a dataset in only two dimensions, like (height,
weight).
• This dataset can be plotted as points in a plane.
• But if we want to tease out variation, PCA finds a new coordinate
system in which every point has a new (x,y) value.
• The axes don't actually mean anything physical; they're
combinations of height and weight called "principal components"
that are chosen to give one axes lots of variation.

14. Cont…
• 3D example
• With three dimensions, PCA is more useful, because
it's hard to see through a cloud of data.
• In the example below, the original data are plotted in
3D, but you can project the data into 2D through a
transformation no different than finding a camera
angle: rotate the axes to find the best angle.
• To see the "official" PCA transformation, click the
"Show PCA" button.
• The PCA transformation ensures that the horizontal
axis PC1 has the most variation, the vertical axis PC2
the second-most, and a third axis PC3 the least.

15. Linear Discriminant Analysis(LDA)
• Linear discriminant analysis (LDA), normal discriminant analysis (NDA),
or discriminant function analysis is a generalization of Fisher's linear discriminant,
a method used in statistics and other fields, to find a linear combination of
features that characterizes or separates two or more classes of objects or events.
• The resulting combination may be used as a linear classifier, or, more commonly,
for dimensionality reduction before later classification.
• LDA is closely related to analysis of variance (ANOVA) and regression analysis,
which also attempt to express one dependent variable as a linear combination of
other features or measurements.
• However, ANOVA uses categorical independent variables and
a continuous dependent variable, whereas discriminant analysis has
continuous independent variables and a categorical dependent variable (i.e. the
class label).
• Logistic regression and probit regression are more similar to LDA than ANOVA is, as
they also explain a categorical variable by the values of continuous independent
variables. These other methods are preferable in applications where it is not
reasonable to assume that the independent variables are normally distributed,
which is a fundamental assumption of the LDA method.

16. Cont…
• LDA is also closely related to principal component analysis (PCA)
and factor analysis in that they both look for linear combinations of
variables which best explain the data.
• LDA explicitly attempts to model the difference between the classes of
data.
• PCA, in contrast, does not take into account any difference in class, and
factor analysis builds the feature combinations based on differences
rather than similarities.
• Discriminant analysis is also different from factor analysis in that it is not
an interdependence technique: a distinction between independent
variables and dependent variables (also called criterion variables) must be
made.
• LDA works when the measurements made on independent variables for
each observation are continuous quantities.
• When dealing with categorical independent variables, the equivalent
technique is discriminant correspondence analysis.

17. Classification Techniques
• Various types of classification algorithms:
• Logistic Regression
• Naive Bayes Classifier
• K-Nearest Neighbors
• Decision Tree
– Random Forest
• Support Vector Machines

18. Cont…
• Logistic Regression
• Logistic regression is a calculation used to predict
a binary outcome: either something happens, or
does not. This can be exhibited as Yes/No,
Pass/Fail, Alive/Dead, etc.
• Naive Bayes Classifier
• Naive Bayes calculates the possibility of whether
a data point belongs within a certain category or
does not. In text analysis, it can be used to
categorize words or phrases as belonging to a
preset “tag” (classification) or not.

19. Cont….
• K-nearest Neighbors
• K-nearest neighbors (k-NN) is a pattern
recognition algorithm that uses training datasets
to find the k closest relatives in future examples.
• When k-NN is used in classification, you calculate
to place data within the category of its nearest
neighbor. If k = 1, then it would be placed in the
class nearest 1. K is classified by a plurality poll of
its neighbors.

20. Cont….
• Decision Tree
• A decision tree is a supervised learning algorithm
that is perfect for classification problems, as it’s
able to order classes on a precise level.
• It works like a flow chart, separating data points
into two similar categories at a time from the
“tree trunk” to “branches,” to “leaves,” where the
categories become more finitely similar.
• This creates categories within categories,
allowing for organic classification with limited
human supervision.

22. Random Forest
• The random forest algorithm is an expansion of
decision tree, in that, you first construct some-
axis real-world decision trees with training data,
then fit your new data within one of the trees as
a “random forest.”
• Support Vector Machines
• A support vector machine (SVM) uses algorithms
to train and classify data within degrees of
polarity, taking it to a degree
beyond X/Y prediction.

23. Nearest Neighbor(NN) Rule
• K-Nearest Neighbors is one of the most basic yet
essential classification algorithms in Machine Learning.
• It belongs to the supervised learning domain and finds
intense application in pattern recognition, data mining
and intrusion detection.
• It is widely disposable in real-life scenarios since it is
non-parametric, meaning, it does not make any
underlying assumptions about the distribution of data
(as opposed to other algorithms such as GMM, which
assume a Gaussian distribution of the given data).

24. Cont…
• In statistics, the k-nearest neighbors algorithm (k-NN) is a non-
parametric machine learning method first developed by Evelyn
Fix and Joseph Hodges in 1951,and later expanded by Thomas
Cover.
• It is used for classification and regression.
• In both cases, the input consists of the k closest training examples
in feature space. The output depends on whether k-NN is used for
classification or regression:
• In k-NN classification, the output is a class membership. An object is
classified by a plurality vote of its neighbors, with the object being
assigned to the class most common among its k nearest neighbors
(k is a positive integer, typically small). If k = 1, then the object is
simply assigned to the class of that single nearest neighbor.
• In k-NN regression, the output is the property value for the object.
This value is the average of the values of k nearest neighbors.

25. Bayes Classifier
• A Naive Bayes classifier is a probabilistic machine
learning model that’s used for classification task.
• The crux of the classifier is based on the Bayes
theorem.
• Bayes Theorem:
• Using Bayes theorem, we can find the probability
of A happening, given that B has occurred. Here, B is
the evidence and A is the hypothesis. The assumption
made here is that the predictors/features are
independent. That is presence of one particular feature
does not affect the other. Hence it is called naive.

26. Types of Naive Bayes Classifier:
• Multinomial Naive Bayes:
• This is mostly used for document classification problem, i.e whether
a document belongs to the category of sports, politics, technology
etc. The features/predictors used by the classifier are the frequency
of the words present in the document.
• Bernoulli Naive Bayes:
• This is similar to the multinomial naive bayes but the predictors are
boolean variables. The parameters that we use to predict the class
variable take up only values yes or no, for example if a word occurs
in the text or not.
• Gaussian Naive Bayes:
• When the predictors take up a continuous value and are not
discrete, we assume that these values are sampled from a gaussian
distribution.

27. Support Vector Machine(SVM)
• In machine learning, support-vector machines (SVMs, also support-
vector networks) are supervised learning models with associated
learning algorithms that analyze data
for classification and regression analysis.
• Developed at AT&T Bell Laboratories by Vapnik with colleagues
(Boser et al., 1992, Guyon et al., 1993, Vapnik et al., 1997), SVMs
are one of the most robust prediction methods, being based on
statistical learning frameworks or VC theory proposed by Vapnik
and Chervonenkis (1974) and Vapnik (1982, 1995).
• Given a set of training examples, each marked as belonging to one
of two categories, an SVM training algorithm builds a model that
assigns new examples to one category or the other, making it a
non-probabilistic binary linear classifier (although methods such
as Platt scaling exist to use SVM in a probabilistic classification
setting).

28. Cont…
• An SVM maps training examples to points in space so as to
maximise the width of the gap between the two categories.
• Support vector machines (SVMs) are powerful yet flexible
supervised machine learning algorithms which are used
both for classification and regression.
• But generally, they are used in classification problems.
• In 1960s, SVMs were first introduced but later they got
refined in 1990.
• SVMs have their unique way of implementation as
compared to other machine learning algorithms.
• Lately, they are extremely popular because of their ability
to handle multiple continuous and categorical variables.

29. Working of SVM
• An SVM model is basically a representation of
different classes in a hyperplane in
multidimensional space.
• The hyperplane will be generated in an
iterative manner by SVM so that the error can
be minimized.
• The goal of SVM is to divide the datasets into
classes to find a maximum marginal
hyperplane (MMH).

30. The followings are important concepts
in SVM −
• Support Vectors − Datapoints that are closest to the
hyperplane is called support vectors. Separating line
will be defined with the help of these data points.
• Hyperplane − As we can see in the above diagram, it is
a decision plane or space which is divided between a
set of objects having different classes.
• Margin − It may be defined as the gap between two
lines on the closet data points of different classes. It
can be calculated as the perpendicular distance from
the line to the support vectors. Large margin is
considered as a good margin and small margin is
considered as a bad margin.

32. K-means clustering
• We are given a data set of items, with certain
features, and values for these features (like a
vector).
• The task is to categorize those items into
groups.
• To achieve this, we will use the kMeans
algorithm; an unsupervised learning
algorithm.

33. Cont….
• (It will help if you think of items as points in an n-
dimensional space). The algorithm will categorize the
items into k groups of similarity. To calculate that
similarity, we will use the euclidean distance as
measurement.
• The algorithm works as follows:
• First we initialize k points, called means, randomly.
• We categorize each item to its closest mean and we
update the mean’s coordinates, which are the averages
of the items categorized in that mean so far.
• We repeat the process for a given number of iterations
and at the end, we have our clusters.

34. The above algorithm in pseudocode:
• Initialize k means with random values
• For a given number of iterations:
• Iterate through items:
• Find the mean closest to the item
• Assign item to mean
• Update mean

35. • Thank you 