2013-1 Machine Learning Lecture 01 - Pattern Recognition

Uploaded on


More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Ch01_Introduction to Pattern Recognition (modified from 패턴인식개론/한학용)
  • 2. Contents 01_ Philosophical Debates on AI 02_ Pattern Recognition (PR) 03_ Features and Patterns 04_ Components of PR and Design Cycle 05_ Category of PR and Classifiers 06_ Performance Evaluation of PR Algorithms 07_ Approaches of PR and its Application Areas 08_ Example of PR Applications 2
  • 3. 01_Philosophical Debates on AI Questions  Is computer merely a calculating machine?  Can computer think and understand languages like human? 3
  • 4. 01_Philosophical Debates on AI Positive opinions on the possibility of AIA.M. Turing(1912~1954) Imitation Game Negative opinions on the possibility of AIJohn Searle(1932~ ) Chinese Room Arguments 4
  • 5. 02_Definition of Pattern Recognition What is PR?  An area of AI that deals with the problems to make computable machines (Turing Machines) to recognize certain objects Cognitive Science AI PR 5
  • 6. 03_Features and Patterns What is feature?  Discernible aspects, qualities, characteristics that a certain object has What is pattern?  A set of traits or features of individual objects 6
  • 7. 03_Features and Patterns Easy features and difficult features Categories of patterns 7
  • 8. 04_Components of PR and Design Cycle Components of PR System and its Process 8
  • 9. 04_ Components of PR and Design Cycle Design steps of PR system  Step 1 : Data gathering  Most time-consuming tedious process in PR tasks  Necessary step to ensure stable PR performance  For stable performance, we need to consider how many samples are needed before the gathering.  Step 2 : Feature selection  Essential part regarding PR system’s performance  We need to decide what features to choose through sufficient prior analysis on the object patters.  Step 3 : Model selection  To decide what approach (model and algorithm) is to be constructed and applied  Need prior knowledge on the features  Need to set up parameters for the model according to the approach 9
  • 10. 04_ Components of PR and Design Cycle Step 4 : Learning  Using the feature sets extracted from the collected data and chosen models, the learning algorithm generates or fills up the model (or hypothesis, classifier)  According to the methods, there are supervised learning, unsupervised learning and reinforcement learning. Step 5 : Recognition  Given a new feature set, the generated hypothesis decide a class or category that the feature set belongs to. 10
  • 11. 05_ Category of PR and Classifiers Categories of problems  Classification  In classification problem, the system needs to output one label in a set of finite number of labels.  Regression  Generalized version of classification  Through regression, the PR system will return a real value score (usually between 0 and 1)  Clustering  The problem of organizing a small number of multiple groups from a certain set  The output of clustering system is a set of pairs (example and its class).  The clustering can be processed in an hierarchical manner such as in phylogenetic tree.  Description  The problem of expressing an object using a set of a prototype or primitive terms 11
  • 12. 05_ Category of PR and Classifiers Classifier  Most classification task in PR is done by classifiers  Classification is to partitioning a feature space composed of feature vectors into decision regions of nominal classes.  We call the boundaries of the regions as decision boundaries  Classification of a feature vector x is to decide what decision region the feature vector belongs to, and to assign x to the class that represents the region 12
  • 13. 05_ Category of PR and Classifiers Classifier can be represented as a set of discriminant functions ∀j=i , if 𝑔 𝑖 𝑥 > 𝑔 𝑗 𝑥 , then we decide that the feature vector 𝑥 ∈ class 𝜔 𝑖 13
  • 14. 06_ Performance Evaluation of PR Algorithms Confusion Matrix Actual Positive Actual Negative Predicted Positive TP FP Predicted Negative FN TN TP Recall rate = TP+FN TP Precision = TP+FP TP True Positive Rate (TPR) = TP+FN FP False Positive Rate (FPR) = FP+TN 14
  • 15. 06_ Performance Evaluation of PR Algorithms Receiver operating characteristic (ROC) Curve 15
  • 16. 06_ Performance Evaluation of PR Algorithms AUROC  Ares under the region of ROC Curve  Closer the curve to top-left corner, more accurate the recognition algorithm  The performance can be evaluated by the amount of AUROC 16
  • 17. 07_ Approaches of PR and its Application Areas Approaches of PR  Template matching  Oldest and easiest  First, prepare the template for the object to compare.  Normalize the pattern to recognize for matching it with the template.  And calculate similarity value such as cross-correlation or distance to perform the recognition  Most important task is to prepare the most general template that explains all the samples in a certain category.  Fast running time, but weak in variation of features  Statistical approaches  Decide the class of unknown pattern bases on decision boundaries of pattern sets.  Each of the pattern sets represent a certain class.  The statistical model of the patterns is a probability density function 𝑃 𝑥|𝑐 𝑖 .  Learning is a process of creating a probability density function and calculating its parameters for each class  Neural networks  Model the relation of connection and integration of the biological neurons  Calculate the response process of neural network for input stimulus  Classify patterns based on the responses 17
  • 18. 07_ Approaches of PR and its Application Areas  Knowledge of the patterns is stored as weights that represent the connection strength of synapse.  Learning is performed similar to biological ways, but the learning process is not a serial algorithm.  The learned knowledge is considered as a black box.  Minimal need for prior knowledge.  With sufficient number of neurons, theoretically any complicated decision boundaries can be constructed, so this approach is very attractive. Structural approaches  Instead of quantitative features, we consider the relationship among the basic prototypes what construct the pattern.  Examples: Character, Fingerprint, Chromosome 18
  • 19. 07_ Approaches of PR and its Application Areas Approaches of PR 19
  • 20. 07_ Approaches of PR and its Application Areas Applications of PR  Character recognition  Convert a scanned text image into character codes which can be edited in a computer  Mail classification, Handwriting recognition, Check and banknote recognition, License plate recognition  Biological recognition and human behavioral pattern recognition  Voice recognition, fingerprint recognition, face recognition, DNA mapping, walking pattern analysis and classification, utterance habit analysis and classification  Diagnostic systems  Car malfunction, medical diagnostics, EEG, ECG signal analysis and classification, X- Ray image pattern recognition 20
  • 21. 07_ Approaches of PR and its Application Areas Prediction system  Weather forecasting based on satellite data, earthquake pattern analysis and earthquake prediction, stock price prediction, etc. Security and military area  Intrusion detection based on network traffic pattern analysis, security screening system, search and attack of terrorist camp and targets using satellite terrain image analysis, radar signal classification, Identification Friend or Foe (IFF) 21
  • 22. 07_ Approaches of PR and its Application AreasRelated Areas Application Areas•Adaptive signal processing •Image•Machine learning processing/segmentation•Artificial Neural networks •Computer Vision•Robotics and Vision •Speech recognition•Cognitive science •Automatic target recognition•Mathematical Statistics •Optical character recognition•Nonlinear optimization •Seismic Analysis•Exploratory Data analysis •Man-machine interaction•Fuzzy and Genetic System •Bio recognition (fingerprint,•Detection and Estimation vein, iris)Theory •Industrial inspection•Formal language •Financial forecast•Structural modeling •Medical analysis•Biological cybernetics •ECG signal analysis•Computational neuroscience 22
  • 23. 08_ Example of PR Applications Simple English character recognition system  feature V : # of vertical lines  feature H : # of horizontal lines  feature O : # of slopes  feature C : # of curves Feature Character V H O C L 1 1 0 0 P 1 0 0 1 O 0 0 0 1 E 1 3 0 0 Q 0 0 1 1 23
  • 24. 08_ Example of PR Applications Automatic fish classification (Sea Bass or Salmon)  A: Conveyor belt for fish  B: Conveyor belt for classified fish  C : Robot arm for grabbing fish  D: Machine vision system with CCD camera  E : Computer that analyze fish image and control the robot arm 24
  • 25. 08_ Example of PR Applications Automatic fish classification  Assume that fish is either salmon or sea bass  Using machine vision system for acquiring new fish image  Normalize the intensities of new fish image using image processing algorithm  Segment fish from the background in the image processing process  Using the prior knowledge that sea bass is bigger than salmon, extract features in the image to measure the length of the new fish  From the training samples of the two fish categories, calculate the distribution of the length, and decide the threshold of decision boundary that minimize the classification error Accuracy : 60% 25
  • 26. 08_ Example of PR Applications Adding features for enhancing recognition rate  The accuracy should be over 95% for stable pattern recognition system  We find that average intensity level is a good feature. 26
  • 27. 08_ Example of PR Applications Enhancing the recognition rate  We generate 2 dimensional feature vector with length and average intensity.  Using a simple linear discriminant function, we enhance the recognition rate. Accuracy : 95.7% 27
  • 28. 08_ Example of PR Applications Cost vs. Classification Rate  To minimize the cost, we adjust the decision boundary 28
  • 29. 08_Example of PR Applications Generalization problem  Using neural network, the performance can be enhanced to 99.9975%  Is this a good result? 29