This document provides an introduction and overview of machine learning and pattern recognition. It discusses what machine learning and pattern recognition are, provides examples of typical pattern recognition systems including decision trees and neural networks, and lists some resources like datasets, journals, and conferences in the field.

1. Introduction to Machine Learning CAP 4630 Xingquan (Hill) Zhu

2. Outline What is Machine Learning? What is Pattern Recognition? Typical Pattern Recognition Systems Resource & References Decision Trees Neural Networks

3. Face Detection Demo http://demo.pittpatt.com/ http://graphics8.nytimes.com/images/2008/03/17/us/17bush-600.jpg http://img.timeinc.net/time/daily/special/photo/siralec/faces.jpg http://demo.pittpatt.com/detection_demo/

4. More Complicated Examples Given The following Examples Which of the following person belong to this small group? Machine Learning Yes or No! How much confidence? Fst: Yes, 90% Snd: No, 70% Thd: No, 90% Fth: No, 80%

5. Neural Networks Applications

6. ALVINN drives 70mph on highways CMU

7. More Complicated Example Regression Example: Price of a used car x : car attributes y : price y = g ( x | θ  ) g ( ) model,  θ parameters y = wx + w 0 http://www.theparticle.com/applets/ml/index.html

8. Outline What is Machine Learning? What is Pattern Recognition? Typical Pattern Recognition Systems Resource & References

9. What is Machine Learning? Humans have developed highly sophisticated skills for sensing their environment and taking actions according to what they observe, e.g., Recognizing a face Understanding spoken words Reading handwriting Distinguishing fresh food from its smell We would like to give similar capabilities to machines

10. What is Machine Learning Programming computers to use example data or past experience Needed in cases where we cannot directly write a computer program but have example data Learning is used when: Human expertise does not exist (navigating on Mars), Humans are unable to explain their expertise (speech recognition) Solution changes in time (routing on a computer network) Solution needs to be adapted to particular cases (user biometrics) Are all problems learnable?

11. “ Learning” … Learning general models from the data of particular examples Data is usually cheap and abundant (data warehouses, data marts); knowledge is expensive and scarce. Data scarcity, learning is possible but knowledge is less reliable Example in retail: Customer transactions to consumer behavior: People who bought “Da Vinci Code” also bought “The Five People You Meet in Heaven” (www.amazon.com) Build a model that is a good and useful approximation to the data.

12. Machine Learning: Creating a Classifier Adaptively Supervised learning Decision Trees Feedforward neural network and backpropagation Unsupervised learning Clustering Grouping similar instances Association analysis People who bought “Da Vinci Code” also bought “The Five People You Meet in Heaven” (www.amazon.com) Reinforcement learning

13. Machine Learning Output The output of Machine learning Patterns Decision trees, data summarization, data generative models Discriminative machines Neural networks (explicit rules are not available) The requirements of the output crucially determine the underlying learning models Class category Confidence (probability) The comprehensibility of the decision model The process of Learning is the process of pattern discovery

14. Outline What is Machine Learning? What is Pattern Recognition? Typical Pattern Recognition Systems Resource & References

15. What is Pattern Recognition A pattern is an entity, vaguely defined, that could be given a name, e.g., fingerprint image handwritten word human face speech signal DNA sequence Pattern recognition is the study of how machines can observe the environment learn to distinguish patterns of interest make sound and reasonable decisions about the categories of the patterns

16. A Simple Example

17. Typical Decision Process Fish Face Recognition? Salmon tastes better? What kind of information can distinguish one species from the others? length, width, weight, number and shape of fins, tail shape, etc. What can cause problems during sensing? lighting conditions, position of fish on the conveyor belt, camera noise, etc. What are the steps in the process? capture image -> isolate fish -> take measurements -> make decision

18. Example: Feature Selection Assume a fisherman (domain knowledge) told us that a sea bass is generally longer than a salmon. We can use length as a feature and decide between sea bass and salmon according to a threshold on length. How can we choose this threshold?

19. Example: Feature Selection

20. Example: Feature Selection Even though sea bass is longer than salmon on the average, there are many examples of fish where this observation does not hold. Try another feature: average lightness of the fish scales.

21. Example: Feature Selection

22. Example: Multiple Features Assume we also observed that sea bass are typically wider than salmon. We can use two features in our decision: lightness: x1 width: x2 Each fish image is now represented as a point ( feature vector ) in a two-dimensional feature space

23. Example: Multiple Features

24. Example: Multiple Features Feature vector - A vector of observations (measurements). - is a point in feature space . Hidden state - Cannot be directly measured. - Patterns with equal hidden state belong to the same class. Task - To design a classifer (decision rule) which decides about a hidden state based on an onbservation. Pattern

25. Example: Multiple Features Does adding more features always improve the results? Avoid unreliable features. Be careful about correlations with existing features. Be careful about measurement costs. Be careful about noise in the measurements. Is there some curse for working in very high dimensions?

26. Example: Decision Boundaries

27. Overfitting and underfitting underfitting overfitting good fit

28. Example: Decision Boundaries

29. Example: Cost of Error We should also consider costs of different errors we make in our decisions. For example, if the fish packing company knows that: Customers who buy salmon will object vigorously if they see sea bass in their cans. Customers who buy sea bass will not be unhappy if they occasionally see some expensive salmon in their cans. How does this knowledge affect our decision?

30. The actual forms of the decision model (Patterns) Decision Rules IF lightness>8 AND length>14 THEN Salmon Decision Trees A tree structure to make decision Probability Models Gaussian models Bayesian decision models Weight functions Neural Networks Linear Discriminate Analysis

31. Outline What is Machine Learning? What is Pattern Recognition? Typical Pattern Recognition Systems Resource & References Interesting Demo Story picturing: http://alipr.com/spe/

32. Resources: Datasets UCI Repository: http://www.ics.uci.edu/~mlearn/MLRepository.html UCI KDD Archive: http ://kdd.ics.uci.edu/summary.data.application.html Statlib: http://lib.stat.cmu.edu/ Delve: http://www.cs.utoronto.ca/~ delve/

33. Resources: Journals Journal of Machine Learning Researc Machine Learning Neural Computation Neural Networks IEEE Transactions on Neural Networks IEEE Transactions on Pattern Analysis and Machine Intelligence Annals of Statistics Journal of the American Statistical Association IEEE Trans. On Knowledge and Data Engineering Data Mining and Knowledge Discovery ...

34. Resources: Conferences International Joint Conference on Artificial Intelligence (IJCAI) I nternational Conference on Machine Learning (ICML) Neural Information Processing Systems (NIPS) American Association for Artificial Intelligence (AAAI) Uncertainty in Artificial Intelligence (UAI) International Conference on Neural Networks (Europe) ACM Knowledge Discovery and Data Mining (KDD) IEEE International Conference on Data Mining (ICDM) ...

35. Outline What is Machine Learning? What is Pattern Recognition? Typical Pattern Recognition Systems Resource & References Decision Trees Neural Networks