CS494/594, Fall  2007
  11:10 AM – 12:25 PM
       Claxton 205



Machine Learning

                    Slides adapted (an...
What is Learning? and Why Learn ?
 Machine learning is programming computers to optimize a
 performance criterion using ex...
What We Talk About When We Talk
About“Learning”
Learning general models from data of particular examples
Data is cheap and...
Data Mining: Application of Machine
       Learning to Large Databases
(also called “Knowledge Discovery in Databases (KDD...
Relevant Disciplines for Machine
Learning
Artificial Intelligence
Bayesian methods
Computational complexity theory
Control...
Some Types of Machine Learning
 Learning Associations: Find
 relationships in the data
 Supervised Learning: We want to
 l...
Learning Associations
Example: Shopping basket analysis
P (Y | X ) probability that somebody who buys X also buys Y
where ...
Classification
(a type of supervised
learning)
  Example: Credit scoring
  Differentiating between
  low-risk and high-ris...
Classification: Applications
Also known as: Pattern recognition
  Face recognition: Pose, lighting, occlusion (glasses, be...
Example Pattern Recognition:
   Face Recognition
Training examples of a person




 Test images




                      ...
Classification: Applications
Also known as: Pattern recognition
  Face recognition: Pose, lighting, occlusion (glasses, be...
Example Pattern Recognition:
    Character Recognition

Want to learn how to
recognize characters,
even if written in diff...
Classification: Applications
Also known as: Pattern recognition
  Face recognition: Pose, lighting, occlusion (glasses, be...
Example Pattern Recognition:
   Speech Recognition




                               14
Classification: Applications
Also known as: Pattern recognition
  Face recognition: Pose, lighting, occlusion (glasses, be...
Example Pattern Recognition:
   Gesture Recognition




                               16
Classification: Applications
Also known as: Pattern recognition
  Face recognition: Pose, lighting, occlusion (glasses, be...
Example Pattern Recognition:
    Medical Diagnosis

Inputs: relevant info
about patient, symptoms,
test results, etc.

Out...
Classification: Applications
Also known as: Pattern recognition
  Face recognition: Pose, lighting, occlusion (glasses, be...
Example Pattern Recognition:
   Interpreting Brainwaves
 EEG electrodes reading brain waves:                    Rotation t...
Classification: Applications
Also known as: Pattern recognition
  Face recognition: Pose, lighting, occlusion (glasses, be...
Example Pattern Recognition:
   Reading text
 Can you read this?
    Aircndcog to a rseerhcaer at Cbiardmge Urensvitiy, it...
Regression
       (another type of supervised learning)
Example:                             Here, a linear regression fun...
Example Regression Applications
 Navigating a car: Angle of the steering wheel (CMU NavLab)
 Kinematics of a robot arm
   ...
Supervised Learning: Handy Uses
Prediction of future cases: Use the rule to predict the output for
future inputs

Knowledg...
Unsupervised Learning
 Learning “what normally happens”
 No output available (i.e., we don’t know the “right” answer)
 Clu...
Reinforcement Learning
 Learning a policy: A sequence of actions to take, given the
 current state
 No supervised output, ...
Where is Machine Learning Headed?
 Today: tip of the iceberg
    First-generation algorithms: neural networks, decision tr...
Resources: Datasets
 UCI Repository: http://www.ics.uci.edu/~mlearn/MLRepository.html
 UCI KDD Archive:
 http://kdd.ics.uc...
Resources: Journals
 Journal of Machine Learning Research
 Machine Learning
 Neural Computation
 Neural Networks
 IEEE Tra...
Resources: Conferences
 International Conference on Machine Learning (ICML)
      ICML07: http://oregonstate.edu/conferenc...
Our First Learning Study:
    Neural Networks

But first, we’ll look at some general
issues in designing a machine learnin...
Upcoming SlideShare
Loading in …5
×

Introduction to Machine Learning

1,767 views
1,694 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,767
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
30
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Introduction to Machine Learning

  1. 1. CS494/594, Fall  2007 11:10 AM – 12:25 PM Claxton 205 Machine Learning Slides adapted (and extended) from: ETHEM ALPAYDIN © The MIT Press, 2004 alpaydin@boun.edu.tr http://www.cmpe.boun.edu.tr/~ethem/i2ml
  2. 2. What is Learning? and Why Learn ? Machine learning is programming computers to optimize a performance criterion using example data or past experience. Learning is used when: Human expertise does not exist (navigating on Mars), Humans are unable to explain their expertise (speech recognition) Solution changes in time (routing on a computer network) Solution needs to be adapted to particular cases (user biometrics) But, not always appropriate For example, there is no need to “learn” to calculate payroll 2
  3. 3. What We Talk About When We Talk About“Learning” Learning general models from data of particular examples Data is cheap and abundant (data warehouses, data marts); knowledge is expensive and scarce. Example in retail: Customer transactions to consumer behavior: People who bought “Da Vinci Code” also bought “The Five People You Meet in Heaven” (www.amazon.com) Build a model that is a good and useful approximation to the data. 3
  4. 4. Data Mining: Application of Machine Learning to Large Databases (also called “Knowledge Discovery in Databases (KDD)”) Retail: Market basket analysis, Customer relationship management (CRM) Finance: Credit scoring, fraud detection Manufacturing: Optimization, troubleshooting Medicine: Medical diagnosis Telecommunications: Quality of service optimization Bioinformatics: Motifs, alignment Web mining: Search engines ... 4
  5. 5. Relevant Disciplines for Machine Learning Artificial Intelligence Bayesian methods Computational complexity theory Control theory Information theory Statistics Philosophy Psychology … 5
  6. 6. Some Types of Machine Learning Learning Associations: Find relationships in the data Supervised Learning: We want to learn a mapping from the input to the output; correct values are provided by supervisor Classification Regression Unsupervised Learning: We have only input data; we want to find regularities in the data. Reinforcement Learning: Learn a policy that maps states to actions. 6
  7. 7. Learning Associations Example: Shopping basket analysis P (Y | X ) probability that somebody who buys X also buys Y where X and Y are products/services. We learn Association Rule: P ( chips | soda ) = 0.7 Use this Association Rule like this: Target customers who bought X, but not Y Try to convince them to buy Y 7
  8. 8. Classification (a type of supervised learning) Example: Credit scoring Differentiating between low-risk and high-risk customers from their income and savings Discriminant: IF income > θ1 AND savings > θ2 THEN low-risk ELSE high-risk Main application: prediction 8
  9. 9. Classification: Applications Also known as: Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Use of a dictionary or the syntax of the language. Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic for speech Gesture recognition: Different hand shapes. Medical diagnosis: From symptoms to illnesses. Brainwave understanding: From signals to “states” of thought Reading text: … 9
  10. 10. Example Pattern Recognition: Face Recognition Training examples of a person Test images AT&T Laboratories, Cambridge UK http://www.uk.research.att.com/facedatabase.html 10
  11. 11. Classification: Applications Also known as: Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Use of a dictionary or the syntax of the language. Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic for speech Gesture recognition: Different hand shapes. Medical diagnosis: From symptoms to illnesses. Brainwave understanding: From signals to “states” of thought Reading text: … 11
  12. 12. Example Pattern Recognition: Character Recognition Want to learn how to recognize characters, even if written in different ways by different people 12
  13. 13. Classification: Applications Also known as: Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Use of a dictionary or the syntax of the language. Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic for speech Gesture recognition: Different hand shapes. Medical diagnosis: From symptoms to illnesses. Brainwave understanding: From signals to “states” of thought Reading text: … 13
  14. 14. Example Pattern Recognition: Speech Recognition 14
  15. 15. Classification: Applications Also known as: Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Use of a dictionary or the syntax of the language. Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic for speech Gesture recognition: Different hand shapes. Medical diagnosis: From symptoms to illnesses. Brainwave understanding: From signals to “states” of thought Reading text: … 15
  16. 16. Example Pattern Recognition: Gesture Recognition 16
  17. 17. Classification: Applications Also known as: Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Use of a dictionary or the syntax of the language. Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic for speech Gesture recognition: Different hand shapes. Medical diagnosis: From symptoms to illnesses. Brainwave understanding: From signals to “states” of thought Reading text: … 17
  18. 18. Example Pattern Recognition: Medical Diagnosis Inputs: relevant info about patient, symptoms, test results, etc. Output: Expected illness or risk factors 18
  19. 19. Classification: Applications Also known as: Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Use of a dictionary or the syntax of the language. Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic for speech Gesture recognition: Different hand shapes. Medical diagnosis: From symptoms to illnesses. Brainwave understanding: From signals to “states” of thought Reading text: … 19
  20. 20. Example Pattern Recognition: Interpreting Brainwaves EEG electrodes reading brain waves: Rotation task, left brain Rotation task, Resting task, right brain Counting task with eye blink 20
  21. 21. Classification: Applications Also known as: Pattern recognition Face recognition: Pose, lighting, occlusion (glasses, beard), make-up, hair style Character recognition: Different handwriting styles. Speech recognition: Temporal dependency. Use of a dictionary or the syntax of the language. Sensor fusion: Combine multiple modalities; eg, visual (lip image) and acoustic for speech Gesture recognition: Different hand shapes. Medical diagnosis: From symptoms to illnesses. Brainwave understanding: From signals to “states” of thought Reading text: … 21
  22. 22. Example Pattern Recognition: Reading text Can you read this? Aircndcog to a rseerhcaer at Cbiardmge Urensvitiy, it dsoen't mtetar in waht oderr the letrtes in a wrod are, the olny ipnaotmrt tihng is taht the fsrit and lsat lteter be at the rgiht plcae. The rset can be a toatl mses and you can slitl raed it wutohit porlebm. Tehy spectluae taht tihs is bseuace the hmaun mnid deos not raed erevy leettr by iesltf but the wrod as a whloe. Wtehehr tihs is ture or not is a ponit of deabte. Clearly, the brain has learned syntax and semantics of language, including contextual dependencies, to make sense of of this ☺ For fun: Here’s a web page where you can create your own jumbled text: http://www.stevesachs.com/jumbler.cgi 22
  23. 23. Regression (another type of supervised learning) Example: Here, a linear regression function: Predict price of a used car (Input) x : car attributes (e.g., mileage) (Output) y : price y = wx+w0 Our task: learn the mapping from input to output We know basic g ( ) model y:  price We want to learn appropriate values for θ parameters that minimize the error in the approximation: y = g (x | θ ) 23 x:  mileage
  24. 24. Example Regression Applications Navigating a car: Angle of the steering wheel (CMU NavLab) Kinematics of a robot arm (x,y) α1= g1(x,y) α2= g2(x,y) α2 α1 Response surface design (using function optimization) 24
  25. 25. Supervised Learning: Handy Uses Prediction of future cases: Use the rule to predict the output for future inputs Knowledge extraction: We can deduce an explanation about the process underlying the data Compression: The rule is simpler than the data it explains Outlier detection: We can find instances that do not obey the rule, and are thus exceptions (e.g., to detect fraud) 25
  26. 26. Unsupervised Learning Learning “what normally happens” No output available (i.e., we don’t know the “right” answer) Clustering (density estimation): Grouping similar instances Example applications: Customer segmentation in CRM (Customer Relationship Management) Company may have different marketing approaches for different groupings of customers Image compression: Color quantization Instead of using 24 bits to represent 16 million colors, reduce to 6 bits and 64 colors, if the image only uses those 64 colors. Bioinformatics: Learning motifs (i.e., sequences of amino acids that occur repeatedly in proteins) 26
  27. 27. Reinforcement Learning Learning a policy: A sequence of actions to take, given the current state No supervised output, but delayed reward is provided Credit assignment problem Game playing Robot in a maze Multiple agents, partial observability, ... 27
  28. 28. Where is Machine Learning Headed? Today: tip of the iceberg First-generation algorithms: neural networks, decision trees, regression… Applied to well-formatted databases Budding industry Opportunity for tomorrow: enormous impact Learn across full mixed-media data Learn across multiple internal databases, plus the web and newsfeeds Learn by active experimentation Learn decisions rather than predictions Cumulative, lifelong learning Programming languages with learning embedded? 28
  29. 29. Resources: Datasets UCI Repository: http://www.ics.uci.edu/~mlearn/MLRepository.html UCI KDD Archive: http://kdd.ics.uci.edu/summary.data.application.html Statlib: http://lib.stat.cmu.edu/ Delve: http://www.cs.utoronto.ca/~delve/ 29
  30. 30. Resources: Journals Journal of Machine Learning Research Machine Learning Neural Computation Neural Networks IEEE Transactions on Neural Networks IEEE Transactions on Pattern Analysis and Machine Intelligence Annals of Statistics Journal of the American Statistical Association ... 30
  31. 31. Resources: Conferences International Conference on Machine Learning (ICML) ICML07: http://oregonstate.edu/conferences/icml2007/ European Conference on Machine Learning (ECML) and European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD) ECML/PKDD07: http://www.ecmlpkdd2007.org/ Neural Information Processing Systems (NIPS) NIPS07: http://nips.cc/ Uncertainty in Artificial Intelligence (UAI) UAI07: http://www.cs.duke.edu/uai07/ Computational Learning Theory (COLT) COLT07: http://www.learningtheory.org/colt2007/ International Joint Conference on Artificial Intelligence (IJCAI) IJCAI07: http://www.ijcai-07.org/ International Conference on Neural Networks (Europe) ICANN07: http://www.icann2007.org/ ... 31
  32. 32. Our First Learning Study: Neural Networks But first, we’ll look at some general issues in designing a machine learning system For next time, read chapter 1 (if time allows, also start reading chapter 4) First project topic we’re working toward (after introduction): Implementing a neural network for character recognition 32

×