Learning from how dogs learn Prof. Bruce Blumberg The Media Lab, MIT [email_address] www.media.mit.edu/~bruce
About me…
About me…
Practical & compelling real-time learning <ul><ul><li>Easy for interactive characters  to learn what they ought to be able...
My bias & focus <ul><li>Learning occurs within an innate structure that biases… </li></ul><ul><ul><li>Attention </li></ul>...
sheep|dog:trial by eire See sheep|dog video on my website
Object persistence See object persistence video on my website
Temporal representation See temporal representation (aka Goatzilla) video on my website
Alpha Wolf See alpha wolf video on my website
[email_address] See rover@home video on my website or go to Scientific American Frontiers website
Dobie T. Coyote Goes to School See Dobie video on my website
Why look at Dog Training? <ul><li>Interactive characters pose unique challenges: </li></ul><ul><ul><li>State, action and s...
Invaluable resources <ul><li>Doing it, and talking to people who do it. </li></ul><ul><li>Wilkes, Pryor, Ramirez </li></ul...
The problem facing dogs (real and synthetic) Set of all possible actions Set of all motivational goals Set of all possible...
The space of possible stimuli is wicked big Time of Occurence State Space Set of all possible stimuli Smells Motion Sounds...
The space of possible actions is also very big Set of all possible actions Action Time of Performance Action Space Figure ...
Who gets credit for good things happening? Yumm.. Action Figure -8 Shake High -5 Beg Down Left ear twitch Modality of Stim...
Who gets credit for good things happening? Yumm.. Time stalk grab-bite eye orient kill-bite chase
Conventional idea: back propagation from goal stalk grab-bite eye orient kill-bite chase Yumm.. Time Credit flows backward
Conventional idea: back propagation from goal stalk grab-bite eye orient kill-bite chase Yumm.. Time Credit flows backward
Conventional idea: back propagation from goal stalk grab-bite eye orient kill-bite chase Yumm.. Time Credit flows backward
The problem <ul><li>If each element in sequence has 3 variants, there are 729 possible combinations of which 1 may work (i...
Leyhausen’s suggestion… stalk grab-bite eye orient kill-bite chase Time Each element is innately self-motivating and has i...
Leyhausen’s suggestion… stalk grab-bite eye orient kill-bite chase Time Each element is innately self-motivating and has i...
Coppinger’s suggestion… stalk grab-bite eye orient kill-bite chase Time Varying innate tendency to follow behavior with “n...
Functional goal plays incidental role stalk grab-bite eye orient kill-bite chase Time Propagated value from functional goa...
Big idea: innate biases make learning possible  <ul><li>Biases include… </li></ul><ul><ul><li>Temporal Proximity implies c...
Good trainers actively guide dog’s exploration <ul><li>Behavioral </li></ul><ul><ul><li>Train behavior, then cue </li></ul...
Dogs constrain search for causal agents Time Consequences Window: Trainer “clicks” signaling reward is coming.  When rewar...
Dogs use implicit feedback to guide perceptual learning Sit Time “ sit-utterance” perceived.  Approach Eat “ click” percei...
Dogs give credit where credit is due… <ul><li>Trainer repeatedly lures dog through a trajectory or into a pose  </li></ul>...
Observation: dogs give credit where credit is due Sit Time “ sit-utterance” perceived.  Approach Eat “ click” perceived.  ...
D.L.: Take Advantage of Predictable Regularities <ul><li>Constrain search for causal agents by taking advantage of tempora...
D.L.: Make Use of All Feedback: Explicit & Implicit <ul><li>Use rewarded action as context for identifying  </li></ul><ul>...
D.L.: Make Them Easy to Train <ul><li>Respond quickly to “obvious” contingencies </li></ul><ul><li>Support Luring and Shap...
The System
Dobie T. Coyote… See dobie video on my website
Limitations and Future Work <ul><li>Important extensions  </li></ul><ul><ul><li>Other kinds of learning (e.g., social or s...
Useful Insights <ul><li>Use </li></ul><ul><ul><li>Temporal proximity to limit search. </li></ul></ul><ul><ul><li>Hierarchi...
Acknowledgements <ul><li>Members of the Synthetic Characters Group, past, present & future </li></ul><ul><li>Gary Wilkes <...
Upcoming SlideShare
Loading in …5
×

Train Your Dog

387 views

Published on

Informational slide that shows how to train your dog properly and what is needed in order to succeed in your dog training.

Published in: Lifestyle
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
387
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Train Your Dog

  1. 1. Learning from how dogs learn Prof. Bruce Blumberg The Media Lab, MIT [email_address] www.media.mit.edu/~bruce
  2. 2. About me…
  3. 3. About me…
  4. 4. Practical & compelling real-time learning <ul><ul><li>Easy for interactive characters to learn what they ought to be able to learn </li></ul></ul><ul><ul><li>Easy for a human trainer to guide learning process </li></ul></ul><ul><ul><li>A compelling user experience </li></ul></ul><ul><ul><li>Provide heuristics and practical design principles </li></ul></ul>
  5. 5. My bias & focus <ul><li>Learning occurs within an innate structure that biases… </li></ul><ul><ul><li>Attention </li></ul></ul><ul><ul><li>Motivation </li></ul></ul><ul><ul><li>Innate frequency, form and organization of behavior </li></ul></ul><ul><ul><li>When certain things are most easily learned </li></ul></ul><ul><li>What are the catalytic components of the scaffolding that make learning possible? </li></ul>
  6. 6. sheep|dog:trial by eire See sheep|dog video on my website
  7. 7. Object persistence See object persistence video on my website
  8. 8. Temporal representation See temporal representation (aka Goatzilla) video on my website
  9. 9. Alpha Wolf See alpha wolf video on my website
  10. 10. [email_address] See rover@home video on my website or go to Scientific American Frontiers website
  11. 11. Dobie T. Coyote Goes to School See Dobie video on my website
  12. 12. Why look at Dog Training? <ul><li>Interactive characters pose unique challenges: </li></ul><ul><ul><li>State, action and state-action spaces are often continuous and far too big to search exhaustively </li></ul></ul><ul><ul><li>To be compelling characters must </li></ul></ul><ul><ul><ul><li>Learn “obvious” contingencies between state, actions and consequences quickly </li></ul></ul></ul><ul><ul><ul><li>Easy to train without visibility into internal state of character. </li></ul></ul></ul><ul><ul><ul><li>Learning is only one thing they have to do. </li></ul></ul></ul><ul><li>Dogs and their trainers seem to solve these problems easily </li></ul>
  13. 13. Invaluable resources <ul><li>Doing it, and talking to people who do it. </li></ul><ul><li>Wilkes, Pryor, Ramirez </li></ul><ul><li>Lindsay, Burch & Bailey, Mackintosh </li></ul><ul><li>Lorenz, Leyhausen, Coppinger & Coppinger </li></ul>
  14. 14. The problem facing dogs (real and synthetic) Set of all possible actions Set of all motivational goals Set of all possible stimuli What do I do, when, in order to best satisfy my motivational goals?
  15. 15. The space of possible stimuli is wicked big Time of Occurence State Space Set of all possible stimuli Smells Motion Sounds Dog sounds Speech Whistles Modality of Stimuli
  16. 16. The space of possible actions is also very big Set of all possible actions Action Time of Performance Action Space Figure -8 Shake Low shake High -5 Beg Down Left ear twitch
  17. 17. Who gets credit for good things happening? Yumm.. Action Figure -8 Shake High -5 Beg Down Left ear twitch Modality of Stimuli Low shake Motion Sounds Dog sounds Speech Whistles
  18. 18. Who gets credit for good things happening? Yumm.. Time stalk grab-bite eye orient kill-bite chase
  19. 19. Conventional idea: back propagation from goal stalk grab-bite eye orient kill-bite chase Yumm.. Time Credit flows backward
  20. 20. Conventional idea: back propagation from goal stalk grab-bite eye orient kill-bite chase Yumm.. Time Credit flows backward
  21. 21. Conventional idea: back propagation from goal stalk grab-bite eye orient kill-bite chase Yumm.. Time Credit flows backward
  22. 22. The problem <ul><li>If each element in sequence has 3 variants, there are 729 possible combinations of which 1 may work (ignoring stimuli) </li></ul><ul><li>If there are 12 possible stimuli, there are 1,586,874,322,944 possible combinations of stimuli-action pairs to explore. </li></ul><ul><li>Don’t know if it is the right sequence until goal is reached </li></ul><ul><li>What happens if “variant” needs to be learned? </li></ul>
  23. 23. Leyhausen’s suggestion… stalk grab-bite eye orient kill-bite chase Time Each element is innately self-motivating and has innate reward metric motivation & reward motivation & reward motivation & reward motivation & reward motivation & reward motivation & reward
  24. 24. Leyhausen’s suggestion… stalk grab-bite eye orient kill-bite chase Time Each element is innately self-motivating and has innate reward metric motivation & reward motivation & reward motivation & reward motivation & reward motivation & reward motivation & reward
  25. 25. Coppinger’s suggestion… stalk grab-bite eye orient kill-bite chase Time Varying innate tendency to follow behavior with “next” in sequence
  26. 26. Functional goal plays incidental role stalk grab-bite eye orient kill-bite chase Time Propagated value from functional goal plays incidental role Yumm..
  27. 27. Big idea: innate biases make learning possible <ul><li>Biases include… </li></ul><ul><ul><li>Temporal Proximity implies causality </li></ul></ul><ul><ul><li>Attend more readily to certain classes of stimuli than to others (motion vs. speech) </li></ul></ul><ul><ul><li>Lazy discovery (pay attention once you have a reason to pay attention) </li></ul></ul><ul><ul><li>Elements may be “innately” self-motivating and have local metric of “goodness” </li></ul></ul>
  28. 28. Good trainers actively guide dog’s exploration <ul><li>Behavioral </li></ul><ul><ul><li>Train behavior, then cue </li></ul></ul><ul><ul><li>Differential rewards encourage variability </li></ul></ul><ul><li>Motor </li></ul><ul><ul><li>Shaping </li></ul></ul><ul><ul><ul><li>Rewarding successive approximations </li></ul></ul></ul><ul><ul><li>Luring </li></ul></ul><ul><ul><ul><li>Pose, e.g. “down” </li></ul></ul></ul><ul><ul><ul><li>Trajectory, e.g. “figure-8” </li></ul></ul></ul>
  29. 29. Dogs constrain search for causal agents Time Consequences Window: Trainer “clicks” signaling reward is coming. When reward is actually received Attention Window: Cue given immediately before or as dog is moving into desired pose Sit Approach Eat Dogs make the problem tractable by constraining search for causal agents to narrow temporal windows
  30. 30. Dogs use implicit feedback to guide perceptual learning Sit Time “ sit-utterance” perceived. Approach Eat “ click” perceived. Dog decides to sit Build & update perceptual model of “sit-utterance” Dogs use rewarded action to identify potentially promising state to explore and to guide formation of perceptual models
  31. 31. Dogs give credit where credit is due… <ul><li>Trainer repeatedly lures dog through a trajectory or into a pose </li></ul><ul><li>Eventually, dog performs behavior spontaneously </li></ul><ul><li>Implication </li></ul><ul><ul><li>Dog associates reward with resulting body configuration or trajectory and not just with “follow-your nose” </li></ul></ul>
  32. 32. Observation: dogs give credit where credit is due Sit Time “ sit-utterance” perceived. Approach Eat “ click” perceived. Dog decides to sit <ul><li>Credit sitting in presence of “sit-utterance” </li></ul><ul><li>Build & update perceptual model of “sit-utterance” </li></ul>
  33. 33. D.L.: Take Advantage of Predictable Regularities <ul><li>Constrain search for causal agents by taking advantage of temporal proximity & natural hierarchy of state spaces </li></ul><ul><ul><li>Use consequences to bias choice of action </li></ul></ul><ul><ul><li>But vary performance and attend to differences </li></ul></ul><ul><li>Explore state and action spaces on “as-needed” basis </li></ul><ul><ul><li>Build models on demand </li></ul></ul>
  34. 34. D.L.: Make Use of All Feedback: Explicit & Implicit <ul><li>Use rewarded action as context for identifying </li></ul><ul><ul><li>Promising state space and action space to explore </li></ul></ul><ul><ul><li>Good examples from which to construct perceptual models, e.g., </li></ul></ul><ul><ul><ul><li>A good example of a “sit-utterance” is one that occurs within the context of a rewarded Sit. </li></ul></ul></ul>
  35. 35. D.L.: Make Them Easy to Train <ul><li>Respond quickly to “obvious” contingencies </li></ul><ul><li>Support Luring and Shaping </li></ul><ul><ul><li>Techniques to prompt infrequently expressed or novel motor actions </li></ul></ul><ul><li>“ Trainer friendly” credit assignment </li></ul><ul><ul><li>Assign credit to candidate that matches trainer’s expectation </li></ul></ul>
  36. 36. The System
  37. 37. Dobie T. Coyote… See dobie video on my website
  38. 38. Limitations and Future Work <ul><li>Important extensions </li></ul><ul><ul><li>Other kinds of learning (e.g., social or spatial) </li></ul></ul><ul><ul><li>Generalization </li></ul></ul><ul><ul><li>Sequences </li></ul></ul><ul><ul><li>Expectation-based emotion system </li></ul></ul><ul><li>How will the system scale? </li></ul>
  39. 39. Useful Insights <ul><li>Use </li></ul><ul><ul><li>Temporal proximity to limit search. </li></ul></ul><ul><ul><li>Hierarchical representations of state, action and state-action space & use implicit feedback to guide exploration </li></ul></ul><ul><ul><li>“trainer friendly” credit assignment </li></ul></ul><ul><li>Luring and shaping are essential </li></ul>
  40. 40. Acknowledgements <ul><li>Members of the Synthetic Characters Group, past, present & future </li></ul><ul><li>Gary Wilkes </li></ul><ul><li>Funded by the Digital Life Consortium </li></ul>

×