Ted Selker - Singularity University - 6 July 2010


Published on

1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This next project is about shooting home video or personal video. The idea is that people shoot a lot of video and often end up with hours of footage to go through and edit which is a laborious process, so we wanted to try to do something to augment and assist this editing process by doing something to automatically index interesting parts of the video. So we wired up the camera operator (me) with some sensors to see if we could tell what video sequences might be interesting later. So, we decided that laughter might be a good indicator of interest and built a laughter recognition system that the camera operator wore around while shooting. We also used the galvactivator to record skin conductivity, and were taking video of the user’s facial expressions.
  • A low level layer includes implicit low granularity information such as key-strokes and mouse movement activity. An intermediate layer includes the activities and information to which some of the low granularity data can be extracted and summarized, such as reading, walking, between tasks, paying attention, etc. And a top layer or knowledge layer includes the information or concepts related to the user goals. A disruption manager based on this three-layer architecture will monitor the user state (current activity), concepts surrounding the user’s goals: history of recently accessed documents, web pages and search queries, the interrupting message relevancy to these concepts, and concept priority and importance. The manager will then identify interrupting messages that should be allowed to interrupt the user or delayed to an appropriate time within task execution.
  • Ted Selker - Singularity University - 6 July 2010

    1. 1. User System Ergonomic Research IBM Almaden Research Center, Stanford University Physical, Graphical, & Cognitive Human Computer Interface MIT Media Lab Context Aware Group 1999 - 2008 CMU Silicon Valley Business mentoring The Generator Fund TED SELKER ALU Reg D M T L B
    2. 2. HCI Field <ul><li>Input techniques </li></ul><ul><ul><li>Direct Manipulation: Select, steer, multidimensional,, orientation, location, </li></ul></ul><ul><ul><li>Audio: sound, voice </li></ul></ul><ul><ul><li>Camera: features, figures, face… </li></ul></ul><ul><li>Output techniques </li></ul><ul><ul><li>Screen: text, GUI, Ecological, abstract </li></ul></ul><ul><ul><li>Audio </li></ul></ul><ul><ul><li>Physical; paper, tangible, robot </li></ul></ul><ul><li>User modeling </li></ul><ul><ul><li>Task, User, System </li></ul></ul><ul><li>Typical scenarios </li></ul><ul><ul><li>Office, data, teaching, collaboration, control </li></ul></ul><ul><li>Evaluation </li></ul><ul><ul><li>Demonstration,, Systematic (Preference, Performance), Ethnographic, … </li></ul></ul>
    3. 3. Finger Control Everywhere! <ul><li>In Keyboard Pointing </li></ul><ul><li>Remote Controls </li></ul><ul><li>Air Traffic Control </li></ul><ul><li>Surgical Instruments </li></ul><ul><li>Sightless Pointing </li></ul><ul><li>Prosthetics pointing </li></ul><ul><li>Steering </li></ul><ul><li>Two Handed </li></ul><ul><li>Industrial Controllers </li></ul><ul><li>Arcade Games </li></ul><ul><li>In Mouse Scrolled </li></ul><ul><li>Multi-touch the endgame? </li></ul>TrackPoint
    4. 4. Graphical interface mappings T1 T2 T3 T4 Day 1 0 2.5 5 7.5 10 12.5 15 17.5 20 Sec.
    5. 5. Towards a Behavioral Motor Match <ul><li>Placement </li></ul><ul><li>Eye Tracking </li></ul><ul><li>Wiggly Fingers </li></ul><ul><li>Going Fast </li></ul><ul><li>Movement Feedback </li></ul><ul><li>Gripiness </li></ul><ul><li>Mouse Lockout </li></ul>Select and Type Sensors:2d 3d, linear, eye motion…
    6. 6. HCI field: things to learn Card & Moran, …,ACM CHI, Interact, IUI <ul><li>Cognitive science </li></ul><ul><ul><li>Social, behavioral, cognitive, perception </li></ul></ul><ul><li>Statistics </li></ul><ul><li>Design </li></ul><ul><ul><li>Product design, interaction design </li></ul></ul><ul><ul><li>ME, EE, Materials </li></ul></ul><ul><li>Computer Science </li></ul>
    7. 7. Mobile Productivity tools
    8. 8. Mobile Communication tools
    9. 9. Mobile Content tools
    10. 10. Mobile Personal management
    11. 11. Ted Selker Associate Director, Cylab Mobility Research <ul><li>NSF FODAVA with Ole Mengshoel </li></ul><ul><ul><li>Multifocus multi view network visualization: Michele Cossalter </li></ul></ul><ul><li>MS RA project MobileEssenceII </li></ul><ul><ul><li>ACMCHI, CTS2010, HotMobile2010 Shoou-Jong Yu </li></ul></ul><ul><li>MS RA project Remember and Compare </li></ul><ul><ul><li>Sona Chaudhuri ,Russel Reed </li></ul></ul><ul><li>Tremor as a metric of health </li></ul><ul><ul><li>Will Dayton, Patricia Collins </li></ul></ul><ul><li>Creative Input … </li></ul>
    12. 12. Understanding Considerate Systems UCS Ted Selker, Carnegie Mellon University Silicon Valley
    13. 13. Context Aware Computing group Ted Selker, Associate Professor MIT Media lab Using Sensors and Virtual sensors To understand and respect human intention
    14. 14. Context Aware Platforms Demonstrations and design tools for recognizing and respecting intention across domains and scenariosT ed Selker [email_address] <ul><li>Annotated smart kitchen Recognizing and coaching activity in kitchen </li></ul><ul><li>Alzheimer's Living Center Adaptive interface to support communication for people with dementia </li></ul><ul><li>Attention vending machine Socially interpreted interactive media in a vending machine </li></ul><ul><li>Audio interface tool Tool for exploring audio interface for voting and telephones </li></ul><ul><li>Attention meter Camera input for interactive </li></ul><ul><li>Bike alert Auto system for telling bike car door might open </li></ul><ul><li>Car coach Feedback to improve driving </li></ul><ul><li>Climbing interface Platform for museum interactive </li></ul><ul><li>Context builder Context aware application engine </li></ul><ul><li>Considerate thing Sensor/ electronics platform for making considerate things </li></ul><ul><li>Chameleon tables Computer interactive tables with height control and sensing </li></ul><ul><li>Digital cigarette Motivation and relationship building demonstration </li></ul><ul><li>Disruption manager Semantic based system to mediate desktop interactions </li></ul><ul><li>Driftcatcher Socially aware email annotation interactive </li></ul><ul><li>Dishmaker Recycling kitchen manufacturing </li></ul><ul><li>e-Bed Eye gesture based GUI </li></ul><ul><li>e-Clay 24 degree of freedom design input tool </li></ul><ul><li>e-Floor Socially aware floor interactive </li></ul><ul><li>e-Helmet Wearable for mediating communication for bicyclist </li></ul><ul><li>e-Threshold Context aware receptionist </li></ul><ul><li>EyeaRe Glasses that can tell interest alertness </li></ul><ul><li>Exercar Pedaling accelerator improves acuity and reduces fatigue </li></ul><ul><li>Face Interface Caricature teaching of facial gesture interface affordances </li></ul><ul><li>Gesture music ball Instrument builds itself for you </li></ul><ul><li>Haptic tuner Adaptive feedback to coordinate and improve peoples musical collaboration </li></ul><ul><li>Interruption Manager GUI mediation for any platform </li></ul><ul><li>Kitchen phone System to control and enhance grocery experiences </li></ul><ul><li>Invision Eye gesture based preference system </li></ul><ul><li>Low Error Voting Interface New approach to ballot design for improved access and accuracy </li></ul><ul><li>Media jukebox Model and feedback smart interactive for selection of media </li></ul><ul><li>Media windshield Augmented reality car </li></ul><ul><li>Minerva Viewing food to suggest recipes </li></ul><ul><li>Mobile essence Meeting collaboration support system </li></ul><ul><li>MrWeb, Collaborative web tools </li></ul><ul><li>PlaceMap Location aware engine </li></ul><ul><li>Power bra Power harvesting system collects 1 watt from breathing </li></ul><ul><li>Secure Architecture For Voting Electronically New approach to reliability/security </li></ul><ul><li>SMS mediator No cell phone software meeting mediator </li></ul><ul><li>Smart dice Progressive math teaching interactive </li></ul><ul><li>Smart spoon Sensors for teaching cooking </li></ul><ul><li>Smart sink Sensing use of sink for ease and safety </li></ul><ul><li>Smart refrigerator Energy reducing, food improving preservation </li></ul><ul><li>Smart utensils Sensing to teach and coach food preparation </li></ul><ul><li>Talking Trivet, e-sleeve, shoulder pet One sensor systems that react to intention </li></ul><ul><li>Thought for food Systems to use common sense and feedback to interact with food and recipes </li></ul><ul><li>USPS concept truck Annotation, location awareness support for commercial delivery, (8 demos) </li></ul><ul><li>Voyager Location aware system for interactive campus, grocery store, etc. </li></ul><ul><li>Wireless dashboard Harvested power for sensing and control </li></ul>New access New scenario
    15. 15. Intention Matters <ul><li>Recognize intention </li></ul><ul><ul><li>Sensors that know what they are looking for </li></ul></ul><ul><ul><li>Static and dynamic models </li></ul></ul><ul><ul><ul><li>of user, system and task </li></ul></ul></ul><ul><li>Respect intention </li></ul><ul><ul><li>Human Social goals come first </li></ul></ul><ul><ul><li>Human attention & flow </li></ul></ul><ul><ul><li>Human task is central </li></ul></ul><ul><ul><li>Solicitors beware </li></ul></ul><ul><li>Act on intention </li></ul><ul><ul><li>When it supports flow </li></ul></ul>… And Social Response Essential…
    16. 16. ALZ: Productively Challenged, Critically sophisticate <ul><li>Room </li></ul><ul><li>Places </li></ul><ul><li>Puzzle </li></ul><ul><li>Arranging flowers </li></ul><ul><li>Art </li></ul><ul><li>Radio </li></ul><ul><li>Catalog </li></ul>
    17. 17. Can Computers read our minds? <ul><li>Language based directive? </li></ul><ul><li>Intentions, Reaction, </li></ul><ul><li>Possibility… </li></ul>
    18. 18. What you do says so much … <ul><li>Where you are tells a story </li></ul><ul><ul><li>Floor imagery comments on what people are doing </li></ul></ul><ul><li>How you react to what you are doing </li></ul><ul><ul><li>Laughcam listens to camera person </li></ul></ul><ul><li>Watching your mouse </li></ul>
    19. 19. Where We Use Computers <ul><li>Wishes from Eye Gestures </li></ul><ul><ul><li>Closed: going to sleep </li></ul></ul><ul><ul><li>Open: no alarm </li></ul></ul><ul><ul><li>Blinks: doesn’t like something </li></ul></ul><ul><ul><li>Wink: selection </li></ul></ul><ul><ul><li>Stare: interest </li></ul></ul><ul><ul><li>Gaze: thinking </li></ul></ul>                                                      This bed, invented by an MIT scientist, has many high-tech features. (ABCNEWS.com) Sleep With Your Computer? MIT Researcher Computerizes Life                              April 10 — Imagine a bed that gently wakes you up at the crack of dawn and reminds you of your upcoming appointments.
    20. 20. Mind full of intentions Learning by watching <ul><li>I look around </li></ul><ul><ul><li>Interest Tracker, Invision </li></ul></ul><ul><li>Eye aRe Personal gaze </li></ul><ul><ul><li>Looking for a sign? </li></ul></ul>- Robot seeks work as fuel tank inspector
    21. 21. Attending visually (Yarbus 1967). 1) Free examination. Before subsequent recordings, the subject was asked to: 2) estimate the material circumstances of the family; 3) give the ages of the people; 4) surmise what the family had been doing before the arrival of the &quot;unexpected visitor;&quot; 5) remember the clothes worn by the people; 6) remember the position of the people and -objects in the room; 7) estimate how long the &quot;unexpected visitor&quot; had been away from the family Learning by watching
    22. 22. Invision Your gaze organizes things
    23. 23. LaughCam <ul><li>Reaction based interface: </li></ul><ul><ul><li>Your reactions edit your video </li></ul></ul>… Index video with laughter and skin conductivity A Lockerd, F.Mueller
    24. 24. AI Can help Human/Human Communication <ul><li>Empathy Buddy- Hugo Liu </li></ul><ul><li>Driftcatcher- Andrea Lockerd </li></ul>
    25. 25. Car COACH (Cognitive Adaptive Computer Help) <ul><li>When to comment on driving? </li></ul><ul><ul><li>When driver is open to comment </li></ul></ul><ul><ul><li>When driver is doing a good job </li></ul></ul><ul><ul><li>When driver is making mistakes: </li></ul></ul><ul><ul><ul><li>Driving smoothly </li></ul></ul></ul><ul><ul><ul><li>Signaling turns, lane changes </li></ul></ul></ul><ul><ul><ul><li>Gas, break, steering </li></ul></ul></ul><ul><ul><ul><li>Honking </li></ul></ul></ul><ul><ul><ul><li>Gas mileage </li></ul></ul></ul><ul><ul><ul><li>Visually checking instruments </li></ul></ul></ul><ul><ul><ul><li>Scanning scene </li></ul></ul></ul><ul><ul><ul><li>Checking blind spots </li></ul></ul></ul><ul><ul><ul><li>Finding controls easily </li></ul></ul></ul>
    26. 26. Face Interface Teaching face interaction from experience <ul><li>Facial gestures </li></ul><ul><ul><li>Caricaturized explanation </li></ul></ul><ul><ul><li>Create keystroke input </li></ul></ul><ul><ul><li>Vending/interaction scenarios </li></ul></ul>
    27. 27. IM Disruption Manager Ernesto Arroyo, Shawn Sullivan <ul><li>User Activity </li></ul><ul><ul><li>Implicit mouse behavior </li></ul></ul><ul><ul><li>User schemas about IM </li></ul></ul><ul><li>Task-oriented strategies </li></ul><ul><ul><ul><li>Task type </li></ul></ul></ul><ul><ul><ul><li>Task stage </li></ul></ul></ul><ul><li>Goal-oriented strategies </li></ul><ul><ul><ul><li>Quantity, Quality, Time, … </li></ul></ul></ul>
    28. 28. <ul><li>25% or more increased performance with disruption manager </li></ul><ul><li>People prefered it </li></ul><ul><li>Worked as well in daily work (10 day exeriment) </li></ul>
    29. 29. Autonomous Considerate things Augmenting existing things banana2.jpg <ul><li>Materials Chameleon mug </li></ul><ul><ul><li>Look like a mug when coffee </li></ul></ul><ul><ul><li>Look like a tumber when ice tea </li></ul></ul><ul><li>Electronic s Left-overs timer </li></ul><ul><ul><li>Count days in fridge </li></ul></ul><ul><li>Smarts … Talking Trivet… </li></ul><ul><ul><li>Needs re-warming </li></ul></ul><ul><ul><li>Hot and ready to eat </li></ul></ul><ul><ul><li>Cooked and ready to take out </li></ul></ul><ul><ul><li>Fire! </li></ul></ul>                  
    30. 30. Progressive relationships…dice Capacitive 3 d sensor teaches -Patterns, -numbers, -counting, -arithmetic
    31. 31. Smart sink <ul><li>Temperature colored water </li></ul><ul><ul><li>Warm for hands, </li></ul></ul><ul><ul><li>hot for filling </li></ul></ul><ul><ul><li>Cold for washing vegetables, </li></ul></ul><ul><li>Height appropriate </li></ul>
    32. 32. A Spoon that teaches cooking <ul><li>Better food </li></ul><ul><li>Teaching cooking </li></ul>
    33. 33. Digital cigarette Win Burleson <ul><li>Comforting or Calming device </li></ul><ul><li>Set pace and timing for activity </li></ul><ul><li>Break ice </li></ul><ul><li>Something for me </li></ul><ul><li>Something to give </li></ul><ul><li>Limit distractions </li></ul><ul><li>Entertainment </li></ul><ul><li>Give me </li></ul><ul><ul><li>Courage, stamina, patience </li></ul></ul><ul><li>Define me … </li></ul><ul><ul><li>outsider, insider, dangerous, grown up, courageous, available </li></ul></ul>http://web.media.mit.edu/~win/Digital%20Cigarette.htm
    34. 34. Bicycling August 2004 <ul><li>Loud outside -amplify noise </li></ul><ul><li>Tip head -blinkers </li></ul><ul><li>Talk -telephone </li></ul><ul><li>Yell -horn </li></ul><ul><li>Quiet -music </li></ul><ul><li>Shake head -”pothole” </li></ul><ul><ul><li> GPS and description </li></ul></ul>Mediating communication: a person and their environment
    35. 35. Postal Truck Business : GPS/ 802.11Delivering Everything Targeted marketing Enlist for traffic control Location & time sensitive services Safety Cameras Surround vehicle Sonic, vibration and LED indicators Lear anti Whiplash seat Automatic emergency break …
    36. 36. Physical virtual world interface <ul><li>Cart Companion or phone </li></ul><ul><ul><li>Teach about products </li></ul></ul><ul><ul><ul><li>Round roast: $1.99 </li></ul></ul></ul><ul><ul><ul><li>Lean meat, Gourmet when </li></ul></ul></ul><ul><ul><ul><li>slow cooked in wine and tarragon </li></ul></ul></ul><ul><ul><li>Buy products </li></ul></ul><ul><ul><ul><li>Meat counter </li></ul></ul></ul><ul><ul><ul><li>Infinite Aisle assists purchases </li></ul></ul></ul><ul><ul><ul><ul><li>Crock Pot for roasts… </li></ul></ul></ul></ul><ul><li>Watch kids </li></ul><ul><ul><li>Camera in arcade </li></ul></ul><ul><ul><li>Controls Xbox </li></ul></ul>
    37. 37. Chandelier Video Projector
    38. 38. Augment and Integrate with annotation
    39. 39. Forms and their implications Keyboard size and implications Hinge and implications <ul><li>Keyboard size and implications </li></ul><ul><li>Hinge implications </li></ul><ul><li>Durability, maintainability </li></ul><ul><li>look </li></ul>
    40. 40. HCI ? <ul><li>Office, data, teaching, collaboration, control… </li></ul><ul><li>Instruments, Home, Mobile, Auto, kitchen, industrial,… </li></ul><ul><li>Assistive Agents </li></ul><ul><li>Advisory Agents </li></ul><ul><li>Persuasive interfaces </li></ul><ul><li>Energy monitoring, control, </li></ul><ul><li>Disaster: photos, communication, coordination, , planning, </li></ul><ul><li>Activist tools: JIT organizing, reflective art, </li></ul>
    41. 41. Product Design? <ul><li>Art: creating ideas to create an emotional reaction </li></ul><ul><li>Science: creating ideas that other people cite ( cause other people to create ideas ) </li></ul><ul><li>Design: is creating things of value </li></ul>
    42. 42. Invention is a performance Personal credo <ul><li>Inventing is like an action movie </li></ul><ul><ul><li>Grab for a branch, it brakes, land in the water and swim… </li></ul></ul><ul><ul><li>The way it doesn’t moves you forward </li></ul></ul><ul><li>Audition Stories and roles </li></ul><ul><ul><li>Think of many possible ways out </li></ul></ul><ul><ul><ul><li>keep thinking about it </li></ul></ul></ul><ul><li>All the world is a stage. </li></ul><ul><ul><li>Slowly replace pantomime for real sets and actors </li></ul></ul><ul><li>Have things/ideas compete not people </li></ul><ul><li>Pretend things are the way you want them </li></ul><ul><li>Create it from anything </li></ul><ul><ul><li>All tools are made of other tools </li></ul></ul><ul><ul><li>Prototype here and now </li></ul></ul>
    43. 43. Technology will help Expectations <ul><li>Coordination ? </li></ul><ul><li>Planning ? </li></ul><ul><li>more control over Shared -VS- personal </li></ul><ul><li>more control over Voyeur -VS- participant </li></ul>
    44. 44. Future Feel of Tools <ul><li>We have been using computers for cataloging </li></ul><ul><li>Context Based computing </li></ul><ul><li>Information is becoming Personal and Personalized </li></ul><ul><li>Technology is inevitably used for Socialization </li></ul><ul><li>Computers will be aware </li></ul><ul><ul><li>reduce transcription </li></ul></ul><ul><li>Computers will understand… </li></ul><ul><ul><li>Teach, augment, proxy </li></ul></ul>