0
Untold Stories of Touch, Gesture, & NUI
        Joe Fletcher
        Design Manager, Microsoft Surface




3/31/2009   Joe...
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              2
Touch is great
   Touch is not                                   Touch is a
                                              ...
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              4
Microsoft Surface




     Microsoft Surface is a revolutionary
     computer that responds to natural hand
     gestures ...
Microsoft Surface




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              6
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              7
Vision                        Strategy   Design

3/31/2009   Joe Fletcher   MIX09, Las Vegas
                             ...
Surface     Massive Multi-touch, Multi-user

     Unique
      Spot
                                              Windows ...
Larger Surface Team




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              10
Touch is great for
 Touch is not good                            Touch is an
                                             ...
Touch                                       Gesture                               Manipulation
            The act of dire...
History and                            Surface Design     Touch
        Ecology                                 Principles...
Surface Design
       History and Ecology                                     Touch Challenges
                           ...
Why Evolve?
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                                            15
Command Line Interface (CLI)




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                             ...
Graphic User Interface (GUI)




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                             ...
Natural User Interface (NUI)




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                             ...
Natural User Interface (NUI)




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                             ...
Natural User Interface (NUI)




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                             ...
Microsoft Surface and NUI




     Microsoft Surface computing uses sensing and display technology to infuse everyday
    ...
NUI
     vs.

     Web 2.0
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              22
NUI & Touch
     should not be a
     novelty
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                ...
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              24
Input Ecology




     Keyboard                          Mouse                Stylus                Single point          ...
[Partial] Touch Ecology




                                                    Multi Touch              NUI
             ...
Tap is not the new click




                                              ≠


3/31/2009   Joe Fletcher   MIX09, Las Vegas...
Context, task, and use define the solution




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                               ...
Context, task, and use define the solution




     Touch is not a
     solution in search of
     a problem
3/31/2009   J...
Touch is great for
 Touch is not good                            Touch is an
                                             ...
Surface Design
       History and
                                                               Touch Challenges
        ...
Create an
     inflection point
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              ...
The start of almost every
     successful technological
     innovation is marked by an
     inflection point, which defin...
Companies spend
     90% of their energy
     trying to imitate
3/31/2009   Joe Fletcher   MIX09, Las Vegas
              ...
Microsoft Surface and principle driven design




                                              Principle Driven Design


...
Microsoft Surface and principle driven design




                                              Principle Driven Design
  ...
CLI                GUI        NUI


                       Text                   Graphics   Objects

3/31/2009   Joe Flet...
Recall                 Recognition   Intuition




                             CLI                GUI            NUI


  ...
Directed               Exploratory   Contextual


                       Recall                 Recognition   Intuition


...
High-Low               Double Medium   Fast Few


                       Directed               Exploratory     Contextual...
Disconnected           Indirect        Unmediated


                       High-Low               Double Medium   Fast Few...
Static                 Responsive      Evocative


                       Disconnected           Indirect        Unmediate...
Principle of Performance Aesthetics    Evocative


                                                                       ...
Microsoft Surface and principle driven design




                                              Principle Driven Design


...
Microsoft Surface and principle driven design




                                              Three Super Principles


3...
Social   Multiple simultaneous users




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                     ...
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              47
Social     Multiple simultaneous users



                                     Seamless   Digital and Physical combined


...
Seamless




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              49
Social     Multiple simultaneous users



                                     Seamless   Digital and Physical combined


...
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              51
Overcome
     Homeostasis

3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              52
Touch is great for
 Touch is not good                            Touch is an
                                             ...
History and                            Surface Design
                                                               Touch...
Let’s become
     practitioners
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              ...
Unique
                                       Spot
                                               PC’s, Form Factors, Tech...
Challenge   Form Factor & Technology




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                     ...
Touch Hardware Ecology




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              58
Touch Hardware Ecology




                                                                                        Stylus
...
Touch Hardware Ecology




                                                                                           Styl...
Challenge   Patterns




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                                     ...
FireFox
        Swipe Left: Go back in history
        Swipe Up: Go to the top of the page




3/31/2009   Joe Fletcher   ...
Apple iPhone
        Swipe Left: Go Forward in History
        Swipe up: Go to the bottom of a page




3/31/2009   Joe Fl...
Challenge   Orientation




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                                  ...
Horizontal   Vertical

3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                                       ...
Challenge   Cardio Effect




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                                ...
Challenge   Paradigms




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                                    ...
To button
     or

     not to button
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                        ...
Challenge   Getting people to touch




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                      ...
Exploration   Instruct




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                                   ...
Challenge   Visualizing Touch




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                            ...
Is it working?



3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              72
Future      Where do we go?




3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              ...
Your effort to remain
     what you are is what
     limits you
     Ghost in the Shell



3/31/2009   Joe Fletcher   MIX0...
Debating the future
     of touch is useless

3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                ...
Creating the future
     is more interesting

3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                ...
Touch is great for
 Touch is not good                            Touch is an
                                             ...
Take advantage of Microsoft Surface




             Download SDK                     • Available from the Surface Communi...
Future Vision




                                              2019
3/31/2009   Joe Fletcher   MIX09, Las Vegas
         ...
3/31/2009   Joe Fletcher   MIX09, Las Vegas
                                              80
Las Vegas, Thank you!
        Joe Fletcher
        joeflet@microsoft.com
        Design Manager, Microsoft Surface




3/3...
Your feedback is important!
Upcoming SlideShare
Loading in...5
×

Touch and Gesture Computing, What You Haven't Heard

5,335

Published on

Learn about new patterns, behaviors, and design approaches for touch and gesture interfaces from a practitioners point of view. Learn early lessons from applied knowledge of touch applications, devices, and design methods.

Published in: Business, Technology
0 Comments
31 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,335
On Slideshare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
725
Comments
0
Likes
31
Embeds 0
No embeds

No notes for slide

Transcript of "Touch and Gesture Computing, What You Haven't Heard"

  1. 1. Untold Stories of Touch, Gesture, & NUI Joe Fletcher Design Manager, Microsoft Surface 3/31/2009 Joe Fletcher MIX09, Las Vegas 1
  2. 2. 3/31/2009 Joe Fletcher MIX09, Las Vegas 2
  3. 3. Touch is great Touch is not Touch is a for specific good for computing aspects everything evolution New technology has enabled touch It is not an end all solution It is an addition element for users to systems, which can enhance certain interact with their content in new and activities unique ways 3/31/2009 Joe Fletcher MIX09, Las Vegas 3
  4. 4. 3/31/2009 Joe Fletcher MIX09, Las Vegas 4
  5. 5. Microsoft Surface Microsoft Surface is a revolutionary computer that responds to natural hand gestures and real-world objects, creating unique new ways for people to collaboratively interact with digital content. 3/31/2009 Joe Fletcher MIX09, Las Vegas 5
  6. 6. Microsoft Surface 3/31/2009 Joe Fletcher MIX09, Las Vegas 6
  7. 7. 3/31/2009 Joe Fletcher MIX09, Las Vegas 7
  8. 8. Vision Strategy Design 3/31/2009 Joe Fletcher MIX09, Las Vegas 8
  9. 9. Surface Massive Multi-touch, Multi-user Unique Spot Windows 7 PC’s, Form Factors, Technology 3/31/2009 Joe Fletcher MIX09, Las Vegas 9
  10. 10. Larger Surface Team 3/31/2009 Joe Fletcher MIX09, Las Vegas 10
  11. 11. Touch is great for Touch is not good Touch is an specific aspects for everything evolution 3/31/2009 Joe Fletcher MIX09, Las Vegas 11
  12. 12. Touch Gesture Manipulation The act of directly touching your digital Hand motions to take an action Alteration or freeform movement of content digital content using touch Natural User Interface The combination of these and other elements the create a unique way to interact with your digital content 3/31/2009 Joe Fletcher MIX09, Las Vegas 12
  13. 13. History and Surface Design Touch Ecology Principles Challenges 3/31/2009 Joe Fletcher MIX09, Las Vegas 13
  14. 14. Surface Design History and Ecology Touch Challenges Principles 3/31/2009 Joe Fletcher MIX09, Las Vegas 14
  15. 15. Why Evolve? 3/31/2009 Joe Fletcher MIX09, Las Vegas 15
  16. 16. Command Line Interface (CLI) 3/31/2009 Joe Fletcher MIX09, Las Vegas 16
  17. 17. Graphic User Interface (GUI) 3/31/2009 Joe Fletcher MIX09, Las Vegas 17
  18. 18. Natural User Interface (NUI) 3/31/2009 Joe Fletcher MIX09, Las Vegas 18
  19. 19. Natural User Interface (NUI) 3/31/2009 Joe Fletcher MIX09, Las Vegas 19
  20. 20. Natural User Interface (NUI) 3/31/2009 Joe Fletcher MIX09, Las Vegas 20
  21. 21. Microsoft Surface and NUI Microsoft Surface computing uses sensing and display technology to infuse everyday surfaces with digital content and is comprised of the following unique attributes: Direct interaction Multi-touch Multi-user Object recognition 3/31/2009 Joe Fletcher MIX09, Las Vegas 21
  22. 22. NUI vs. Web 2.0 3/31/2009 Joe Fletcher MIX09, Las Vegas 22
  23. 23. NUI & Touch should not be a novelty 3/31/2009 Joe Fletcher MIX09, Las Vegas 23
  24. 24. 3/31/2009 Joe Fletcher MIX09, Las Vegas 24
  25. 25. Input Ecology Keyboard Mouse Stylus Single point Multi point Good for Good for Good for Good for Good for Accuracy Moving across the Freeform input Direct object selection Manipulating objects screen efficiently Social Con Con Con Con Con Indirect Not for input Limited Low accuracy Not for quick input Low accuracy Need large controls Low accuracy Need large controls Need of large controls Support of hands and Mis-hits from screen fingernails of perceived hit area 3/31/2009 Joe Fletcher MIX09, Las Vegas 25
  26. 26. [Partial] Touch Ecology Multi Touch NUI Multi point (screen) (Gesture & Physical/Digital) (screen) Reacts to multiple Reacts to multiple Reacts to direct input inputs inputs Orients UI to the Spatial Recognition - direction of the touch Orients UI to the direction of the touch Spatial recognition of system Object recognition - Blurring physical and digital 3/31/2009 Joe Fletcher MIX09, Las Vegas 26
  27. 27. Tap is not the new click ≠ 3/31/2009 Joe Fletcher MIX09, Las Vegas 27
  28. 28. Context, task, and use define the solution 3/31/2009 Joe Fletcher MIX09, Las Vegas 28
  29. 29. Context, task, and use define the solution Touch is not a solution in search of a problem 3/31/2009 Joe Fletcher MIX09, Las Vegas 29
  30. 30. Touch is great for Touch is not good Touch is an specific aspects for everything evolution 3/31/2009 Joe Fletcher MIX09, Las Vegas 30
  31. 31. Surface Design History and Touch Challenges Ecology Principles 3/31/2009 Joe Fletcher MIX09, Las Vegas 31
  32. 32. Create an inflection point 3/31/2009 Joe Fletcher MIX09, Las Vegas 32
  33. 33. The start of almost every successful technological innovation is marked by an inflection point, which defines a moment of dramatic change. 3/31/2009 Joe Fletcher MIX09, Las Vegas 33
  34. 34. Companies spend 90% of their energy trying to imitate 3/31/2009 Joe Fletcher MIX09, Las Vegas 34
  35. 35. Microsoft Surface and principle driven design Principle Driven Design 3/31/2009 Joe Fletcher MIX09, Las Vegas 35
  36. 36. Microsoft Surface and principle driven design Principle Driven Design Using history to develop the future 3/31/2009 Joe Fletcher MIX09, Las Vegas 36
  37. 37. CLI GUI NUI Text Graphics Objects 3/31/2009 Joe Fletcher MIX09, Las Vegas 37
  38. 38. Recall Recognition Intuition CLI GUI NUI Text Graphics Objects 3/31/2009 Joe Fletcher MIX09, Las Vegas 38
  39. 39. Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects 3/31/2009 Joe Fletcher MIX09, Las Vegas 39
  40. 40. High-Low Double Medium Fast Few Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects 3/31/2009 Joe Fletcher MIX09, Las Vegas 40
  41. 41. Disconnected Indirect Unmediated High-Low Double Medium Fast Few Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects 3/31/2009 Joe Fletcher MIX09, Las Vegas 41
  42. 42. Static Responsive Evocative Disconnected Indirect Unmediated High-Low Double Medium Fast Few Directed Exploratory Contextual Recall Recognition Intuition CLI GUI NUI Text Graphics Objects 3/31/2009 Joe Fletcher MIX09, Las Vegas 42
  43. 43. Principle of Performance Aesthetics Evocative Unmediated Principle of Direct Manipulation Principle of Scaffolding Fast Few Principle of Contextual Environments Contextual Principle of the Super Real Intuition NUI 3/31/2009 Joe Fletcher MIX09, Las Vegas 43
  44. 44. Microsoft Surface and principle driven design Principle Driven Design 3/31/2009 Joe Fletcher MIX09, Las Vegas 44
  45. 45. Microsoft Surface and principle driven design Three Super Principles 3/31/2009 Joe Fletcher MIX09, Las Vegas 45
  46. 46. Social Multiple simultaneous users 3/31/2009 Joe Fletcher MIX09, Las Vegas 46
  47. 47. 3/31/2009 Joe Fletcher MIX09, Las Vegas 47
  48. 48. Social Multiple simultaneous users Seamless Digital and Physical combined 3/31/2009 Joe Fletcher MIX09, Las Vegas 48
  49. 49. Seamless 3/31/2009 Joe Fletcher MIX09, Las Vegas 49
  50. 50. Social Multiple simultaneous users Seamless Digital and Physical combined Spatial Kinesiology 3/31/2009 Joe Fletcher MIX09, Las Vegas 50
  51. 51. 3/31/2009 Joe Fletcher MIX09, Las Vegas 51
  52. 52. Overcome Homeostasis 3/31/2009 Joe Fletcher MIX09, Las Vegas 52
  53. 53. Touch is great for Touch is not good Touch is an specific aspects for everything evolution 3/31/2009 Joe Fletcher MIX09, Las Vegas 53
  54. 54. History and Surface Design Touch Challenges Ecology Principles 3/31/2009 Joe Fletcher MIX09, Las Vegas 54
  55. 55. Let’s become practitioners 3/31/2009 Joe Fletcher MIX09, Las Vegas 55
  56. 56. Unique Spot PC’s, Form Factors, Technology 3/31/2009 Joe Fletcher MIX09, Las Vegas 56
  57. 57. Challenge Form Factor & Technology 3/31/2009 Joe Fletcher MIX09, Las Vegas 57
  58. 58. Touch Hardware Ecology 3/31/2009 Joe Fletcher MIX09, Las Vegas 58
  59. 59. Touch Hardware Ecology Stylus Capacitive Capacitive Infrared Vision system 3/31/2009 Joe Fletcher MIX09, Las Vegas 59
  60. 60. Touch Hardware Ecology Stylus Capacitive Capacitive Infrared Vision system Wacom (1) N-Trig (2-10) N-trig (2-10) HP (2) Surface (52) 3/31/2009 Joe Fletcher MIX09, Las Vegas 60
  61. 61. Challenge Patterns 3/31/2009 Joe Fletcher MIX09, Las Vegas 61
  62. 62. FireFox Swipe Left: Go back in history Swipe Up: Go to the top of the page 3/31/2009 Joe Fletcher MIX09, Las Vegas 62
  63. 63. Apple iPhone Swipe Left: Go Forward in History Swipe up: Go to the bottom of a page 3/31/2009 Joe Fletcher MIX09, Las Vegas 63
  64. 64. Challenge Orientation 3/31/2009 Joe Fletcher MIX09, Las Vegas 64
  65. 65. Horizontal Vertical 3/31/2009 Joe Fletcher MIX09, Las Vegas 65
  66. 66. Challenge Cardio Effect 3/31/2009 Joe Fletcher MIX09, Las Vegas 66
  67. 67. Challenge Paradigms 3/31/2009 Joe Fletcher MIX09, Las Vegas 67
  68. 68. To button or not to button 3/31/2009 Joe Fletcher MIX09, Las Vegas 68
  69. 69. Challenge Getting people to touch 3/31/2009 Joe Fletcher MIX09, Las Vegas 69
  70. 70. Exploration Instruct 3/31/2009 Joe Fletcher MIX09, Las Vegas 70
  71. 71. Challenge Visualizing Touch 3/31/2009 Joe Fletcher MIX09, Las Vegas 71
  72. 72. Is it working? 3/31/2009 Joe Fletcher MIX09, Las Vegas 72
  73. 73. Future Where do we go? 3/31/2009 Joe Fletcher MIX09, Las Vegas 73
  74. 74. Your effort to remain what you are is what limits you Ghost in the Shell 3/31/2009 Joe Fletcher MIX09, Las Vegas 74
  75. 75. Debating the future of touch is useless 3/31/2009 Joe Fletcher MIX09, Las Vegas 75
  76. 76. Creating the future is more interesting 3/31/2009 Joe Fletcher MIX09, Las Vegas 76
  77. 77. Touch is great for Touch is not good Touch is an specific aspects for everything evolution 3/31/2009 Joe Fletcher MIX09, Las Vegas 77
  78. 78. Take advantage of Microsoft Surface Download SDK • Available from the Surface Community site Workstation Edition • Learn how to build applications from scratch on Surface Develop for Surface by viewing videos on Surface.com • Interested in purchasing a Surface? Email to Purchase a Surface sbizdesk@microsoft.com 3/31/2009 Joe Fletcher MIX09, Las Vegas 78
  79. 79. Future Vision 2019 3/31/2009 Joe Fletcher MIX09, Las Vegas 79
  80. 80. 3/31/2009 Joe Fletcher MIX09, Las Vegas 80
  81. 81. Las Vegas, Thank you! Joe Fletcher joeflet@microsoft.com Design Manager, Microsoft Surface 3/31/2009 Joe Fletcher MIX09, Las Vegas 81
  82. 82. Your feedback is important!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×