Multi-Touch Interaction Overview

40,202
-1

Published on

This presentation provides an overview of multi-touch hardware, products, applications and market examples as well as samples of projects of TNO. More information on http://www.tno.nl/nui

Published in: Technology
43 Comments
241 Likes
Statistics
Notes
No Downloads
Views
Total Views
40,202
On Slideshare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
0
Comments
43
Likes
241
Embeds 0
No embeds

No notes for slide

Multi-Touch Interaction Overview

  1. 1. Natural User Interfaces Natural User Hardware Interfaces Markets and Markets and Hardware and software applications Software applications History and future Timeline Multi-touch interaction TNO work TNO Work Arnoud de Jong Arnout de Vries TNO Information and Communication Technologies Multi- Multi- Summary touch touch in and products the Conclusions 1 movies
  2. 2. History of Multi-touch interaction Pierre Wellner paper Foundation on multi touch and iGesture Pad and NUI Group First multi pinching motions TouchStream keyboard hand display by Fingerworks Bell Labs Palm Sony labs Microsoft Pre First MT SmartSkin surface PlayAnywhere display Toronto Mixed Reality table of Touch and object university Microsoft Research detection Hitachi Q2 2009 Smartboard 2002 1982 2005 1991 1999 2006 2001 1984 FX77 Duo 1980 2009 Jun 2007 2002 Q3 2008 Q1 2009 2003 1992 1982 1985 2006 Q1 2006 University of 2001 Movie: 3D multi-touch Toronto Tron Microsoft sensitive tablet Simon - first Multi-finger Windows 7 smartphone Multi-hand Multi-user Lucid With Mitsubishi Touch interaction iPhone touchscreen MERL R&D Lab release DiamondTouch HP-150 world’s Microsoft Surface 1983 first Jeff Han SMART touchscreen PC Movie: @TED table Minority report 2
  3. 3. Hardware technology • Multi-point detection • Multi-object detection • Setup 3
  4. 4. 4
  5. 5. 5
  6. 6. 6
  7. 7. 7
  8. 8. 8
  9. 9. 9
  10. 10. 10
  11. 11. 11
  12. 12. Setups 12
  13. 13. Products • Table tops • Tablet PC’s • Whiteboards • Large surfaces (walls, floors, tables) • Desktop PC’s • Netbooks / UMPC’s • Mobile phones • TV’s 13
  14. 14. Fraunhofer IGD multi-touch table • The table is running on Linux and uses the InstantReality framework for displaying and interacting. • The finger tracking is done by the Fraunhofer software VisionLib which is not available for pulic right now. • Table size is 150x90 cm 14
  15. 15. Mitsubishi: DiamondTouch • Front-projection on typically 32" or 42" diagonal • Multiple touches support by a single user • Distinguishes between simultaneous inputs from multiple users. • DiamondTouch SDK or DTFlash toolkit. 15
  16. 16. Gesturetek 16
  17. 17. Horizon
  18. 18. SMART table
  19. 19. Microsoft: Surface • www.surface.com • Object on table are identified by ‘domino tags’ • http://worldofitgeeks.blogspot.com/2007/10/domino-theories-for-microsofts-surface.html 19
  20. 20. HP TouchSmart 20
  21. 21. HP Touchsmart TX2 tablet PC
  22. 22. Dell Latitude XT 22
  23. 23. Hitachi Starboard FX77 Duo 23
  24. 24. Multi touch desktop LCD’s 24
  25. 25. TV 25
  26. 26. iPhone en iPod Touch • The Apple iPhone has a multitouch display for resizing, zooming and rotating purposes. • Multi touch sensative screen • Motion sensor • Types of multi touch gestures: • Zoom, rotate 26
  27. 27. G1 Google Android phone 27
  28. 28. Palm Pre • WebOS. It’s a web-centric multi- touch platform that makes us of CSS, XHTML and Javacript making it accessible and compatible with a wide range of applications. 28
  29. 29. And more mobile phone devices Meizu M8 HTC Touch Synaptics Onyx Nokia N97 29
  30. 30. Apple: MacBook Air • The Apple MacBook Air has multitouch touchpad for resizing, zooming and rotating purposes. 30
  31. 31. Eee PC 31
  32. 32. MultiTouch Cell • The MultiTouch LCDs are controlled by the MultiTouch CornerStone software. Cornerstone handles the core multitouch functionality, tracking users hands as they move on the screen. 32
  33. 33. Other vendors Accenture Panasonic Hitachi Philips Entertaible Perceptive Pixel Sony Smartskin HP Misto Sharp optical LCD Touchable
  34. 34. Market share of products with multi-touch • Multi-touch ability (or NUI) is competitor’s edge that is used in marketing • Market share is quickly rising • 10 mio iPhones sold in first 6 months • Table top sales booming since end 2008 Multi-touch is currently an “infant” technology • Different multi-touch imlementations (Apple, Microsoft, HP, etc.) • All are limited to a handful of gestures • Still a lot of traditional GUI interaction 34
  35. 35. Future product list • Input devices • Apple Mighty Mouse • Wacom tablets • Handheld game consoles • PSP2 • E-book readers • Desktops • Larger and flat tabletops • TV screens 35
  36. 36. Natural User Interfaces (NUI’s) Introduction • Touch screen interaction • Multi-touch screen interaction The vision • Evolution of computer user interfaces Reality • Natural User Interfaces (NUI) • Tangible interfaces • Multi modal interfaces • NUI design 36
  37. 37. Touch screen interaction Touch interaction • 1960: Touch • 1982: Multi touch • 2006: Lucid touch • Multi modal interfaces, controlled by • Touch • Gestures • Speech • Objects 39
  38. 38. Single Touch 40
  39. 39. Touch screen interfaces (since 1960’s) • Interaction via • Stylus, light pen, finger, hand,  Finger stress  Fingernail interaction  Fingerprints  “Gorilla arm” • No intelligent interaction • Gestures • Touch force sensing • Object sensing (finger vs hand) 41
  40. 40. Multi-touch & Natural User Interfaces 42
  41. 41. Multi-touch interaction • In the realworld we do lots of actions with two hands or more than one finger. e.g. driving a car • Operating a mouse are serial, discrete events and single point input • Multi point interaction allows more parallel interaction • Reduce task complexity of single input techniques • Increases parallesm in multi-finger interaction reduces time 43
  42. 42. Tangible interfaces 44
  43. 43. Interact with real objects 45
  44. 44. Haptics 46
  45. 45. Haptics 47
  46. 46. Haptics today 48
  47. 47. Lucid Touch 49
  48. 48. But lucid touch interaction is very complex 50
  49. 49. Gesture interaction 51
  50. 50. Gesture interaction today 52
  51. 51. Multimodal interaction 53
  52. 52. Multimodal interaction is still the future… 54
  53. 53. Paradigm shift Command Line Interfaces (CLI) 55
  54. 54. Paradigm shift Graphical User Interfaces (GUI) 56
  55. 55. Paradigm shift Natural User Interfaces (NUI) 57
  56. 56. Paradigm shift • Static • Responsive • Evocative • Disconnected • Indirect • Unmediated • High-low • Double medium • Fast few • Directed • Exploratory • Contextual • Recall • Recognition • Intuition • Abstract • Indirect • Direct • Textual • Graphical • Physical 58 Source: Dennis Wixon
  57. 57. Overview of Human Computer Interfaces Microsoft Microsoft Windows NT Windows Vista Alto PC Microsoft Microsoft Xerox Parc Windows 1.0 Windows 95 Microsoft MS-DOS Windows 98 Microsoft Microsoft Microsoft Q2 2009 Windows 3.0 Windows XP Windows 7 1993 1995 2001 1982 1985 1998 2007 1990 1973 1970 2010 Jun 2007 1987 1997 Microsoft 2001 1984 1992 2005 2009 Windows Snow Tablet PC Leopard? Apple Macintosh edition IBM OS/2 2.0 Apple Mac OS 8 Apple Mac OS X 10.5 Apple Mac OS X 10.0 Apple Mac Color Note: Linux excluded
  58. 58. Interaction styles and elements 1. Punch-card input (50’s-60’s) 2. Teletype writer connected to mainframe (60’s -80’s) EVOLUTION 3. CLI (80’s) 4. GUI (90’s-00’s) • Classic WIMP (Window, Icon, Menu, Pointer) • Desktop metaphor • Post-WIMP interfaces • Computer games • Virtual reality systems • Physical controls (e.g. iPod wheel) • Multi-touch interaction 3. NUI • STaG (Speech, Touch and Gesture) 4. XUI • Synthesis, anticipatory, constant fluid interaction 60
  59. 59. The vision of Natural User Experience (NUE) is to deliver intuitive, seamless experiences that unfold through natural human input.” • “Surfaces that respond to us” • “The content is the interface” 61
  60. 60. Surface: design for emotion • Natural & Intuitive • Unique & Magical • Social & Together • Aware & Responsive • Premium & Authentic 62
  61. 61. Surface guidelines 1. Seamless 2. Social 3. Spatial 4. Super realism 5. Contextual environments 6. Scaffolding 7. Performance aesthetics 8. Direct manipulation
  62. 62. User Interface Design for NUI’s Difference in designing Tabletop interfaces? • No fixed top-bottom orientation; • Multiple “mouse cursors”; • Multiple simultaneous users; • Multiple user contexts; • Multiple simultaneous actions, such as ‘drag & drop’; • No more tradional data input means. 64
  63. 63. Multi-touch in the movies • 1982: Tron • 2002: Minority report • 2005: The Island • 2008: Quantum of Solace (James Bond) • 2008: CSI: Miami 65
  64. 64. 1982: Tron 66
  65. 65. 2002: Minority Report 67
  66. 66. 2005: The Island 68
  67. 67. 2008: Quantum of Solace 69
  68. 68. 2008: CSI: Miami 70
  69. 69. Markets and applications • Entertainment: (serious) gaming and fun • Public safety and crisis management • Retail • Public Places • Healthcare • Education • Financial • Office • Industrial use 71
  70. 70. Healthcare • Photo tool for dementia patients • Tool for patient-doctor relationships 72
  71. 71. Entertainment • Games (serious/fun) 73
  72. 72. Public places • Information centre New York • Why? • Intuitive, interactive and focussed on discovering New York • Target group: Tourists • Device • Gesturetek illuminate tables • Application: • Maps, POI, multimedia content 74 Information centre New York
  73. 73. Museums 75
  74. 74. TV presentations CNN – presidential elections
  75. 75. Retail 78
  76. 76. Horeca • Restaurants, hotels • Viewing menu • Entertainment (games, gambling) • Tourist information Surfaceware: detect drink level 79
  77. 77. iBar: object recognizing bar • Bar detects all objects on the surface • It detects the shape, it doesn’t identify the object. • future releases will also be able to identify objects • http://www.i-bar.ch/info/ 80
  78. 78. Education 81
  79. 79. Control Rooms AT&T Telecomunication control room OODA loop 1. Static risk map 2. real-time or simulated flooding 3. Real-time information feeds 82
  80. 80. Public safety • Manages planning, tracking and report security matters surrounding an event. The Super Bowl E•SPONDER solution 83
  81. 81. Financial services Use in a bank lobby for customers to look at account features and sit down with a banker to discuss their financial needs. 84
  82. 82. i-phone multi-touch applications • Photo’s: create a collage • Graffiti • Scratch Mixmeister • Games Foto collages maken Muziek scratchen en mixen puistjes uitdrukken 85
  83. 83. Industrial use • Insight in processes • Observe, control and decide • Collaborative design • Fraunhofer: • Virtual factory on a tabletop
  84. 84. Collaborative work in the office 87
  85. 85. Office Digidesk
  86. 86. TNO Work • Multi -touch in Control Rooms • Multi –touch in Education • Multi-touch : complex gesture support • MiReCol: collaboration and co-design • Slide overview + video Maurice 89
  87. 87. Control Rooms
  88. 88. Experiment area 91
  89. 89. Visualisation of information (1.4)(2.4)(21.6)(4.2)(7.6)(2.6)(9.8)(3.0) Quick insight Large data amounts Unstructured Time critical Diverse sources
  90. 90. Data sources information exchange Virtual teams Control room teams Control room decision centre 93
  91. 91. Control Rooms AT&T Telecomunication control room OODA loop 1. Static risk map 2. real-time or simulated flooding 3. Real-time information feeds 94
  92. 92. Crisis management scenario Satellite or street map Multi-user interface Ten Boer Risk objects Real-time communication log Annotate Analyse Anticipate Change view, filter, log
  93. 93. MiReCol: Mixed Reality Collaboration
  94. 94. Mixed reality collaboration Augmented collaboration Shared Augmented virtuality understanding collaboration Measuring and facilitating collaboration
  95. 95. Collaboration • Co-creation, co-design • Decision making • CSCW support in general 100
  96. 96. Teliris telepresence
  97. 97. Education 102
  98. 98. Serious gaming & multi touch interactie in het onderwijs Research goal: Does the combination of serious gaming and multi touch interaction add value to elementary education. Serious games Target group: School children aged 8 until 12 with a social disorder (PDD-NOS). Multi touch Results: a serious game was developed to support and stimulate several collaboration forms in calculation tasks. Through multi- touch and multi user interaction these children could cooperate using one Education system. Partners: Bladergroenschool (elementary school for special education) and RENN4.
  99. 99. Supporting gesture interaction
  100. 100. Supporting gesture interaction with gesture previews Gestures • In general: non-verbal communication. • Specific: a set of measured movements over time. • Examples: letter recognition, unlocking your iPhone. Advanced gestures • Multi-touch gestures: zooming an image. • Collaborative gestures: clearing a drawing board. Problem statement • Most gestures have no intrinsic meaning. • Users have to know gestures in advance. • A steep learning curve especially for complex systems using advanced gestures.
  101. 101. 106
  102. 102. Solution: gesture previews • Overlay with visual feedback. • Shows what gestures can be performed. • Shows where others can join in. • Indicates the meaning of gestures. • Previews do not limit expert users. • Increase detection rate, decrease learning curve.
  103. 103. Proof of Concept • Bejeweled clone, uses gestures to identify users. With some twists. • Gestures are stored in XML. <?xml version="1.0" encoding="utf-8"?> <root> <segmentDeclaration name="line" uniformScaling="false" <point x="0" y="0" /> <point x="0" y="0,1" /> </segmentDeclaration>
  104. 104. Wrap up Knowledge • Started at TNO just a few years ago, experimental. Now this topic is booming, backed up by large vendors and coming to the market. Hardware/software • More vendors enabling MT for their devices • Current software / applications are dedicated solutions • Wide adoption needs more hardware and applications Interaction • Multi touch is part of the NUI paradigm shift in Human Machine Interaction Markets / applications • Wow-technology, but the real added value and useful applications still have to prove themselves TNO • Added value in collaboration environments (crisismanagement, education and many more projects) 109
  105. 105. Arnout de Vries Senior scientist Natural User Interfaces Arnout.devries@tno.nl http://www.tno.nl/nui

×