Multi-Touch Interaction Overview

Natural User Interfaces Natural User Hardware Interfaces Markets and Markets and Hardware and software applications Software applications History and future Timeline Multi-touch interaction TNO work TNO Work Arnoud de Jong Arnout de Vries TNO Information and Communication Technologies Multi- Multi- Summary touch touch in and products the Conclusions 1 movies

History of Multi-touch interaction Pierre Wellner paper Foundation on multi touch and iGesture Pad and NUI Group First multi pinching motions TouchStream keyboard hand display by Fingerworks Bell Labs Palm Sony labs Microsoft Pre First MT SmartSkin surface PlayAnywhere display Toronto Mixed Reality table of Touch and object university Microsoft Research detection Hitachi Q2 2009 Smartboard 2002 1982 2005 1991 1999 2006 2001 1984 FX77 Duo 1980 2009 Jun 2007 2002 Q3 2008 Q1 2009 2003 1992 1982 1985 2006 Q1 2006 University of 2001 Movie: 3D multi-touch Toronto Tron Microsoft sensitive tablet Simon - first Multi-finger Windows 7 smartphone Multi-hand Multi-user Lucid With Mitsubishi Touch interaction iPhone touchscreen MERL R&D Lab release DiamondTouch HP-150 world’s Microsoft Surface 1983 first Jeff Han SMART touchscreen PC Movie: @TED table Minority report 2

Hardware technology • Multi-point detection • Multi-object detection • Setup 3

Setups 12

Products • Table tops • Tablet PC’s • Whiteboards • Large surfaces (walls, floors, tables) • Desktop PC’s • Netbooks / UMPC’s • Mobile phones • TV’s 13

Fraunhofer IGD multi-touch table • The table is running on Linux and uses the InstantReality framework for displaying and interacting. • The finger tracking is done by the Fraunhofer software VisionLib which is not available for pulic right now. • Table size is 150x90 cm 14

Mitsubishi: DiamondTouch • Front-projection on typically 32" or 42" diagonal • Multiple touches support by a single user • Distinguishes between simultaneous inputs from multiple users. • DiamondTouch SDK or DTFlash toolkit. 15

Gesturetek 16

Horizon

SMART table

Microsoft: Surface • www.surface.com • Object on table are identified by ‘domino tags’ • http://worldofitgeeks.blogspot.com/2007/10/domino-theories-for-microsofts-surface.html 19

HP TouchSmart 20

HP Touchsmart TX2 tablet PC

Dell Latitude XT 22

Hitachi Starboard FX77 Duo 23

Multi touch desktop LCD’s 24

iPhone en iPod Touch • The Apple iPhone has a multitouch display for resizing, zooming and rotating purposes. • Multi touch sensative screen • Motion sensor • Types of multi touch gestures: • Zoom, rotate 26

G1 Google Android phone 27

Palm Pre • WebOS. It’s a web-centric multitouch platform that makes us of CSS, XHTML and Javacript making it accessible and compatible with a wide range of applications. 28

And more mobile phone devices Meizu M8 HTC Touch Synaptics Onyx Nokia N97 29

Apple: MacBook Air • The Apple MacBook Air has multitouch touchpad for resizing, zooming and rotating purposes. 30

Eee PC 31

MultiTouch Cell • The MultiTouch LCDs are controlled by the MultiTouch CornerStone software. Cornerstone handles the core multitouch functionality, tracking users hands as they move on the screen. 32

Other vendors Accenture Panasonic Hitachi Philips Entertaible Perceptive Pixel Sony Smartskin HP Misto Sharp optical LCD Touchable

Market share of products with multi-touch • Multi-touch ability (or NUI) is competitor’s edge that is used in marketing • Market share is quickly rising • 10 mio iPhones sold in first 6 months • Table top sales booming since end 2008 Multi-touch is currently an “infant” technology • Different multi-touch imlementations (Apple, Microsoft, HP, etc.) • All are limited to a handful of gestures • Still a lot of traditional GUI interaction 34

Future product list • Input devices • Apple Mighty Mouse • Wacom tablets • Handheld game consoles • PSP2 • E-book readers • Desktops • Larger and flat tabletops • TV screens 35

Natural User Interfaces (NUI’s) Introduction • Touch screen interaction • Multi-touch screen interaction The vision • Evolution of computer user interfaces Reality • Natural User Interfaces (NUI) • Tangible interfaces • Multi modal interfaces • NUI design 36

Touch screen interaction Touch interaction • 1960: Touch • 1982: Multi touch • 2006: Lucid touch • Multi modal interfaces, controlled by • Touch • Gestures • Speech • Objects 39

Single Touch 40

Touch screen interfaces (since 1960’s) • Interaction via • Stylus, light pen, finger, hand,  Finger stress  Fingernail interaction  Fingerprints  “Gorilla arm” • No intelligent interaction • Gestures • Touch force sensing • Object sensing (finger vs hand) 41

Multi-touch & Natural User Interfaces 42

Multi-touch interaction • In the realworld we do lots of actions with two hands or more than one finger. e.g. driving a car • Operating a mouse are serial, discrete events and single point input • Multi point interaction allows more parallel interaction • Reduce task complexity of single input techniques • Increases parallesm in multi-finger interaction reduces time 43

Tangible interfaces 44

Interact with real objects 45

Haptics 46

Haptics 47

Haptics today 48

Lucid Touch 49

But lucid touch interaction is very complex 50

Gesture interaction 51

Gesture interaction today 52

Multimodal interaction 53

Multimodal interaction is still the future… 54

Paradigm shift Command Line Interfaces (CLI) 55

Paradigm shift Graphical User Interfaces (GUI) 56

Paradigm shift Natural User Interfaces (NUI) 57

Paradigm shift • Static • Responsive • Evocative • Disconnected • Indirect • Unmediated • High-low • Double medium • Fast few • Directed • Exploratory • Contextual • Recall • Recognition • Intuition • Abstract • Indirect • Direct • Textual • Graphical • Physical 58 Source: Dennis Wixon

Overview of Human Computer Interfaces Microsoft Microsoft Windows NT Windows Vista Alto PC Microsoft Microsoft Xerox Parc Windows 1.0 Windows 95 Microsoft MS-DOS Windows 98 Microsoft Microsoft Microsoft Q2 2009 Windows 3.0 Windows XP Windows 7 1993 1995 2001 1982 1985 1998 2007 1990 1973 1970 2010 Jun 2007 1987 1997 Microsoft 2001 1984 1992 2005 2009 Windows Snow Tablet PC Leopard? Apple Macintosh edition IBM OS/2 2.0 Apple Mac OS 8 Apple Mac OS X 10.5 Apple Mac OS X 10.0 Apple Mac Color Note: Linux excluded

Interaction styles and elements 1. Punch-card input (50’s-60’s) 2. Teletype writer connected to mainframe (60’s -80’s) EVOLUTION 3. CLI (80’s) 4. GUI (90’s-00’s) • Classic WIMP (Window, Icon, Menu, Pointer) • Desktop metaphor • Post-WIMP interfaces • Computer games • Virtual reality systems • Physical controls (e.g. iPod wheel) • Multi-touch interaction 3. NUI • STaG (Speech, Touch and Gesture) 4. XUI • Synthesis, anticipatory, constant fluid interaction 60

The vision of Natural User Experience (NUE) is to deliver intuitive, seamless experiences that unfold through natural human input.” • “Surfaces that respond to us” • “The content is the interface” 61

Surface: design for emotion • Natural & Intuitive • Unique & Magical • Social & Together • Aware & Responsive • Premium & Authentic 62

Surface guidelines 1. Seamless 2. Social 3. Spatial 4. Super realism 5. Contextual environments 6. Scaffolding 7. Performance aesthetics 8. Direct manipulation

User Interface Design for NUI’s Difference in designing Tabletop interfaces? • No fixed top-bottom orientation; • Multiple “mouse cursors”; • Multiple simultaneous users; • Multiple user contexts; • Multiple simultaneous actions, such as ‘drag & drop’; • No more tradional data input means. 64

Multi-touch in the movies • 1982: Tron • 2002: Minority report • 2005: The Island • 2008: Quantum of Solace (James Bond) • 2008: CSI: Miami 65

1982: Tron 66

2002: Minority Report 67

2005: The Island 68

2008: Quantum of Solace 69

2008: CSI: Miami 70

Markets and applications • Entertainment: (serious) gaming and fun • Public safety and crisis management • Retail • Public Places • Healthcare • Education • Financial • Office • Industrial use 71

Healthcare • Photo tool for dementia patients • Tool for patient-doctor relationships 72

Entertainment • Games (serious/fun) 73

Public places • Information centre New York • Why? • Intuitive, interactive and focussed on discovering New York • Target group: Tourists • Device • Gesturetek illuminate tables • Application: • Maps, POI, multimedia content 74 Information centre New York

Museums 75

TV presentations CNN – presidential elections

Retail 78

Horeca • Restaurants, hotels • Viewing menu • Entertainment (games, gambling) • Tourist information Surfaceware: detect drink level 79

iBar: object recognizing bar • Bar detects all objects on the surface • It detects the shape, it doesn’t identify the object. • future releases will also be able to identify objects • http://www.i-bar.ch/info/ 80

Education 81

Control Rooms AT&T Telecomunication control room OODA loop 1. Static risk map 2. real-time or simulated flooding 3. Real-time information feeds 82

Public safety • Manages planning, tracking and report security matters surrounding an event. The Super Bowl E•SPONDER solution 83

Financial services Use in a bank lobby for customers to look at account features and sit down with a banker to discuss their financial needs. 84

i-phone multi-touch applications • Photo’s: create a collage • Graffiti • Scratch Mixmeister • Games Foto collages maken Muziek scratchen en mixen puistjes uitdrukken 85

Industrial use • Insight in processes • Observe, control and decide • Collaborative design • Fraunhofer: • Virtual factory on a tabletop

Collaborative work in the office 87

Office Digidesk

TNO Work • Multi -touch in Control Rooms • Multi –touch in Education • Multi-touch : complex gesture support • MiReCol: collaboration and co-design • Slide overview + video Maurice 89

Control Rooms

Experiment area 91

Visualisation of information (1.4)(2.4)(21.6)(4.2)(7.6)(2.6)(9.8)(3.0) Quick insight Large data amounts Unstructured Time critical Diverse sources

Data sources information exchange Virtual teams Control room teams Control room decision centre 93

Control Rooms AT&T Telecomunication control room OODA loop 1. Static risk map 2. real-time or simulated flooding 3. Real-time information feeds 94

Crisis management scenario Satellite or street map Multi-user interface Ten Boer Risk objects Real-time communication log Annotate Analyse Anticipate Change view, filter, log

MiReCol: Mixed Reality Collaboration

Mixed reality collaboration Augmented collaboration Shared Augmented virtuality understanding collaboration Measuring and facilitating collaboration

Collaboration • Co-creation, co-design • Decision making • CSCW support in general 100

Teliris telepresence

Education 102

Serious gaming & multi touch interactie in het onderwijs Research goal: Does the combination of serious gaming and multi touch interaction add value to elementary education. Serious games Target group: School children aged 8 until 12 with a social disorder (PDD-NOS). Multi touch Results: a serious game was developed to support and stimulate several collaboration forms in calculation tasks. Through multitouch and multi user interaction these children could cooperate using one Education system. Partners: Bladergroenschool (elementary school for special education) and RENN4.

Supporting gesture interaction

Supporting gesture interaction with gesture previews Gestures • In general: non-verbal communication. • Specific: a set of measured movements over time. • Examples: letter recognition, unlocking your iPhone. Advanced gestures • Multi-touch gestures: zooming an image. • Collaborative gestures: clearing a drawing board. Problem statement • Most gestures have no intrinsic meaning. • Users have to know gestures in advance. • A steep learning curve especially for complex systems using advanced gestures.

Solution: gesture previews • Overlay with visual feedback. • Shows what gestures can be performed. • Shows where others can join in. • Indicates the meaning of gestures. • Previews do not limit expert users. • Increase detection rate, decrease learning curve.

Proof of Concept • Bejeweled clone, uses gestures to identify users. With some twists. • Gestures are stored in XML. <?xml version="1.0" encoding="utf-8"?> <root> <segmentDeclaration name="line" uniformScaling="false" <point x="0" y="0" /> <point x="0" y="0,1" /> </segmentDeclaration>

Wrap up Knowledge • Started at TNO just a few years ago, experimental. Now this topic is booming, backed up by large vendors and coming to the market. Hardware/software • More vendors enabling MT for their devices • Current software / applications are dedicated solutions • Wide adoption needs more hardware and applications Interaction • Multi touch is part of the NUI paradigm shift in Human Machine Interaction Markets / applications • Wow-technology, but the real added value and useful applications still have to prove themselves TNO • Added value in collaboration environments (crisismanagement, education and many more projects) 109

Arnout de Vries Senior scientist Natural User Interfaces Arnout.devries@tno.nl http://www.tno.nl/nui

http://www.touchscreentables.com	1551
http://moodle.flshase.uni.lu	630
http://www.touchuserinterface.com	433
http://www.slideshare.net	221
http://abbuzz.blogspot.com	161
http://www.gadgetsenonderwijs.nl	145
http://cityray.blogspot.com	140
http://conociendoelmundodigital.wikispaces.com	139
http://www.techgig.com	79
http://cityray.blogspot.tw	63
http://narrativisions.com	62
http://ctoutcom.blogspirit.com	42
http://moita.blogs.ua.sapo.pt	34
http://abbuzz.blogspot.it	31
http://www.levidepoches.blogs.com	28
http://pjarte.fr	21
http://mindpractice.tumblr.com	15
http://user-experience-specialist.blogspot.com	11
http://translate.googleusercontent.com	11
http://dmt-digitalmedia.blogspot.com	9
http://static.slidesharecdn.com	9
https://jujo00obo2o234ungd3t8qjfcjrs3o6k-a-sites-opensocial.googleusercontent.com	6
https://www.facebook.com	5
http://kodblog.com	4
http://narrativisions.posterous.com	4
http://www.levidepoches.fr	3
http://www.linkedin.com	2
http://10.150.200.57	2
http://www.cityray.blogspot.com	2
http://abbuzz.blogspot.in	2
http://abbuzz.blogspot.com.au	2
http://www.kodblog.com	2
https://webcourses.niu.edu	2
http://hosup.textcube.com	2
http://cityray.blogspot.nl	1
http://feeds.feedburner.com	1
http://cityray.blogspot.hk	1
http://cityray.blogspot.de	1
http://edcecocity.blogspot.in	1
http://touchscreendisplay.blogspot.com	1
http://gotactile.fr	1
http://user-experience-specialist.blogspot.pt	1
http://touchscreendisplay.blogspot.pt	1
http://cityray.blogspot.jp	1
http://cityray.blogspot.gr	1
http://74.125.153.132	1
https://si0.twimg.com	1
http://haig.soup.io	1
http://bb9dev.valpo.edu	1
https://remote.khm.be	1
http://levidepoches.blogs.com	1

Multi-Touch Interaction Overview

by TNO on Dec 01, 2009

Accessibility

Categories

Upload Details

Usage Rights

59 Embeds 3,898

Statistics

Multi-Touch Interaction Overview Presentation Transcript