1. Natural User Interfaces
Natural User
Hardware
Interfaces
Markets and
Markets and
Hardware and software applications
Software applications
History and future Timeline
Multi-touch interaction TNO work
TNO
Work
Arnoud de Jong
Arnout de Vries
TNO Information and Communication Technologies
Multi- Multi-
Summary
touch touch in
and
products the
Conclusions
1 movies

2. History of Multi-touch interaction
Pierre Wellner paper
Foundation
on multi touch and
iGesture Pad and NUI Group
First multi pinching motions
TouchStream keyboard
hand display by Fingerworks
Bell Labs Palm
Sony labs Microsoft Pre
First MT
SmartSkin surface PlayAnywhere
display
Toronto Mixed Reality table of Touch and object
university Microsoft Research detection Hitachi
Q2 2009
Smartboard
2002
1982
2005
1991
1999
2006
2001
1984
FX77 Duo
1980 2009
Jun 2007
2002
Q3 2008
Q1 2009
2003
1992
1982
1985
2006
Q1 2006
University of
2001
Movie:
3D multi-touch Toronto
Tron Microsoft
sensitive tablet Simon - first Multi-finger
Windows 7
smartphone Multi-hand
Multi-user Lucid
With Mitsubishi Touch
interaction iPhone
touchscreen MERL R&D Lab release
DiamondTouch
HP-150 world’s Microsoft
Surface
1983
first
Jeff Han SMART
touchscreen PC Movie: @TED table
Minority
report
2

3. Hardware technology
• Multi-point detection
• Multi-object detection
• Setup
3

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. 11

12. Setups
12

13. Products
• Table tops
• Tablet PC’s
• Whiteboards
• Large surfaces (walls, floors, tables)
• Desktop PC’s
• Netbooks / UMPC’s
• Mobile phones
• TV’s
13

14. Fraunhofer IGD multi-touch table
• The table is running on Linux
and uses the InstantReality
framework for displaying and
interacting.
• The finger tracking is done by
the Fraunhofer software
VisionLib which is not available
for pulic right now.
• Table size is 150x90 cm
14

15. Mitsubishi: DiamondTouch
• Front-projection on
typically 32" or 42"
diagonal
• Multiple touches
support by a single
user
• Distinguishes between
simultaneous inputs
from multiple users.
• DiamondTouch SDK
or DTFlash toolkit.
15

16. Gesturetek
16

17. Horizon

18. SMART table

19. Microsoft: Surface
• www.surface.com
• Object on table are identified by ‘domino tags’
• http://worldofitgeeks.blogspot.com/2007/10/domino-theories-for-microsofts-surface.html
19

20. HP TouchSmart
20

21. HP Touchsmart TX2 tablet PC

22. Dell Latitude XT
22

23. Hitachi Starboard FX77 Duo
23

24. Multi touch desktop LCD’s
24

25. TV
25

26. iPhone en iPod Touch
• The Apple iPhone has a
multitouch display for resizing,
zooming and rotating purposes.
• Multi touch sensative screen
• Motion sensor
• Types of multi touch gestures:
• Zoom, rotate
26

27. G1 Google Android phone
27

28. Palm Pre
• WebOS. It’s a web-centric multi-
touch platform that makes us of
CSS, XHTML and Javacript
making it accessible and
compatible with a wide range of
applications.
28

29. And more mobile phone devices
Meizu M8 HTC Touch Synaptics Onyx
Nokia N97
29

30. Apple: MacBook Air
• The Apple MacBook Air
has multitouch touchpad
for resizing, zooming
and rotating purposes.
30

31. Eee PC
31

32. MultiTouch Cell
• The MultiTouch LCDs are controlled by the MultiTouch CornerStone software.
Cornerstone handles the core multitouch functionality, tracking users hands as
they move on the screen.
32

33. Other vendors
Accenture Panasonic Hitachi
Philips Entertaible Perceptive Pixel Sony Smartskin
HP Misto Sharp optical LCD Touchable

34. Market share of products with multi-touch
• Multi-touch ability (or NUI) is competitor’s edge that is used in
marketing
• Market share is quickly rising
• 10 mio iPhones sold in first 6 months
• Table top sales booming since end 2008
Multi-touch is currently an “infant” technology
• Different multi-touch imlementations (Apple, Microsoft, HP, etc.)
• All are limited to a handful of gestures
• Still a lot of traditional GUI interaction
34

35. Future product list
• Input devices
• Apple Mighty Mouse
• Wacom tablets
• Handheld game consoles
• PSP2
• E-book readers
• Desktops
• Larger and flat tabletops
• TV screens
35

36. Natural User Interfaces (NUI’s)
Introduction
• Touch screen interaction
• Multi-touch screen interaction
The vision
• Evolution of computer user interfaces
Reality
• Natural User Interfaces (NUI)
• Tangible interfaces
• Multi modal interfaces
• NUI design
36

39. Touch screen interaction
Touch interaction
• 1960: Touch
• 1982: Multi touch
• 2006: Lucid touch
• Multi modal interfaces, controlled by
• Touch
• Gestures
• Speech
• Objects
39

40. Single Touch
40

41. Touch screen interfaces (since 1960’s)
• Interaction via
• Stylus, light pen, finger, hand,
 Finger stress
 Fingernail interaction
 Fingerprints
 “Gorilla arm”
• No intelligent interaction
• Gestures
• Touch force sensing
• Object sensing (finger vs hand)
41

42. Multi-touch &
Natural User Interfaces
42

43. Multi-touch interaction
• In the realworld we do lots of actions with two hands
or more than one finger. e.g. driving a car
• Operating a mouse are serial, discrete events and
single point input
• Multi point interaction allows more parallel interaction
• Reduce task complexity of single input techniques
• Increases parallesm in multi-finger interaction reduces time
43

44. Tangible interfaces
44

45. Interact with real objects
45

46. Haptics
46

47. Haptics
47

48. Haptics today
48

49. Lucid Touch
49

50. But lucid touch interaction is very complex
50

51. Gesture interaction
51

52. Gesture interaction today
52

53. Multimodal interaction
53

54. Multimodal interaction is still the future…
54

55. Paradigm shift
Command Line Interfaces (CLI)
55

56. Paradigm shift
Graphical User Interfaces (GUI)
56

57. Paradigm shift
Natural User Interfaces (NUI)
57

58. Paradigm shift
• Static • Responsive • Evocative
• Disconnected • Indirect • Unmediated
• High-low • Double medium • Fast few
• Directed • Exploratory • Contextual
• Recall • Recognition • Intuition
• Abstract • Indirect • Direct
• Textual • Graphical • Physical
58
Source: Dennis Wixon

59. Overview of Human Computer Interfaces
Microsoft
Microsoft
Windows NT
Windows Vista
Alto PC Microsoft Microsoft
Xerox Parc Windows 1.0 Windows 95
Microsoft
MS-DOS Windows 98
Microsoft Microsoft Microsoft
Q2 2009
Windows 3.0 Windows XP Windows 7
1993
1995
2001
1982
1985
1998
2007
1990
1973
1970 2010
Jun 2007
1987
1997
Microsoft
2001
1984
1992
2005
2009
Windows
Snow
Tablet PC
Leopard?
Apple Macintosh edition
IBM OS/2 2.0
Apple Mac OS 8
Apple Mac OS X 10.5
Apple Mac OS X 10.0
Apple Mac Color
Note: Linux excluded

60. Interaction styles and elements
1. Punch-card input (50’s-60’s)
2. Teletype writer connected to mainframe (60’s -80’s)
EVOLUTION
3. CLI (80’s)
4. GUI (90’s-00’s)
• Classic WIMP (Window, Icon, Menu, Pointer)
• Desktop metaphor
• Post-WIMP interfaces
• Computer games
• Virtual reality systems
• Physical controls (e.g. iPod wheel)
• Multi-touch interaction
3. NUI
• STaG (Speech, Touch and Gesture)
4. XUI
• Synthesis, anticipatory, constant fluid interaction
60

61. The vision of Natural User Experience
(NUE) is to deliver intuitive, seamless
experiences that unfold through natural
human input.”
• “Surfaces that respond to us”
• “The content is the interface”
61

62. Surface: design for emotion
• Natural & Intuitive
• Unique & Magical
• Social & Together
• Aware & Responsive
• Premium & Authentic
62

63. Surface guidelines
1. Seamless
2. Social
3. Spatial
4. Super realism
5. Contextual environments
6. Scaffolding
7. Performance aesthetics
8. Direct manipulation

64. User Interface Design for NUI’s
Difference in designing Tabletop interfaces?
• No fixed top-bottom orientation;
• Multiple “mouse cursors”;
• Multiple simultaneous users;
• Multiple user contexts;
• Multiple simultaneous actions, such as ‘drag & drop’;
• No more tradional data input means.
64

65. Multi-touch in the movies
• 1982: Tron
• 2002: Minority report
• 2005: The Island
• 2008: Quantum of Solace (James Bond)
• 2008: CSI: Miami
65

66. 1982: Tron
66

67. 2002: Minority Report
67

68. 2005: The Island
68

69. 2008: Quantum of Solace
69

70. 2008: CSI: Miami
70

71. Markets and applications
• Entertainment: (serious) gaming and fun
• Public safety and crisis management
• Retail
• Public Places
• Healthcare
• Education
• Financial
• Office
• Industrial use
71

72. Healthcare
• Photo tool for dementia patients
• Tool for patient-doctor relationships
72

73. Entertainment
• Games (serious/fun)
73

74. Public places
• Information centre New York
• Why?
• Intuitive, interactive and focussed
on discovering New York
• Target group: Tourists
• Device
• Gesturetek illuminate tables
• Application:
• Maps, POI, multimedia content
74
Information centre New York

75. Museums
75

77. TV presentations
CNN – presidential elections

78. Retail
78

79. Horeca
• Restaurants, hotels
• Viewing menu
• Entertainment (games, gambling)
• Tourist information
Surfaceware:
detect drink level
79

80. iBar: object recognizing bar
• Bar detects all objects on the surface
• It detects the shape, it doesn’t identify the object.
• future releases will also be able to identify objects
• http://www.i-bar.ch/info/
80

81. Education
81

82. Control Rooms
AT&T Telecomunication control room
OODA loop
1. Static risk map 2. real-time or simulated flooding 3. Real-time information feeds
82

83. Public safety
• Manages planning, tracking and report
security matters surrounding an event.
The Super Bowl E•SPONDER solution
83

84. Financial services
Use in a bank lobby for
customers to look at
account features and sit
down with a banker to
discuss their financial
needs.
84

85. i-phone multi-touch applications
• Photo’s: create a collage
• Graffiti
• Scratch Mixmeister
• Games
Foto collages maken
Muziek scratchen en mixen
puistjes uitdrukken
85

86. Industrial use
• Insight in processes
• Observe, control and decide
• Collaborative design
• Fraunhofer:
• Virtual factory on a tabletop

87. Collaborative work in the office
87

88. Office
Digidesk

89. TNO Work
• Multi -touch in Control Rooms
• Multi –touch in Education
• Multi-touch : complex gesture support
• MiReCol: collaboration and co-design
• Slide overview + video Maurice
89

90. Control Rooms

91. Experiment
area
91

92. Visualisation of information
(1.4)(2.4)(21.6)(4.2)(7.6)(2.6)(9.8)(3.0)
Quick insight
Large data amounts
Unstructured
Time critical
Diverse sources

93. Data sources
information exchange
Virtual teams
Control room teams
Control room
decision centre
93

94. Control Rooms
AT&T Telecomunication control room
OODA loop
1. Static risk map 2. real-time or simulated flooding 3. Real-time information feeds
94

95. Crisis management scenario Satellite or street map
Multi-user interface
Ten Boer Risk objects
Real-time
communication log
Annotate
Analyse
Anticipate
Change view, filter, log

96. MiReCol: Mixed Reality Collaboration

97. Mixed reality collaboration
Augmented collaboration
Shared
Augmented virtuality understanding
collaboration
Measuring and facilitating
collaboration

100. Collaboration
• Co-creation, co-design
• Decision making
• CSCW support in general
100

101. Teliris telepresence

102. Education
102

103. Serious gaming & multi touch interactie in het onderwijs
Research goal: Does the
combination of serious gaming and
multi touch interaction add value to
elementary education. Serious games
Target group: School children aged
8 until 12 with a social disorder
(PDD-NOS).
Multi touch
Results: a serious game was
developed to support and stimulate
several collaboration forms in
calculation tasks. Through multi-
touch and multi user interaction these
children could cooperate using one Education
system.
Partners: Bladergroenschool
(elementary school for special
education) and RENN4.

104. Supporting gesture interaction

105. Supporting gesture interaction with gesture previews
Gestures
• In general: non-verbal communication.
• Specific: a set of measured movements over time.
• Examples: letter recognition, unlocking your iPhone.
Advanced gestures
• Multi-touch gestures: zooming an image.
• Collaborative gestures: clearing a drawing board.
Problem statement
• Most gestures have
no intrinsic meaning.
• Users have to know gestures
in advance.
• A steep learning curve
especially for complex systems
using advanced gestures.

106. 106

107. Solution: gesture previews
• Overlay with visual feedback.
• Shows what gestures can be performed.
• Shows where others can join in.
• Indicates the meaning of gestures.
• Previews do not limit expert users.
• Increase detection rate, decrease learning curve.

108. Proof of Concept
• Bejeweled clone, uses gestures to identify users. With some twists.
• Gestures are stored in XML.
<?xml version="1.0" encoding="utf-8"?>
<root>
<segmentDeclaration name="line" uniformScaling="false"
<point x="0" y="0" />
<point x="0" y="0,1" />
</segmentDeclaration>

109. Wrap up
Knowledge
• Started at TNO just a few years ago, experimental. Now this topic is booming,
backed up by large vendors and coming to the market.
Hardware/software
• More vendors enabling MT for their devices
• Current software / applications are dedicated solutions
• Wide adoption needs more hardware and applications
Interaction
• Multi touch is part of the NUI paradigm shift in Human Machine Interaction
Markets / applications
• Wow-technology, but the real added value and useful applications still have to
prove themselves
TNO
• Added value in collaboration environments (crisismanagement, education and
many more projects)
109

110. Arnout de Vries
Senior scientist
Natural User Interfaces
Arnout.devries@tno.nl
http://www.tno.nl/nui