Natural User Interfaces
Hardware and software applications
History and future Timeline
Multi-touch interaction TNO work
Arnoud de Jong
Arnout de Vries
TNO Information and Communication Technologies
touch touch in
History of Multi-touch interaction
Pierre Wellner paper
on multi touch and
iGesture Pad and NUI Group
First multi pinching motions
hand display by Fingerworks
Bell Labs Palm
Sony labs Microsoft Pre
SmartSkin surface PlayAnywhere
Toronto Mixed Reality table of Touch and object
university Microsoft Research detection Hitachi
3D multi-touch Toronto
sensitive tablet Simon - first Multi-finger
With Mitsubishi Touch
touchscreen MERL R&D Lab release
HP-150 world’s Microsoft
Jeff Han SMART
touchscreen PC Movie: @TED table
Fraunhofer IGD multi-touch table
• The table is running on Linux
and uses the InstantReality
framework for displaying and
• The finger tracking is done by
the Fraunhofer software
VisionLib which is not available
for pulic right now.
• Table size is 150x90 cm
• Front-projection on
typically 32" or 42"
• Multiple touches
support by a single
• Distinguishes between
from multiple users.
• DiamondTouch SDK
or DTFlash toolkit.
iPhone en iPod Touch
• The Apple iPhone has a
multitouch display for resizing,
zooming and rotating purposes.
• Multi touch sensative screen
• Motion sensor
• Types of multi touch gestures:
• Zoom, rotate
• The MultiTouch LCDs are controlled by the MultiTouch CornerStone software.
Cornerstone handles the core multitouch functionality, tracking users hands as
they move on the screen.
Accenture Panasonic Hitachi
Philips Entertaible Perceptive Pixel Sony Smartskin
HP Misto Sharp optical LCD Touchable
Market share of products with multi-touch
• Multi-touch ability (or NUI) is competitor’s edge that is used in
• Market share is quickly rising
• 10 mio iPhones sold in first 6 months
• Table top sales booming since end 2008
Multi-touch is currently an “infant” technology
• Different multi-touch imlementations (Apple, Microsoft, HP, etc.)
• All are limited to a handful of gestures
• Still a lot of traditional GUI interaction
Future product list
• Input devices
• Apple Mighty Mouse
• Wacom tablets
• Handheld game consoles
• E-book readers
• Larger and flat tabletops
• TV screens
Natural User Interfaces (NUI’s)
• Touch screen interaction
• Multi-touch screen interaction
• Evolution of computer user interfaces
• Natural User Interfaces (NUI)
• Tangible interfaces
• Multi modal interfaces
• NUI design
• In the realworld we do lots of actions with two hands
or more than one finger. e.g. driving a car
• Operating a mouse are serial, discrete events and
single point input
• Multi point interaction allows more parallel interaction
• Reduce task complexity of single input techniques
• Increases parallesm in multi-finger interaction reduces time
Overview of Human Computer Interfaces
Alto PC Microsoft Microsoft
Xerox Parc Windows 1.0 Windows 95
MS-DOS Windows 98
Microsoft Microsoft Microsoft
Windows 3.0 Windows XP Windows 7
Apple Macintosh edition
IBM OS/2 2.0
Apple Mac OS 8
Apple Mac OS X 10.5
Apple Mac OS X 10.0
Apple Mac Color
Note: Linux excluded
The vision of Natural User Experience
(NUE) is to deliver intuitive, seamless
experiences that unfold through natural
• “Surfaces that respond to us”
• “The content is the interface”
Surface: design for emotion
• Natural & Intuitive
• Unique & Magical
• Social & Together
• Aware & Responsive
• Premium & Authentic
4. Super realism
5. Contextual environments
7. Performance aesthetics
8. Direct manipulation
User Interface Design for NUI’s
Difference in designing Tabletop interfaces?
• No fixed top-bottom orientation;
• Multiple “mouse cursors”;
• Multiple simultaneous users;
• Multiple user contexts;
• Multiple simultaneous actions, such as ‘drag & drop’;
• No more tradional data input means.
Multi-touch in the movies
• 1982: Tron
• 2002: Minority report
• 2005: The Island
• 2008: Quantum of Solace (James Bond)
• 2008: CSI: Miami
• Information centre New York
• Intuitive, interactive and focussed
on discovering New York
• Target group: Tourists
• Gesturetek illuminate tables
• Maps, POI, multimedia content
Information centre New York
• Restaurants, hotels
• Viewing menu
• Entertainment (games, gambling)
• Tourist information
detect drink level
iBar: object recognizing bar
• Bar detects all objects on the surface
• It detects the shape, it doesn’t identify the object.
• future releases will also be able to identify objects
Serious gaming & multi touch interactie in het onderwijs
Research goal: Does the
combination of serious gaming and
multi touch interaction add value to
elementary education. Serious games
Target group: School children aged
8 until 12 with a social disorder
Results: a serious game was
developed to support and stimulate
several collaboration forms in
calculation tasks. Through multi-
touch and multi user interaction these
children could cooperate using one Education
(elementary school for special
education) and RENN4.
Supporting gesture interaction with gesture previews
• In general: non-verbal communication.
• Specific: a set of measured movements over time.
• Examples: letter recognition, unlocking your iPhone.
• Multi-touch gestures: zooming an image.
• Collaborative gestures: clearing a drawing board.
• Most gestures have
no intrinsic meaning.
• Users have to know gestures
• A steep learning curve
especially for complex systems
using advanced gestures.
Solution: gesture previews
• Overlay with visual feedback.
• Shows what gestures can be performed.
• Shows where others can join in.
• Indicates the meaning of gestures.
• Previews do not limit expert users.
• Increase detection rate, decrease learning curve.
Proof of Concept
• Bejeweled clone, uses gestures to identify users. With some twists.
• Gestures are stored in XML.
<?xml version="1.0" encoding="utf-8"?>
<segmentDeclaration name="line" uniformScaling="false"
<point x="0" y="0" />
<point x="0" y="0,1" />
• Started at TNO just a few years ago, experimental. Now this topic is booming,
backed up by large vendors and coming to the market.
• More vendors enabling MT for their devices
• Current software / applications are dedicated solutions
• Wide adoption needs more hardware and applications
• Multi touch is part of the NUI paradigm shift in Human Machine Interaction
Markets / applications
• Wow-technology, but the real added value and useful applications still have to
• Added value in collaboration environments (crisismanagement, education and
many more projects)
Arnout de Vries
Natural User Interfaces