The document discusses natural user interfaces (NUI) and natural user experience (NUX). It provides an overview and introduction to NUI/NUX, examples of NUI/NUX in media and real world devices, and tools and SDKs for developing NUI applications. It also previews demos of hand tracking and gestures using the Kinect SDK.

1. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Global Services
Michael Heydt
Principal Global Technologist
Natural User Interfaces in a Nutshell
October 15, 2011

2. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Agenda
! Introduction
! What is NUI/NUX
! NUX In the media and the wild
! Real-world devices
! Tools and SDK’s
! Demos using Kinect
! Come visit our booth for more!

3. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
About Mike
! Principal global technologist, Advanced Technology
Services, SunGard Global Services
! Technology focus on .NET, cloud, parallel systems, rich
interfaces and NUX
! Executive Masters Technology Management and
Innovation, UPenn / Wharton
! mike@heydt.org, michael.heydt@sungard.com
! @mikeheydt
! 42spikes.com

4. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission 4
SGS Advanced Technology
What is Advanced Technology?
SunGard Consulting Services Approach
Typical Client Engagements
! Our consultants offer deep domain experience and a broad set of leading edge technology skills,
allowing companies to optimize their business processes
! We use a variety of Agile project management and execution techniques to facilitate quick delivery and
reduce risk
! We take an objective approach in the selection of tools and products, and are comfortable working
with a variety of vendor products and platforms
! We have applied our advanced technology skills to provide solutions to customers in finance, energy
and media sectors
! Typical assignments involve deploying highly skilled specialists with relevant domain knowledge who
work with customers in a collaborative way to design and build custom solutions
! MapReduce/Hadoop
! Complex Events Processing
! GPU Mathematical Pricing
! Mashups
! Web 2.0
! Cloud & Grid Computing
! Functional Languages
! Mobile Applications
! RIA (Silverlight/Flex)
! Semantic Web
! REST/JSON
! Grails

5. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Opening Questions…
! Who here has used a Kinect on the Xbox?
! Who has programmed a Kinect?
! Who has used a WiiMote on the Wii?
! Who has programmed a wiimote in their application?
! Who has seen Minority Report?

6. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
What is NUI/NUX?
! In computing, a natural user
interface, or NUI, is the
common parlance used by
designers and developers of
computer interfaces to refer to a
user interface that is effectively
invisible, or becomes invisible
with successive learned
interactions, to its users.
! NUI Covers the class of
interfaces where the user is
enabled to interact with the
computer in a more natural
manner.

7. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Business
! Interactive data manipulation and visualization
! Free-form visual application development
! Media presentations
! Home control
! Oil reserve exploration

8. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Genesis
! Conceptually has been thought of for several decades
! Made popular by Minority Report
! iPad has changed the game with single-task gestural
apps
! Xbox Kinect has broken open the flood gates for NUX
! NUX is starting to show adoption in operating systems
through the use of tablet and trackpad gestures

9. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
in the Media and the Wild
! XBox360
! Operating systems
! Movies and television
! Science

10. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
XBox360’s New Dashboard

11. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
OS X Lion

12. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Windows 8 / Metro

13. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
TV: Caprica

14. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Movies: Minority Report

15. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Movies: Minority Report

16. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Movies: Minority Report

17. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Science: World Wide Telescope

18. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Real World NUX Devices
! Surface
! Kinect
! PrimeSense
! Wiimote
! Emotiv

19. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Surface

20. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Surface Capabilities
! Multi-user
! Multi-touch
! Also camera based identification and tracking
! Built in wireless to devices

21. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Surface 2.0

22. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect
! Laser Depth Imager
! Video Camera
! Microphone Array
! Very XBox Specific Interface (non-standard USB)
! Microsoft SDK, non-commercial license
! $150

23. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect: An Evolution?

24. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
PrimeSense
! Only laser depth finder
! Standard USB
! OpenNI drivers
! Semi-open source
! commercial license

25. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Johnny 5

26. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Wiimote

27. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Emotiv
http://www.vitodibari.com/en/tech-
entrepreneur-tan-le-explains-emotiv-turn-
thoughts-actions-2.html

28. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Elements of NUX
! Depth Imaging
! Skeleton Tracking
! Hand and Finger Tracking
! Gesture Identification
! Voice Recognition
! Thought Recognition (yep, you heard it here first)

29. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect: What You Need
! A Kinect
! Fastest selling consumer electronics device ever
! A PC or Mac (or Linux)
! SDKs

30. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect SDK
http://research.microsoft.com/en-
us/um/redmond/projects/kinectsdk/

31. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
OpenNI
• www.openni.org/
• Pseudo-open source

32. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Data from the Kinect
! Multiple streams of data:
! Video
! Audio
! Depth
! Skeleton(s)

33. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
The How of the Kinect: Depths
! You are scanned and tracked by an IR laser
! Depths are measured to the millimeter at 320 x 240
! From depths skeletons are identified and tracked

34. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Kinect: What You Don’t See

35. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Depth Mapping
• Distance to the millimeter
• Used to create maps for
“skeletons”

36. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Skeleton Tracking
• Built from the depth map
• 1..n skeletons
• Done in software not the
hardware

37. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Hand and Finger Tracking
Not a part of any SDK

38. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Speech
! Kinect has a very good set of microphones
! Speech recognition is not in the Kinect SDK - use
the .NET libraries
! Must be bridged with Microsoft Speech Platform SDK
! I can’t stress this enough – this is excellent
augmentation

39. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Gesture Recognition
! Not built into any of the devices
! Typically implemented with
Dynamic Time Warping
algorithms (DTW)
! Record sequences of skeleton
locations, match against live
streams

40. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Hand Recognition
! Not built into any of the devices
! General Algorithm:
! Get depth data
! Point cloud identification
! Center of mass determination
! K-Curvature Analysis
! Least squares fitting
! Rendering and event processing

41. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Difficulties
! Many devices
! Immature API’s and libraries
! Small body of work
! Nonstandard USB
! IR data is noisy
! Low resolution
! Single device focus
! No off the shelf gesture
engines
! No standard pattern
languages for gestures
! Point clouds don’t map
100% to video
! Setting are finicky… Tend to
be very distance and angle
specific

42. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
What I’m Building: NuiDotNet
! NuiDotNet (nuidotnet.codeplex.com, nuidot.net)
! Abstracts devices from the application
! Allows configuration and location based construction of services
! Multiple and prioritized services and interactions mapped to
application functions
! Gesture engine and domain specific language for gestures
! Abstraction of details of gestures and algorithms
! Release date? Soon…
! I’ll show some of it in a little bit…

43. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Directions
! Define useful set of gesture patterns for data
manipulation
! Build finger / hand tracking capabilities
! Make the DTW gesture engine robust
! Tracking of specific parts of the body
! Architecture and software to plug into business
applications
! Provide a library of pluggable UI elements and gestures
! Easing of the issues with noise, distance and angles

44. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Interactive Demos…
! Hand Tracking
! Finger Pointing
! Hand Gestures
! Voice Recognition
! Skeleton Tracking
! Gestures

45. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
NuiDotNet Basics: Hand Tracking / Visualization

46. www.sungard.com/globalservicesProprietary and Confidential. Not to be distributed or reproduced without permission
Gesture Models in RavenDB