David Silverlight's powerpoint presentation on the Kinect for Windows SDK. Feb. 29, 2012 NUI = Natural User Interface: it's an invisible interface, the content is the interface, removing the proxy, direct manipulation, gestural interfaces Kinect for Windows SDK: 1. Kinect explorer 2. Installing & using the Kinect sensor 3. Setting up your dev environment 4. Skeletal tracking fundamentals 5. Working with depth data 6. Audio fundamentals 7. Camera fundamentals

1. KINECT FOR WINDOWS
It’s All NUI to me:
Building Apps with the Microsoft Kinect SDK
David Silverlight
Founder and Chief Software Architect
TheSilverlightGroup.com
IsYourWebMasterDead.com

2. KINECT FOR WINDOWS
Agenda
• NUI – What?
• Demo’s – See what all the hullabaloo is about
• Getting Started/Kinect Explorer/QuickStarts
• Building Your Own Kinect apps.
• (Video, Depth, Skeleton, Audio)
• Resources

3. KINECT FOR WINDOWS
Kinect: Setter of World Records!

4. KINECT FOR WINDOWS
And Some Disasters

5. KINECT FOR WINDOWS
It’s in the headlines

6. KINECT FOR WINDOWS
What is NUI?
• Natural User Interface
• Common term used by designers and
developers of computer interfaces to refer to a
user interface that is effectively invisible, or
becomes invisible with successive learned
interactions, to its users

7. KINECT FOR WINDOWS
What is NUI?
• An Invisible Interface.
• The Content is the Interface.
• Removing the Proxy
• Direct Manipulation
• Gestural interfaces

8. KINECT FOR WINDOWS
Seeing is believing – Kinect Development
Allows you to capture
your movements and
opens up a whole new
world to animators on
a tight budget.
12 Best Kinect Hacks Personal Data Mirror
Catch up with the
latest headlines,
weather and health
all whilst brushing
your teeth
Real Time Motion
Capture with XNA
Sample showing just
what people are
doing with the
Kinect

9. KINECT FOR WINDOWS
Fitnect Virtual Vacation
Now another reason
to never leave the
house.
Interactive Fitting
room showing how
Kinect and business
can connect.
Seeing is believing – Kinect Development

10. KINECT FOR WINDOWS
What can you do with Kinect on a PC or Mac ?

11. KINECT FOR WINDOWS
RGB CAMERA
MULTI-ARRAY MIC MOTORIZED TILT
3D DEPTH SENSORS

12. KINECT FOR WINDOWS
Getting Started: Building Your Own Apps
• Designed to get you started
• Assumes you have some programming experience
• Managed API - Concepts work in both VB/C#

13. KINECT FOR WINDOWS
Kinect USB cable

14. KINECT FOR WINDOWS
System Requirements
• You must run applications that are built by using the Kinect for Windows
SDK in a native Windows environment. You cannot run applications in a
virtual machine, because the Microsoft Kinect drivers and this SDK must be
installed on the computer that is running the application.
• Supported Operating Systems and Architectures
• Windows 7 (x86 or x64)
• Hardware Requirements
• Computer with a dual-core, 2.66-GHz or faster processor
• Windows 7–compatible graphics card that supports Microsoft® DirectX® 9.0c
capabilities
• 2 GB of RAM
• Kinect for Xbox 360® sensor—retail edition, which includes special USB/power cabling

15. KINECT FOR WINDOWS
System Requirements
• Software Requirements
• Microsoft Visual Studio® 2010 Express or other Visual Studio 2010 edition
• Microsoft .NET Framework 4.0 (installed with Visual Studio 2010)
• For C++ SkeletalViewer samples:
• DirectX Software Development Kit, June 2010 or later version
• DirectX End-User Runtime Web Installer
• For Speech sample (x86 only):
• Microsoft Speech Platform - Server Runtime, version 10.2 (x86 edition)
• Microsoft Speech Platform - Software Development Kit, version 10.2 (x86
edition)
• Kinect for Windows Runtime Language Pack, version 0.9
(acoustic model from Microsoft Speech Platform for the Kinect for Windows
SDK Beta)

16. KINECT FOR WINDOWS
It all starts here: http://www.microsoft.com/en-us/kinectforwindows/
Developing your own Kinect aps

17. KINECT FOR WINDOWS
Kinect for Windows SDK Quickstarts
• http://channel9.msdn.com/Series/KinectQuickstart - All are
great, IMHO
• The Kinect Explorer is new from the Beta
• Kinect for Windows SDK Browser.
• Installing and Using the Kinect Sensor
• Setting up your development environment
• Skeletal Tracking fundamentals
• Camera Fundamentals
• Working with Depth Data
• Audio Fundamentals

18. KINECT FOR WINDOWS
SETTING UP
YOUR DEVELOPMENT
ENVIRONMENT

19. KINECT FOR WINDOWS
Setting up your Development Environment
Add our references to:
- The Microsoft.Samples.Kinect.WpfViewers Project
- The Coding4Fun.Wpf.dll
Reference the assembly:
• using Microsoft.Kinect;

20. KINECT FOR WINDOWS
KINECT API BASICS
• Manage Kinect state
• Connected
• Enable Color, Depth, Skeleton
• Start Kinect
• Get Data
• Events - AllFramesReady
• Polling – OpenNextFrame

21. KINECT FOR WINDOWS
RESOURCES
• Install Kinect Explorer
• KinectWpfViewers
• Coding4Fun Toolkit
• Skeletal scaling

22. KINECT FOR WINDOWS
SETTING UP YOUR DEV
ENVIRONMENT
DEMO

23. KINECT FOR WINDOWS
Setting the
Camera Tilt Angle
CAMERA
FUNDAMENTALS

24. KINECT FOR WINDOWS
Setting the Camera Tilt Angle
• Warning: The tilt mechanism in the sensor array is not rated for
frequent use. Your code should not make calls to tilt the device more
than 15 times in any two-minute window. Changing the tilt too often
results in an error message from the function.
• To adjust the tilt or pitch, set the ElevationAngle property to a value
between –27 (Camera.ElevationMinimum) and +27
(Camera.ElevationMaximum). The code below sets the camera to the
maximum elevation:
• kinectSensorChooser.Kinect.ElevationAngle =
Convert.ToInt16(slider1.Value);

25. KINECT FOR WINDOWS
CAMERA
FUNDAMENTALS

26. KINECT FOR WINDOWS
RESOLUTIONS
• Color
• 12 FPS: 1280X960 RGB
• 15 FPS: Raw YUV 640x480
• 30 FPS: 640x480
• Depth
• 30 FPS: 80x60, 320x240, 640x480

27. KINECT FOR WINDOWS
Camera Data

28. KINECT FOR WINDOWS
CAMERA DATA
• Events return ImageFrame
• PixelDataLength
• FrameNumber
• Timestamp
• Dimensions: Height, Width
• Use AllFramesReady event to synchronize

29. KINECT FOR WINDOWS
CAMERA FUNDAMENTALS
DEMO

30. KINECT FOR WINDOWS
WORKING WITH
DEPTH DATA

31. KINECT FOR WINDOWS
DEPTH DATA
• Returns the distance and player for every pixel
• Ex: 320x240 = 76,800 pixels
• Distance
• Distance in mm from Kinect ex: 2,000mm (6.56 feet)
• Player
• 1-6 players

32. KINECT FOR WINDOWS
MODE
Mode Depth & Player Center Hip Joint Other 19 Joints
Default Yes Yes Yes
Near Yes Yes No, for v1.0

33. KINECT FOR WINDOWS
DISTANCES
1.3’ 2.6’ 9.8’ 13.1’ 26.2’
.4 .8 3 4 8
Default
Mode
Near
Mode
Feet
Meters

34. KINECT FOR WINDOWS
FORMULAS
• Distance Formula
int depth = depthPoint >> DepthImageFrame.PlayerIndexBitmaskWidth;
• Player Formula
int player = depthPoint & DepthImageFrame.PlayerIndexBitmask;

35. KINECT FOR WINDOWS
WORKING WITH DEPTH DATA
DEMO

36. KINECT FOR WINDOWS
SKELETAL
TRACKING
FUNDAMENTALS

37. KINECT FOR WINDOWS
Joints
• Maximum two players tracked at once
• Six player proposals
• Each player with set of <x, y, z> joints in meters
• Each joint has associated state
• Tracked, Not tracked, or Inferred
• Inferred - Occluded, clipped, or low confidence joints
• Not Tracked - Rare, but your code must check for this state

38. KINECT FOR WINDOWS
Skeleton Data

39. KINECT FOR WINDOWS
Skeleton API

40. KINECT FOR WINDOWS
SKELETAL TRACKING
DEMO

41. KINECT FOR WINDOWS
AUDIO
FUNDAMENTALS

42. KINECT FOR WINDOWS
QUICKSTART SERIES
• Installing and Using the Kinect Sensor
• Setting up your Development Environment
• Camera Fundamentals
• Working with Depth Data
• Skeletal Tracking Fundamentals
• Audio Fundamentals

43. KINECT FOR WINDOWS
SOUND POSITION
• Sound Source Angle – the angle and confidence level of where
audio is coming from
• Beam Angle – The angle used to record audio that you can set as a
“directional microphone”
-50 0 +50

44. KINECT FOR WINDOWS
RECORDING
• KinectAudioSource audio buffer
• Use Echo Cancellation to not record PC sound

45. KINECT FOR WINDOWS
SPEECH RECOGNITION
• Grammar – What we are listening for
• Code – GrammarBuilder, Choices
• Note: Ensure AutomaticGainControl = false

46. KINECT FOR WINDOWS
GRAMMAR
<!-- Confirmation_Yes._value: string ["Yes"]
-->
<rule id="Confirmation_Yes" scope="public">
<example> yes </example>
<example> yes please </example>
<one-of>
<item> yes </item>
<item> yeah </item>
<item> yep </item>
<item> ok </item>
</one-of>
<item repeat="0-1"> please </item>
<tag> out._value = "Yes";</tag>
var grammar = new Choices();
grammar.Add(“yes please");
grammar.Add(“yes");
grammar.Add(“yeah");
grammar.Add(“ok");

47. KINECT FOR WINDOWS
MULTI-MODAL FEEDBACK
• Engagement model
– Mic indicator for speech-enabled menus
– Keyword to engage
• Feedback and confirmation

48. KINECT FOR WINDOWS
Audio Data

49. KINECT FOR WINDOWS
AUDIO
DEMO

50. KINECT FOR WINDOWS
The Future of Kinect
• Better close up

51. KINECT FOR WINDOWS
Gesture Libraries

52. KINECT FOR WINDOWS
Gesture Libraries

53. KINECT FOR WINDOWS
Kinect version of Siri (other CodePlex)

54. KINECT FOR WINDOWS

55. KINECT FOR WINDOWS
Resources
• Coding 4 Fun - Coding4Fun Kinect Toolkit
• OpenSource Projects
• SilverlightKinect

56. KINECT FOR WINDOWS
What we have learned
• Examples of the amazing things that can be
done with the Kinect
• How amazing the Kinect Device is, equivalent to
$200,000 in capabilities
• How to create our own Kinect applications,
Video, Skeletal and Audio tracking.
• The future of Kinect

57. KINECT FOR WINDOWS
Thank You
• David Silverlight
• David@TheSilverlightGroup.com