This document provides an overview of the Microsoft Kinect for Windows SDK and examples for accessing the Kinect sensor's camera, depth, skeletal tracking, and audio capabilities using the Kinect API in C#. It discusses initializing the Kinect sensor, retrieving synchronized color and depth frames, mapping skeletal joint data to coordinate spaces, and recording or processing audio data from the Kinect microphone array. The presentation aims to demonstrate basic usage of the Kinect for Windows SDK for computer vision and interaction applications.

1. Microsoft Kinect for Windows
Presented by Reuben Ahmed
rahmed@bennettadelson.com

2. Microsoft Kinect for Windows
o Disclaimer
o Smelly code may be shown
o Purpose
o Overview of Kinect SDK Feb. 2012 SDK
o WPF/C# related examples
o C++ not covered

3. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

4. o Microsoft Kinect for Windows

5. o Microsoft Kinect for Windows
o What’s the “Windows” Kinect?
o Has the Kinect Logo 
o Commercial License
o XBOX360 for DEV
o Cannot develop for XBOX360!
o Costs $250!
o Near-field optimized camera

6. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

7. Microsoft Kinect for Windows
o SDK
o http://www.microsoft.com/en-us/kinectforwindows/develop/
o Installs 4 programs

8. Microsoft Kinect for Windows
• Microphone Array
Manage Audio Devices
• Sensor Blinking Green
• “Did you plug it in?”

9. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

10. o Microsoft Kinect for Windows
Create WPF Kinect Project
• Reference Microsoft.Kinect
• Microsoft.Kinect.KinectSensor _sensor;
• Initialize Kinect
• Check Sensor Status
• .Stop() - Cleanup

11. o Microsoft Kinect for Windows
Let’s capture color, depth, and
skeleton information
• AllFramesReady event

12. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

13. Microsoft Kinect for Windows
o Resolutions
• Color Frames
• 12FPS (1280x960)
• 15FPS (RAW 640x480)
• 30FPS (640x480)
• Depth Frames
• 30FPS (80x60, 320x240,
640x480)

14. Microsoft Kinect for Windows
o AllFramesReady
o “Synchronized Data”
o Fires at slowest speed
o Sign up for individual events
o Display RGB (BGR32) Camera
o byte[] of pixel data

15. Microsoft Kinect for Windows
1 pixel = 4 bytes
For first pixel:
• byte[0] = Blue
• byte[1] = Green
• byte[2] = Red
• byte[3] = null

16. o Microsoft Kinect for Windows
Display RGB (BGR32) Camera
o ColorImageFrame frame =
e.OpenColorImageFrame()
o Check if frame was dropped!
Convert frame to BitmapSource
o Calculate “stride”
o Capture higher resolution

17. o Microsoft Kinect for Windows
Modify Kinect Tilt
o KinectSensor.ElevationAngle
o -27 to 27 degrees
o Motor burn-out!
o ElevationAngle is relative!

18. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

19. Microsoft Kinect for Windows
o DepthImageFrame
o Similar to ColorImageFrame, but data is
represented by short[]
o Contains distance and player data
o i.e. this pixel is 1000 mm. away and
represents player 1
o Default vs. Near mode
o Too near  normal  too far

20. Microsoft Kinect for Windows
How do we get data out of a short[]?
o short[0] represents first pixel
o short pixelDepth = short[0];
Bit-shifting
int depth = pixelDepth >> DepthImageFrame.PlayerIndexBitmaskWidth;
int player = pixelDepth & DepthImageFrame.PlayerIndexBitmask;

21. Microsoft Kinect for Windows
Visualize Depth Data as Image
o DepthImageFrame frame =
e.OpenDepthImageFrame()
Convert frame to BitmapSource
o Create color-map with 4 bytes
(BGR), based on distance
o Identify a player with a single
color

22. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

23. o Microsoft Kinect for Windows

24. o Microsoft Kinect for Windows
o Each Joint has an X, Y, and Z
coordinate
o Tracked, not tracked, inferred
o Inferred – low confidence, clipped
o TransformSmoothParameters

25. o Microsoft Kinect for Windows
o Maximum 2 skeletons are tracked
o 6 proposed players
o TrackingID identifies skeleton

26. o Microsoft Kinect for Windows
o SkeletonFrame sf = e.OpenSkeletonFrame()
o sf.CopySkeletonDataTo()
o Retrieve first tracked Skeleton
SkeletonPoint  DepthImagePoint  ColorImagePoint
o Get all skeletons (6)
o Plot skeleton point on Canvas

27. o Microsoft Kinect for Windows
o DepthImageFrame.MapFromSkeletonPoint()
o DepthImageFrame.MapToColorImagePoint()
o Canvas.SetTop() and Canvas.SetLeft()
o Plot skeleton point on Canvas (cont.)

28. o Microsoft Kinect for Windows
o TransformSmoothParameters
o new TransformSmoothParameters()
{Smoothing=0.3f, JitterRadius=1.0f,
MaxDeviationRadius=0.5f}

29. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

30. o Microsoft Kinect for Windows
o Sound Position
o Sound Source Angle
o Angle and confidence level of audio heard
o Beam Angle
o Directional microphone
o KinectAudioSource
o Recording

31. o Microsoft Kinect for Windows
o Speech Recognition
o Grammar/Choices
o Built via Declarative XML or C#

32. o Microsoft Kinect for Windows
Speech Recognition Engine
• Microsoft.Speech

33. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows

34. o Credits / Resources
o Channel 9 Kinect SDK Quickstart:
http://channel9.msdn.com/Series/KinectQuickstart
o Kinect Mock - Mocking recorded inputs
• http://channel9.msdn.com/coding4fun/kinect/
Kinect-I-Moc-you-with-MocKinect
o Kinect Services - Metro apps (Kinect data
exposed as a service via sockets, for viewing on
a remote machine) - example shows streaming
data to windows phone 7
o Microsoft Kinect for Windows