2. Microsoft Kinect for Windows
o Disclaimer
o Smelly code may be shown
o Purpose
o Overview of Kinect SDK Feb. 2012 SDK
o WPF/C# related examples
o C++ not covered
3. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
5. o Microsoft Kinect for Windows
o What’s the “Windows” Kinect?
o Has the Kinect Logo
o Commercial License
o XBOX360 for DEV
o Cannot develop for XBOX360!
o Costs $250!
o Near-field optimized camera
6. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
7. Microsoft Kinect for Windows
o SDK
o http://www.microsoft.com/en-us/kinectforwindows/develop/
o Installs 4 programs
8. Microsoft Kinect for Windows
• Microphone Array
Manage Audio Devices
• Sensor Blinking Green
• “Did you plug it in?”
9. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
10. o Microsoft Kinect for Windows
Create WPF Kinect Project
• Reference Microsoft.Kinect
• Microsoft.Kinect.KinectSensor _sensor;
• Initialize Kinect
• Check Sensor Status
• .Stop() - Cleanup
11. o Microsoft Kinect for Windows
Let’s capture color, depth, and
skeleton information
• AllFramesReady event
12. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
13. Microsoft Kinect for Windows
o Resolutions
• Color Frames
• 12FPS (1280x960)
• 15FPS (RAW 640x480)
• 30FPS (640x480)
• Depth Frames
• 30FPS (80x60, 320x240,
640x480)
14. Microsoft Kinect for Windows
o AllFramesReady
o “Synchronized Data”
o Fires at slowest speed
o Sign up for individual events
o Display RGB (BGR32) Camera
o byte[] of pixel data
15. Microsoft Kinect for Windows
1 pixel = 4 bytes
For first pixel:
• byte[0] = Blue
• byte[1] = Green
• byte[2] = Red
• byte[3] = null
16. o Microsoft Kinect for Windows
Display RGB (BGR32) Camera
o ColorImageFrame frame =
e.OpenColorImageFrame()
o Check if frame was dropped!
Convert frame to BitmapSource
o Calculate “stride”
o Capture higher resolution
17. o Microsoft Kinect for Windows
Modify Kinect Tilt
o KinectSensor.ElevationAngle
o -27 to 27 degrees
o Motor burn-out!
o ElevationAngle is relative!
18. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
19. Microsoft Kinect for Windows
o DepthImageFrame
o Similar to ColorImageFrame, but data is
represented by short[]
o Contains distance and player data
o i.e. this pixel is 1000 mm. away and
represents player 1
o Default vs. Near mode
o Too near normal too far
20. Microsoft Kinect for Windows
How do we get data out of a short[]?
o short[0] represents first pixel
o short pixelDepth = short[0];
Bit-shifting
int depth = pixelDepth >> DepthImageFrame.PlayerIndexBitmaskWidth;
int player = pixelDepth & DepthImageFrame.PlayerIndexBitmask;
21. Microsoft Kinect for Windows
Visualize Depth Data as Image
o DepthImageFrame frame =
e.OpenDepthImageFrame()
Convert frame to BitmapSource
o Create color-map with 4 bytes
(BGR), based on distance
o Identify a player with a single
color
22. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
24. o Microsoft Kinect for Windows
o Each Joint has an X, Y, and Z
coordinate
o Tracked, not tracked, inferred
o Inferred – low confidence, clipped
o TransformSmoothParameters
25. o Microsoft Kinect for Windows
o Maximum 2 skeletons are tracked
o 6 proposed players
o TrackingID identifies skeleton
26. o Microsoft Kinect for Windows
o SkeletonFrame sf = e.OpenSkeletonFrame()
o sf.CopySkeletonDataTo()
o Retrieve first tracked Skeleton
SkeletonPoint DepthImagePoint ColorImagePoint
o Get all skeletons (6)
o Plot skeleton point on Canvas
27. o Microsoft Kinect for Windows
o DepthImageFrame.MapFromSkeletonPoint()
o DepthImageFrame.MapToColorImagePoint()
o Canvas.SetTop() and Canvas.SetLeft()
o Plot skeleton point on Canvas (cont.)
28. o Microsoft Kinect for Windows
o TransformSmoothParameters
o new TransformSmoothParameters()
{Smoothing=0.3f, JitterRadius=1.0f,
MaxDeviationRadius=0.5f}
29. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
30. o Microsoft Kinect for Windows
o Sound Position
o Sound Source Angle
o Angle and confidence level of audio heard
o Beam Angle
o Directional microphone
o KinectAudioSource
o Recording
31. o Microsoft Kinect for Windows
o Speech Recognition
o Grammar/Choices
o Built via Declarative XML or C#
32. o Microsoft Kinect for Windows
Speech Recognition Engine
• Microsoft.Speech
33. o Hardware Overview
o Installation
o API basics
o Camera
o Depth
o Skeletals
o Audio
o Q&A
Microsoft Kinect for Windows
34. o Credits / Resources
o Channel 9 Kinect SDK Quickstart:
http://channel9.msdn.com/Series/KinectQuickstart
o Kinect Mock - Mocking recorded inputs
• http://channel9.msdn.com/coding4fun/kinect/
Kinect-I-Moc-you-with-MocKinect
o Kinect Services - Metro apps (Kinect data
exposed as a service via sockets, for viewing on
a remote machine) - example shows streaming
data to windows phone 7
o Microsoft Kinect for Windows
Editor's Notes
Some versions of the Kinect do not come with a power cable Kinect sensor lights up, “NotPowered” status
Read e.OpenColorImageFrame() in all frames Ready event
Only work if frame isn't null, you can lose frames based on calculations (e.g. you may sometimes be receiving less than 30FPS)
Smelly code - creating bitmap every frame, should use a writable bitmap.
Read e.OpenColorImageFrame() in all frames Ready event
Only work if frame isn't null, you can lose frames based on calculations (e.g. you may sometimes be receiving less than 30FPS)
Smelly code - creating bitmap every frame, should use a writable bitmap.
Cannot modify elevation angle frequently.
ElevationAngle is relative!
Talk about what’s in the short[]
Mention how byte[] is constructed again for pixels
int depth = depthData[depthIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth;
int player = depthData[depthIndex] & DepthImageFrame.PlayerIndexBitmask;
Have code sample on side, too hard to remember
TransformSmoothParameters - interpolate in between motion captures to smoothen rendering, but introduces lag
TrackingID not always consistent if player leaves screen and comes back?
Skeleton[] _skeletons = new Skeleton[6];
Skeleton firstSkeleton = (from s in _skeletons
where s.TrackingState == SkeletonTrackingState.Tracked
select s).FirstOrDefault();
Show inferred point off screen, or track joint off screen
Skeleton[] _skeletons = new Skeleton[6];
Show inferred point off screen, or track joint off screen
