Kinect: It's All NUI To Me!

3,019 views

Published on

David Silverlight's powerpoint presentation on the Kinect for Windows SDK. Feb. 29, 2012

Published in: Technology, Education
3 Comments
8 Likes
Statistics
Notes
No Downloads
Views
Total views
3,019
On SlideShare
0
From Embeds
0
Number of Embeds
40
Actions
Shares
0
Downloads
0
Comments
3
Likes
8
Embeds 0
No embeds

No notes for slide

Kinect: It's All NUI To Me!

  1. 1. It’s All NUI to me:Building Apps with the Microsoft Kinect SDKDavid SilverlightFounder and Chief Software ArchitectTheSilverlightGroup.comIsYourWebMasterDead.com KINECT FOR WINDOWS
  2. 2. Agenda• NUI – What?• Demo’s – See what all the hullabaloo is about• Getting Started/Kinect Explorer/QuickStarts• Building Your Own Kinect apps. • (Video, Depth, Skeleton, Audio)• Resources KINECT FOR WINDOWS
  3. 3. Kinect: Setter of World Records! KINECT FOR WINDOWS
  4. 4. And Some Disasters KINECT FOR WINDOWS
  5. 5. It’s in the headlines KINECT FOR WINDOWS
  6. 6. What is NUI?• Natural User Interface• Common term used by designers and developers of computer interfaces to refer to a user interface that is effectively invisible, or becomes invisible with successive learned interactions, to its users KINECT FOR WINDOWS
  7. 7. What is NUI?• An Invisible Interface.• The Content is the Interface.• Removing the Proxy• Direct Manipulation• Gestural interfaces KINECT FOR WINDOWS
  8. 8. Seeing is believing – Kinect Development Real Time Motion Personal Data Mirror Capture with XNA12 Best Kinect HacksSample showing just Catch up with the Allows you to capturewhat people are latest headlines, your movements anddoing with the weather and health opens up a whole newKinect all whilst brushing world to animators on your teeth a tight budget. KINECT FOR WINDOWS
  9. 9. Seeing is believing – Kinect DevelopmentFitnect Virtual VacationInteractive Fitting Now another reasonroom showing how to never leave theKinect and business house.can connect. KINECT FOR WINDOWS
  10. 10. What can you do with Kinect on a PC or Mac ? KINECT FOR WINDOWS
  11. 11. 3D DEPTH SENSORS RGB CAMERAMULTI-ARRAY MIC MOTORIZED TILT KINECT FOR WINDOWS
  12. 12. Getting Started: Building Your Own Apps • Designed to get you started • Assumes you have some programming experience • Managed API - Concepts work in both VB/C# KINECT FOR WINDOWS
  13. 13. Kinect USB cable KINECT FOR WINDOWS
  14. 14. System Requirements• You must run applications that are built by using the Kinect for Windows SDK in a native Windows environment. You cannot run applications in a virtual machine, because the Microsoft Kinect drivers and this SDK must be installed on the computer that is running the application.• Supported Operating Systems and Architectures • Windows 7 (x86 or x64)• Hardware Requirements • Computer with a dual-core, 2.66-GHz or faster processor • Windows 7–compatible graphics card that supports Microsoft® DirectX® 9.0c capabilities • 2 GB of RAM • Kinect for Xbox 360® sensor—retail edition, which includes special USB/power cabling KINECT FOR WINDOWS
  15. 15. System Requirements• Software Requirements • Microsoft Visual Studio® 2010 Express or other Visual Studio 2010 edition • Microsoft .NET Framework 4.0 (installed with Visual Studio 2010)• For C++ SkeletalViewer samples: • DirectX Software Development Kit, June 2010 or later version • DirectX End-User Runtime Web Installer• For Speech sample (x86 only): • Microsoft Speech Platform - Server Runtime, version 10.2 (x86 edition) • Microsoft Speech Platform - Software Development Kit, version 10.2 (x86 edition) • Kinect for Windows Runtime Language Pack, version 0.9 (acoustic model from Microsoft Speech Platform for the Kinect for Windows SDK Beta) KINECT FOR WINDOWS
  16. 16. Developing your own Kinect apsIt all starts here: http://www.microsoft.com/en-us/kinectforwindows/ KINECT FOR WINDOWS
  17. 17. Kinect for Windows SDK Quickstarts• http://channel9.msdn.com/Series/KinectQuickstart - All are great, IMHO• The Kinect Explorer is new from the Beta • Kinect for Windows SDK Browser.• Installing and Using the Kinect Sensor• Setting up your development environment• Skeletal Tracking fundamentals• Camera Fundamentals• Working with Depth Data• Audio Fundamentals KINECT FOR WINDOWS
  18. 18. SETTING UPYOUR DEVELOPMENTENVIRONMENT KINECT FOR WINDOWS
  19. 19. Setting up your Development Environment Add our references to: - The Microsoft.Samples.Kinect.WpfViewers Project - The Coding4Fun.Wpf.dll Reference the assembly: • using Microsoft.Kinect; KINECT FOR WINDOWS
  20. 20. KINECT API BASICS• Manage Kinect state • Connected • Enable Color, Depth, Skeleton • Start Kinect• Get Data • Events - AllFramesReady • Polling – OpenNextFrame KINECT FOR WINDOWS
  21. 21. RESOURCES• Install Kinect Explorer • KinectWpfViewers• Coding4Fun Toolkit • Skeletal scaling KINECT FOR WINDOWS
  22. 22. SETTING UP YOUR DEV ENVIRONMENTDEMO KINECT FOR WINDOWS
  23. 23. CAMERAFUNDAMENTALSSetting theCamera Tilt Angle KINECT FOR WINDOWS
  24. 24. Setting the Camera Tilt Angle• Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function.• To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 (Camera.ElevationMinimum) and +27 (Camera.ElevationMaximum). The code below sets the camera to the maximum elevation:• kinectSensorChooser.Kinect.ElevationAngle = Convert.ToInt16(slider1.Value); KINECT FOR WINDOWS
  25. 25. CAMERAFUNDAMENTALS KINECT FOR WINDOWS
  26. 26. RESOLUTIONS• Color • 12 FPS: 1280X960 RGB • 15 FPS: Raw YUV 640x480 • 30 FPS: 640x480• Depth • 30 FPS: 80x60, 320x240, 640x480 KINECT FOR WINDOWS
  27. 27. Camera Data KINECT FOR WINDOWS
  28. 28. CAMERA DATA• Events return ImageFrame • PixelDataLength • FrameNumber • Timestamp • Dimensions: Height, Width• Use AllFramesReady event to synchronize KINECT FOR WINDOWS
  29. 29. CAMERA FUNDAMENTALSDEMO KINECT FOR WINDOWS
  30. 30. WORKING WITHDEPTH DATA KINECT FOR WINDOWS
  31. 31. DEPTH DATA• Returns the distance and player for every pixel • Ex: 320x240 = 76,800 pixels• Distance • Distance in mm from Kinect ex: 2,000mm (6.56 feet)• Player • 1-6 players KINECT FOR WINDOWS
  32. 32. MODEMode Depth & Player Center Hip Joint Other 19 JointsDefault Yes Yes Yes Near Yes Yes No, for v1.0 KINECT FOR WINDOWS
  33. 33. DISTANCES Feet 1.3’ 2.6’ 9.8’ 13.1’ 26.2’ Meters .4 .8 3 4 8Default Mode Near Mode KINECT FOR WINDOWS
  34. 34. FORMULAS• Distance Formula int depth = depthPoint >> DepthImageFrame.PlayerIndexBitmaskWidth;• Player Formula int player = depthPoint & DepthImageFrame.PlayerIndexBitmask; KINECT FOR WINDOWS
  35. 35. WORKING WITH DEPTH DATADEMO KINECT FOR WINDOWS
  36. 36. SKELETALTRACKINGFUNDAMENTALS KINECT FOR WINDOWS
  37. 37. Joints• Maximum two players tracked at once • Six player proposals• Each player with set of <x, y, z> joints in meters• Each joint has associated state • Tracked, Not tracked, or Inferred• Inferred - Occluded, clipped, or low confidence joints• Not Tracked - Rare, but your code must check for this state KINECT FOR WINDOWS
  38. 38. Skeleton Data KINECT FOR WINDOWS
  39. 39. Skeleton API KINECT FOR WINDOWS
  40. 40. SKELETAL TRACKINGDEMO KINECT FOR WINDOWS
  41. 41. AUDIOFUNDAMENTALS KINECT FOR WINDOWS
  42. 42. QUICKSTART SERIES• Installing and Using the Kinect Sensor• Setting up your Development Environment• Camera Fundamentals• Working with Depth Data• Skeletal Tracking Fundamentals• Audio Fundamentals KINECT FOR WINDOWS
  43. 43. SOUND POSITION• Sound Source Angle – the angle and confidence level of where audio is coming from• Beam Angle – The angle used to record audio that you can set as a ―directional microphone‖ -50 0 +50 KINECT FOR WINDOWS
  44. 44. RECORDING• KinectAudioSource audio buffer• Use Echo Cancellation to not record PC sound KINECT FOR WINDOWS
  45. 45. SPEECH RECOGNITION• Grammar – What we are listening for • Code – GrammarBuilder, Choices• Note: Ensure AutomaticGainControl = false KINECT FOR WINDOWS
  46. 46. GRAMMAR<!-- Confirmation_Yes._value: string ["Yes"]--><rule id="Confirmation_Yes" scope="public"> <example> yes </example> var grammar = new Choices(); <example> yes please </example> grammar.Add(“yes please"); <one-of> <item> yes </item> grammar.Add(“yes"); <item> yeah </item> grammar.Add(“yeah"); <item> yep </item> grammar.Add(“ok"); <item> ok </item> </one-of> <item repeat="0-1"> please </item> <tag> out._value = "Yes";</tag> KINECT FOR WINDOWS
  47. 47. MULTI-MODAL FEEDBACK• Engagement model – Mic indicator for speech-enabled menus – Keyword to engage• Feedback and confirmation KINECT FOR WINDOWS
  48. 48. Audio Data KINECT FOR WINDOWS
  49. 49. AUDIODEMO KINECT FOR WINDOWS
  50. 50. The Future of Kinect• Better close up KINECT FOR WINDOWS
  51. 51. Gesture Libraries KINECT FOR WINDOWS
  52. 52. Gesture Libraries KINECT FOR WINDOWS
  53. 53. Kinect version of Siri (other CodePlex) KINECT FOR WINDOWS
  54. 54. KINECT FOR WINDOWS
  55. 55. Resources• Coding 4 Fun - Coding4Fun Kinect Toolkit• OpenSource Projects • SilverlightKinect KINECT FOR WINDOWS
  56. 56. What we have learned• Examples of the amazing things that can be done with the Kinect• How amazing the Kinect Device is, equivalent to $200,000 in capabilities• How to create our own Kinect applications, Video, Skeletal and Audio tracking.• The future of Kinect KINECT FOR WINDOWS
  57. 57. Thank You• David Silverlight• David@TheSilverlightGroup.com KINECT FOR WINDOWS

×