Hacking the Kinect with GAFFTA Day 4


Published on

Last day of my GAFFTA workshop. Working with openNI and skeleton tracking

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Hacking the Kinect with GAFFTA Day 4

  1. 1. OPEN FRAMEWORKS + KINECT PART IV @GAFFTAThursday, March 14, 13
  2. 2. Skeleton tracking with the Kinect So far this class has used kinect in several ways: Open CV + Contour Finder for blobs Point Cloud to visualize the raw 3D dataThursday, March 14, 13
  3. 3. Skeleton tracking with the Kinect But one of the more exciting possibilities is user or skeleton tracking.Thursday, March 14, 13
  4. 4. How does it work ? Images are tagged and colored to represent portions of a persons body. A large data set ( very large , like millions of tagged images ) are fed into a large computing cluster The cluster then actively learns ( hence why it’s called machine learning ) and the algorithm can be applied to a live video stream to extract the appropriate data.Thursday, March 14, 13
  5. 5. How does it work ? This is why the user is always first removed or separated from the scene so the machine learning can ignore unimportant pixel data.Thursday, March 14, 13
  6. 6. How do I use Skeleton Tracking ? Skeleton tracking if often implemented by something called a “middleware” - a layer of software between the OS and your application to interface with your camera device ( kinect ) Luckily there are lots of middleware options to implement an application using skeleton tracking depending on your camera.Thursday, March 14, 13
  7. 7. Kinect for Windows SDK One of the more stable, publicly available middlewares. Works exclusively with both kinect models. Includes skeleton tracking, seated skeleton, facial mesh tracking, directional microphone access. But is Windows Only.Thursday, March 14, 13
  8. 8. SoftKinetic IISU ( interface is you ) Very robust and versatile platform. Works with 4 different camera manufacturers. Very robust toolbox and low level API in c#, c++, unity3d, flash. Also the most expensive middleware. Includes skeleton tracking, cursors, volumetric, close range interaction ( including fingers and grasping )Thursday, March 14, 13
  9. 9. Omek One of the lesser developed middlewares. Works with PMD and Panasonic devices. The “Grasp” close range SDK which is in beta looks very promising. Omek ‘long range’ is more geared towards gaming and 1-2 skeleton tracking.Thursday, March 14, 13
  10. 10. OpenNI OpenNI is maintained by prime sense, who developed the technology behind the first Kinect. OpenNI is a very big larger scale project for modular gesture reconition systems. The ones we will focus on are OpenNI + NITE Skeleton Tracking. OpenNI is open source, and is supported by all primesense devices, the kinect for XBOX, and even the Panasonic D-Imager. Includes access for finding cursors ( high detail controllers ) and for full skeleton tracking.Thursday, March 14, 13
  11. 11. Quick git setup ( OS X ) In Xcode make sure you have Command Line Tools installed Then head over to homebrew : http://mxcl.github.com/homebrew/ and use the terminal to install it ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)" then all you need is to add ‘brew install git’ and you’re done ! https://help.github.com/articles/set-up-git can help with the rest.Thursday, March 14, 13
  12. 12. Open Sound Control Networking Open Sound Control or OSC is a standard developed to send data from one application or device to another. It’s the new MIDI , but awesomer and wireless. Open Frameworks natively supports OSC with it’s core addon ‘ofxOsc’ One of the benefits of OSC over say : TCP or UDP is that OSC is broadcast and listener based.Your applications don’t need to first establish a handshake. If both are on the same internet and listening on the same port they can communicate to each other. OSC also natively wraps data types like strings, and floats. So you don’t have to parse raw bytes of data.Thursday, March 14, 13
  13. 13. Sending Skeleton Data over OSC Splits up computing power between apps. Easy to send basic data sets between applications.Thursday, March 14, 13