This document discusses using skeleton tracking with the Kinect. It explains that skeleton tracking works by using machine learning on a large dataset of tagged images to extract body pose data from a live video stream. It then discusses several middleware options for implementing skeleton tracking applications with cameras like the Kinect, including the Kinect for Windows SDK, SoftKinetic IISU, and OpenNI. It also briefly introduces using Open Sound Control for networking between applications using skeleton tracking data.

1. OPEN FRAMEWORKS +
KINECT PART IV
@GAFFTA
Thursday, March 14, 13

2. Skeleton tracking with the Kinect
So far this class has used kinect in several ways:
Open CV + Contour Finder for blobs
Point Cloud to visualize the raw 3D data
Thursday, March 14, 13

3. Skeleton tracking with the Kinect
But one of the more exciting possibilities is user or skeleton
tracking.
Thursday, March 14, 13

4. How does it work ?
Images are tagged and colored to represent portions of a
persons body.
A large data set ( very large , like millions of tagged images )
are fed into a large computing cluster
The cluster then actively learns ( hence why it’s called
machine learning ) and the algorithm can be applied to a live
video stream to extract the appropriate data.
Thursday, March 14, 13

5. How does it work ?
This is why the user is always first removed or separated
from the scene so the machine learning can ignore
unimportant pixel data.
Thursday, March 14, 13

6. How do I use Skeleton Tracking ?
Skeleton tracking if often implemented by something called a
“middleware” - a layer of software between the OS and
your application to interface with your camera device
( kinect )
Luckily there are lots of middleware options to implement
an application using skeleton tracking depending on your
camera.
Thursday, March 14, 13

7. Kinect for Windows SDK
One of the more stable, publicly available middlewares.
Works exclusively with both kinect models.
Includes skeleton tracking, seated skeleton, facial mesh
tracking, directional microphone access. But is Windows
Only.
Thursday, March 14, 13

8. SoftKinetic IISU ( interface is you )
Very robust and versatile platform. Works with 4 different
camera manufacturers. Very robust toolbox and low level API
in c#, c++, unity3d, flash. Also the most expensive middleware.
Includes skeleton tracking, cursors, volumetric, close range
interaction ( including fingers and grasping )
Thursday, March 14, 13

9. Omek
One of the lesser developed middlewares. Works with PMD
and Panasonic devices. The “Grasp” close range SDK which
is in beta looks very promising.
Omek ‘long range’ is more geared towards gaming and 1-2
skeleton tracking.
Thursday, March 14, 13

10. OpenNI
OpenNI is maintained by prime sense, who developed the technology
behind the first Kinect. OpenNI is a very big larger scale project for
modular gesture reconition systems. The ones we will focus on are
OpenNI + NITE Skeleton Tracking. OpenNI is open source, and is
supported by all primesense devices, the kinect for XBOX, and even
the Panasonic D-Imager.
Includes access for finding cursors ( high detail controllers ) and for
full skeleton tracking.
Thursday, March 14, 13

11. Quick git setup ( OS X )
In Xcode make sure you have Command Line Tools installed
Then head over to homebrew : http://mxcl.github.com/homebrew/
and use the terminal to install it
ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"
then all you need is to add ‘brew install git’ and you’re done !
https://help.github.com/articles/set-up-git can help with the rest.
Thursday, March 14, 13

12. Open Sound Control Networking
Open Sound Control or OSC is a standard developed to send
data from one application or device to another. It’s the new MIDI ,
but awesomer and wireless.
Open Frameworks natively supports OSC with it’s core addon
‘ofxOsc’
One of the benefits of OSC over say : TCP or UDP is that OSC is
broadcast and listener based.Your applications don’t need to first
establish a handshake. If both are on the same internet and
listening on the same port they can communicate to each other.
OSC also natively wraps data types like strings, and floats. So you
don’t have to parse raw bytes of data.
Thursday, March 14, 13

13. Sending Skeleton Data over OSC
Splits up computing power between apps. Easy to send basic data
sets between applications.
Thursday, March 14, 13