Kinect on Android Pandaboard

ARDIC 2010ARDIC 2010
KURUMSAL SUNUMKURUMSAL SUNUM
KINECT ON ANDROIDKINECT ON ANDROID
PANDABOARDPANDABOARD
ÜMİT UZUNÜMİT UZUN
Senior Software Engineer
1ARDIC A.S CONFIDENTIAL

Agenda
2© ARDIC, Inc. 2013
• Introduce most known Kinect sensor device
• Introduce PrimeSense technology behind
Kinect
• Introduce OpenNI library and architecture
leaded by PrimeSense
• Introduce NITE middleware library developed
by PrimeSense
• Show Demo

Kinect Sensor

Kinect Sensor
The Kinect Sensor
was originally
intended to be a
motion sensing input
device for the
Xbox 360,
allowing the user to
control games via
gestures and spoken
commands.

Kinect Teardown

Kinect IR Depth Sensor
• The IR light source is a laser that projects
an irregular pattern of dots.
• The monochrome CMOS IR sensor
measures the dots' “time of flight” after
they reflect off objects in the scene.
• In the same way as sonar, knowing how
long the light takes to return, the Kinect can
calculate how far away an object is located.
• IR is used to reduce the problem of ambient
light affecting the calculations.

IR Projected Dots

Kinect Development Software
There are four main Kinect development libraries:
• OpenKinect's libfreenect (
http://openkinect.org/wiki/Main_Page)
• CLNUI (http://codelaboratories.com/nui/)
• OpenNI (http://www.openni.org/)
• Microsoft's Kinect for Windows SDK (
http://www.microsoft.com/en-us/kinectforwindow
)

Kinect Development Software
• OpenKinect's libfreenect is derived from a reverse-
engineered/hacked Kinect driver, which works across
Windows, Linux and MacOS X.
• CLNUI (http://codelaboratories.com/nui/) is aimed at
Windows only, but allows multiple Kinects to work
together.
• It's really not fair to compare libfreenect or CLNUI with
OpenNI, which is intended to act as an interface
between a variety of sensor drivers (including a Kinect
driver) and middleware for manipulating the sensor
data in a higher-level manner (e.g. for recognizing
user gestures as commands).

PrimeSense - Technology
• PrimeSense 3D sensing technology
gives digital devices the ability to
observe a scene in three
dimensions.
• It translates these observations
into a synchronized image stream
(depth and color) – just like
humans do.
• It then takes those synchronized
images and translates them into
information.
• We call this depth sensing, which is
made possible through the cutting-
edge technology embedded in
primesense sensors and
middleware.

PrimeSense Reference Design

PrimeSense - Sensors
• PrimeSense 3D Sensors give
digital devices the gift of sight,
enabling Natural Interaction
between people and devices
and between devices and their
surroundings.
• Using cutting-edge technology,
sensors are robust, accurate
and affordable.
• They are used in mass-market
commercial devices. With more
options than ever – mid-range,
close range, a peripheral and an
embedded sensor.

PrimeSense - NITE
• PrimeSense 3D sensing technology is a
layered solution, combining both
hardware and software.
• The PrimeSense NiTE™ perception
algorithms layer is the most robust
and advanced 3D middleware
available in the market.
• It provides the application with a clear
user-control API, whether it is hand-
based control or a full-body control.
• The algorithms utilize the depth, color,
IR and audio information received from
the hardware device, which enable
them to perform functions such as
hand locating and tracking; user
segmentation; user skeleton joint
tracking; and more.

PrimeSense – Ecosystem

PrimeSense - Capri

PrimeSense – Capri
• Take the 3D sensor inside the Microsoft
Kinect, shrink it down to a tenth of its original
size and add a bunch of mobile capabilities,
and you have yourself PrimeSense’s latest
conquest, better known as Capri.
• The company, which is the brains behind the
Kinect, has been openly working on bringing
a tiny-yet-advanced 3D experience to tablets,
televisions and smartphones for quite some
time now.

OpenNI Layers

OpenNI Kinect Driver
• OpenNI's Kinect driver is called
SensorKinect, and is derived from open
sourced (LGPL) code from PrimeSense,
the company behind Kinect's depth
sensor technology.
• One advantage of libfreenect is API
support for controlling the Kinect's tilt
motor, status light, and accelerometer,
which is missing from the SensorKinect
driver.

KinectSDK vs OpenNI
• The main area where the SDK wins over
OpenNI is audio. Although OpenNI contains
audio-related APIs, not fully implemented.
• Other pluses for Microsoft's SDK are its
extensive documentation and examples,
and its ease of installation on Windows 7.
• The main drawback of the SDK is that it
only works on Windows 7, not XP or any
non-windows platform.
• The KinectSDK is free, but limited to non-
commercial purposes.

NITE – Natural Interaction
Middleware
• An important reason for using OpenNI is
its support for middleware.
• The NITE library understands different
hand movements as gesture types
based on how 'hand points' change over
time.
• NITE gestures include pushing, swiping,
holding steady (i.e. no hand movement),
waving, and hand circling.

NITE – SkeletonTracker

OpenNI SDK – Production Nodes

OpenNI SDK – Production Nodes
• The main functional element in OpenNI is
the production node.
• A node may represent a sensor's
functionality (e.g. depth detection,
infrared, audio) or a higher-level
abstraction (e.g. user positioning, hand
tracking).
• An OpenNI application always starts by
initializing the nodes it needs for its
particular task.

OpenNI SDK – Generators

The MapGenerator subclasses typically output
streams of images which are called maps in OpenNI.
The sensor-related Generator subclasses are:
• AudioGenerator: for producing an audio stream.
• DepthGenerator: for creating a stream of depth
maps.
• ImageGenerator: for creating colored image maps.
• IRGenerator: for creating infrared image maps.

The middleware-related classes are:
• GesturesGenerator: for recognizing hand gestures,
such as waving and swiping.
• HandsGenerator: for hand detection and tracking;
• SceneAnalyzer: for separating the foreground from
the background in a scene, labeling the figures,
and detecting the floor. The main output is a
stream of labeled depth maps.
• UserGenerator: generates a representation of a
(full or partial) body in the scene.

OpenNI SDK - Data and Metadata

OpenNI SDK - Data and Metadata
• The map generators (the depth,
image, IR and scene generators) make
their data available as subclasses of
Map and MapMetaData.
• Metadata are data properties (as
opposed to the data itself), such as
map resolution.

OpenNI SDK - Listeners

OpenNI SDK - Listeners
• Generators can also interact with user
programs via listeners (called
callbacks in OpenNI).
• The MapGenerators tend to use
StateChangedObservable listeners,
while other generators (e.g. for hands
and gesture recognition) use
Observable<EventArgs> listeners,
with different EventArgs subclasses.

OpenNI SDK - Capabilities

OpenNI SDK - Capabilities
• OpenNI is intended to work with
different sensors, so the capabilities
of an instantiated production node
may vary. For instance, some
webcams may allow an image map to
be cropped.
• Each capability is represented by a
subclass of CapabilityBase.

NiSimpleViewer

NiUserTracker

Kinect on Android Pandaboard

More Related Content

What's hot

Similar to Kinect on Android Pandaboard

Recently uploaded

Kinect on Android Pandaboard