This document discusses the development of Concerto Con Moto, an open-source digital signage system that adds interactivity to Concerto through gesture detection and recognition using the Kinect. The developer has enhanced the DepthJS library to support Linux, created a framework using Chromium and a packaged Chrome application, and integrated it with the existing Concerto system. Future plans include configurable options for the Concerto Player and integrating gesture recognition with Concerto's content display.

1. Kenley
Cheung
Rensselaer
Center
for
Open
Source
So6ware

2. Introduc:on
• Concerto
is
an
open
source
digital
signage
system
used
here
at
RPI.
• Concerto
has
operates
on
a
client-­‐server
model.
– Clients
(Concerto
Player
instances)
download
content
from
the
content
server
and
display
them.
– Concerto
Player
currently
displays
content
in
a
carousel-­‐like
fashion.
– No
interac:vity
with
the
viewer.

3. Introduc:on
• Concerto
Con
Moto
makes
digital
signage
more
interac:ve
with
mo:on!
– Con
moto
means
“with
mo:on”
– Keeping
with
the
use
music
terminolog,
it’s
an
appropriate
name
for
a
product.
• Con
Moto
u:lizes
the
Xbox
Kinect
to
deliver
gesture
detec:on
and
recogni:on.
– U:lizes
three
open-­‐source
libraries
• Libfreenect
–
open-­‐source,
cross-­‐plaRorm
libusb-­‐based
driver
for
the
Kinect.
• OpenCV
–
open-­‐source
computer
vision
library
• DepthJS
–
open-­‐source
Kinect-­‐JavaScript
bridge
with
u:lizing
a
built-­‐in
gesture
detec:on
engine

4. What’s
been
done
so
far?

5. What’s
been
done
so
far?
• Numerous
DepthJS
enhancements
and
fixes
– Research
project
at
MIT
Media
Lab.
– Upstream
code
didn’t
work.
Needed
a
lot
of
tender
loving
care
to
get
it
working!
– Theore:cal
support
for
Linux.
• “It
should
work,
but
we
haven’t
tested
it
in
a
while.”
– Needed
several
modifica:ons
in
order
to
work
• Created
Linux
build
script
• Rewrote
parts
of
the
code
to
work
against
a
new
libfreenect
(Kinect
driver)
API
(fun!)
– Received
assistance
from
the
#OpenKinect
channel
on
FreeNode.
• Learned
how
to
build
shared
libraries
on
Linux
(even
more
fun!)
– Learned
how
the
compiler
works
to
produce
shared
libraries.
– Sta:cally
linked
libfreenect
and
OpenCV
-­‐
allows
us
to
distribute
DepthJS
binaries
with
no
addi:onal
dependencies!
• Modified
Chrome
extension
to
detect
and
load
the
appropriate
shared
library
for
a
supported
plaRorm.
• Fixed
bug
where
hand
gesture
events
were
not
being
triggering
JavaScript
events.

6. What’s
been
done
so
far?
• Framework
for
Concerto
Con
Moto
– LiveUSB
+
Chromium
+
Packaged
Chrome
Applica:on
– Concerto
Player
replacement
• No
longer
requires
internet
connec:on
to
ini:alize.
• Classic
exponen:al
back-­‐off
when
network
disrup:ons
occur.
• Skeleton
code
for
con
moto
overlay
checked
in.
• UI
mockup
of
Concerto
con
moto
overlay

7. What’s
in
store…
• Concerto
Player
configura:on
– Powered
by
HTML5
localStorage
– Set
Concerto
server
endpoint
• Not
all
Concerto
users
are
at
RPI
• One-­‐size-­‐fits-­‐all
Concerto
Player
–
No
need
to
generate
new
disk
images
– Choice
between
overlay
or
simple
control
of
screen
content
• Integra:on
with
exis:ng
Concerto
screen.js
and
Concerto
client
infrastructure

8. What’s
in
store…
• Demo
:me!