J Gabriel Lima - http://jgabriellima.in

1. CASSANDRA Framework: A Service Oriented Distributed Multimedia
Content Analysis Engine
Fons de Lange, Jan Nesvadba, Johan Lukkien
Philips Research Eindhoven, The Netherlands Eindhoven University of Technology
fons.de.lange@philips.com j.j.lukkien@tue.nl
Abstract consumer-driven personal content production stir
consumers into the dilemma of multimedia retrieval
Connected Consumer Electronics (CE) devices face and management, equivalent to the search of the needle
an exponential growth in storage, processing in the haystack.
capabilities and connectivity bandwidth. This As a matter of fact, petabytes of content distributed
increased complexity calls for new software across a network of memory devices, similar to the
architectures that naturally admit self-organization, incredible amount of memory scattered across the
resource management for efficient workload human brain, are only manageable under the condition
distribution, and a transparent cooperation of of (semantic) content- and to some extent also context
connected CE-terminals. The interconnected nature awareness. Hence, content-awareness creation is of
makes advanced applications feasible such as the utmost importance. Fortunately, the distributed, but
smart distribution of complex multimedia content connected processing and memory faculties of CE
analysis algorithms, resulting in automatic semantic networks provide sufficient computational resources to
content-awareness creation. In this paper we describe
perform the required multimedia content analysis and
a modular, self-aware, self-organizing, real-time and
to memorize the generated content-awareness-creating
distributed multimedia content analysis system. Based
on a Service Oriented Architecture it is, furthermore, content descriptors. It is, therefore, foreseeable, that
capable of efficiently and dynamically using the such networks will efficiently share functionality,
available resources in a grid-like manner. resources (memory, processing) and content such as to
support distributed analysis applications. Hence,
1. Introduction connected CE devices will attract consumers soon
through smartly interoperating with each other and
Petabytes of storage, a million MIPS (Million through their integration with e.g. internet services
Instructions Per Second) at an 1000-Euro-pricepoint providing intuitive, state-of-the-art capabilities and
and several gigabits-per-second Internet-Protocol- applications, respectively.
based (IP) connectivity are realistic technical
capabilities of next decades Consumer Electronics The underlying required domain-specific audio-,
(CE) networks [1]. But, next to the technology also music-, speech-, image- or video content analysis
consumers’ behaviors evolve. Today’s consumers, for (expert) systems, each exploiting one sensorial signal
example, gradually change from passive absorbers to in isolation, have been developed during the past
active and often internet-community-attached decade by various independent expert teams. The
multimedia content producers. Unfortunately, this majority, whereof, has been developed in isolation of
results in a massive production of individual content- the other expert systems and for one specific
unaware multimedia assets, made easy by ubiquitous application domain only, mainly because of capability
and pervasive content creation CE devices such as constraints. Unfortunately, human-communication-like
camera mobile phones, camcorders, video recorders content-awareness, enabling e.g. human-like search
and PC software. Hence, the enormous memory queries, has to be based on smart combinations of
resources scattered across distributed, but those orthogonal multimedia content analysis results as
interconnected CE devices combined with the massive done by our human brains automatically. In order to
acquisition of commercial content and the massive enable smart collaborations of those individual expert
Eight International Workshop on Image Analysis for Multimedia Interactive Services(WIAMIS'07)
0-7695-2818-X/07 $20.00 © 2007

2. systems, spanning from music genre analysis to easily integrate new pieces of software of unknown
Automatic Speech Recognition (ASR) and from face quality. Our software model supports this by
localization / identification to emotion analysis, they describing applications as the coordination of passive
have to be embedded in a flexible, extendable, scalable services running on devices in the network. In order to
and upgradeable system [2]. Flexibility, extendibility explain this we first look at more traditional ways to
and scalability are guaranteed through a rigid modular look at distributed applications.
approach, wherein each individual expert system (e.g. Figure 1 shows a simplified view of a running
an individual analysis component) is encapsulated into program, which we call a component. It has four
a functional unit, further called Service Unit (SU), with dependencies: the operating system platform (4), the
clearly defined and standardized Input / Output (I/O) user interface (3), services provided (2) and services
interfaces. The latter guarantee hassle-free obtained from the network (1).
upgradeability of individual SU expert systems through
Graphics
simple replacement of the involved analysis User Interface
algorithms, provided that metadata I/O formats do not
change. 3: User Interactions
2: Provided
The complex nature of such a system, i.e. a 1: Required
Services
Component
(Application)
Services
(to network)
distributed service-oriented analysis engine hosting a (from network)
multitude of disciplinary-orthogonal analysis 4: Platform Dependencies
algorithms, has to provide ease-of-use, robustness and
interoperability. Platform
Particularly, the integration of know-how provided (Operating System)
by various disciplinary-orthogonal expert teams and
the management of such a system should be time and Figure 1. Basic model of network component.
effort efficient.
Each component has dependencies (4); a typical
In this paper we describe a distributed multimedia client application has dependencies (1) and (3); a
content analysis prototyping framework, built typical server has dependency (2) and a stand-alone
according to service oriented principles [3], which
application has just (3). In our approach we use mainly
allows algorithm expert teams to efficiently and
two types of components. First, the ones with
transparently integrate their algorithms into the system,
but also to effortlessly upgrade (replace) them and, dependencies (1) and (2), i.e., components that provide
finally, to co-develop combined joint solutions. This services to the network while possibly using services
framework also serves as a demonstration of themselves. These services are made available on the
automatically cooperating devices. The next section network through interfaces comprising function calls,
elaborates the service concepts in more detail. eventing and connection points (called pins) for
streaming data. In our system, called the Cassandra
2. The service oriented architecture Framework, we call these components Service Units
(SU). Each SU can be run on a particular device, called
2.1 Global Structure a Node, while nodes may run several components and
component instances at the same time.
Our aim is to develop a software architecture that
facilitates the composition of applications from Given a set of available passive services there is the
distributed components. The platform of our question where the control lies. This control pertains to
applications consists of a network of cooperating establishing connections between services as well as
devices such as personal computers, consumer the control flow within a set of connected services. In
electronics equipments as well as mobile terminals a regular client-server application both types of control
such as telephones. Hence, our applications are usually lie at the client. The second type of component
naturally distributed over multiple devices and must be in the Cassandra Framework is therefore a coordinator
designed such that they can function in a changing engine. The task of the coordinator is to setup a
environment. The internal software of devices comes collection of services according to a given description
from different manufacturers and we must be able to and to connect them accordingly. This is done a.o.
through dynamic service discovery. In terms of Figure
Eight International Workshop on Image Analysis for Multimedia Interactive Services(WIAMIS'07)
0-7695-2818-X/07 $20.00 © 2007

3. 1 it uses just (1) although the services it accesses are This service-oriented organization of the system
not for its own use. Instead it instructs services to build introduces significant benefits.
connections between them coupling data pins as well First, by separating composition and configuration
as control interfaces. The control flow is then entirely entirely from the services themselves we postpone the
defined by this connectivity pattern and the behavior of decision how to use the services. In fact, by improving
the services. In many cases, a simple data flow is used our coordination software we improve the operation of
but more advanced behavior is possible as well. Note the system without having to change the services. This
that the coordinator is actually not a part of the concept is used in the robustness. The services have a
application; after setting up the network state it might separate interface supporting error monitoring and
as well disappear. In the framework we call the error reporting; a monitor in the network may use this
coordinator the use-case manager and a composition of interface to improve the robustness [5]. However, this
services a use-case. is not required and the system will function without it.
Second, the architecture makes it straightforward to
For the purpose of setting up a collection of services, take decisions based on resource information and
each Node runs an instance of the so-called Component performance indicators. In this way load-balancing
Repository service unit. The Component Repository is decisions are made upon startup of a use case. In
responsible for maintaining a cache of locally-available principle it is just as easy to do re-balancing when a
components and keeping them synchronized with a new use case needs to be instantiated.
Master Repository. This is done using UPnP Third, the integration of new functionality is just as
networking [4], where the Component Repository of easy as deploying it as a service and exposing it to the
each Node functions as a UPnP device, announcing it network. Although in practice a standard middleware
presence on the network and where the Master approach will be used to do this, it is not required. The
Component Repository Manager is a UPnP control system does not depend on any (OS-)platform or
point, capable discovering all UPnP devices such as the language. The only dependence is through the
Component Repository of each Node. A possible setup interfaces found on the network. Hence, the
of services and service units is given in Figure 2 standardization is on the format and semantics of the
below. exchanged messages. This leads to an approach of
node component node building new functionality by extension and re-
component composition rather than reworking existing software.
repository
service unit Fourth, the architecture supports the cooperation of
component
Use-case
Master
component
start/stop devices without this pre-installed knowledge of the
component component
repository
repository
manager
sync
cooperation context, providing application and data
browse get / change
service
unit (SU)
SU SU transparency. The applications in our system (the use
use-case configuration cases) are in fact scripts that coordinate the services.
start/stop component
node As such they could be executed anywhere in the
service X start/stop
Use-case
manager
component
repository component
system.
component service unit
SU SU SU
2.2 Communication styles and performance
Services can be used through interfaces. Two styles
Figure 2. Overall system architecture.
of interfaces are used: control-type interfaces
comprising function calls and eventing, and streaming
The Component Repository starts or stops a
type interfaces enabling only simple data pushing and
component upon a request of the use-case manager.
pulling. Compared to the simple data push-pull
After starting a component, the Component Repository
interface, the control-type interface is more expressive
service keeps track of its execution lifetime. To this
in terms of actions and data types, but has more
end, the Master Repository maintains an up-to-date list
performance overhead. In the Cassandra Framework,
of all nodes and their configuration, i.e. the available
the control-type communication between services is
components on these nodes and the services that they
based on UPnP, while the simple data push/pull
implement (Figure 2). Note that the configuration may
communication is based on TCP/IP. In both cases
change dynamically, when components are added or
communication is based on ports and IP addresses, but
removed to the system.
UPnP communication involves more complex protocol
Eight International Workshop on Image Analysis for Multimedia Interactive Services(WIAMIS'07)
0-7695-2818-X/07 $20.00 © 2007

4. handling and time consuming message interpretation, approach. The framework has been tested and shown
which is not done in case of basic TCP/IP streaming effective by integrating components on different
communication. operating systems and in different languages. The
As an example, Figure 3 shows a commercial block current system works well because of over-
detector consisting of several connected services. It dimensioning, particularly of the network. We want to
explicitly shows the connection entities that take care study an approach in which Quality of Service
of the network communication. In our system, these tradeoffs must be made. On the one hand this may lead
can be UPnP device protocol stacks, TCP/IP sockets or to re-balancing from the perspective of the use-case
a combination of both. Note that the figure does not manager.
As an example, Figure 4 shows the Graphics User
show the repository managers and nodes, since these
Interface of the Cassandra Use-Case Manager for an
are not relevant to the discussion. Components take
example network of Service Units (SUs), realizing a
care of setting up the UPnP device stack for control complex Content Analysis System. By means of this
type communication and TCP/IP sockets for data GUI, networks can be defined, instantiated, executed,
push/pull communication. controlled and monitored.
component
component GUI
component Client
audio
channel bit rate silence
select setting detect conn.
client client SU
connect connect connect
conn. conn. conn. conn. conn. conn.
AV time black frame
Tuner heuristic
Codec stamp frame repeat
SU decider
SU generator detect analyzer
SU
SU SU SU
comp. comp.
comp. comp. component
Figure 3. Commercial block detect use-case, Figure 4. Use Case Manager GUI, showing
showing network connection points. execution of complex network of SUs.
The real-time nature of the stream processing poses an 10. References
extra challenge for the software architecture. After a
use case has been setup, the data that flows between [1] PC technology trends,
connected services should not pass up and down a http://pcpitstop.com/research/default.asp
complicated software stack for interpretation. Instead, [2] CASSANDRA Project,
after setup the data should flow smoothly between http://www.research.philips.com/technologies/storage/cassan
functional processing units making up the services. dra
This can be hardware processing as well (e.g. an [3] Sayed Hashimi, Service Oriented Architecture
encoder or decoder). One way to look at this is that the Explained,
control part of the system is used to configure the http://www.ondotnet.com/pub/a/dotnet/2003/08/18/soa_expla
hardware plane perpendicular to it. The simple data ined.html
push/pull is the best choice for realizing such [4] UPnP framework, http://www.upnp.org .
communication. In Figure 3 the AV streams are best [5] C. Fetzer, M. Raynal, and F. Tronel. An adaptive failure
communicated via sockets, while the Channel select detection protocol. Proc. of the 8th IEEE Pacific Rim Symp.
and bit-rate-setting clients operate best via a (UPnP) on Dependable Computing. Seoul, Korea, 2001.
control-type interface. The GUI may use a mix of both.
3. Conclusions
We have presented the architecture of a robust
distributed media processing framework that supports
cooperation by composition through a Service Oriented
Eight International Workshop on Image Analysis for Multimedia Interactive Services(WIAMIS'07)
0-7695-2818-X/07 $20.00 © 2007