This presentations explains the foundations of Stream Processing and shows how elegant Stream Processing Architectures can be built by using in synergy DDS and CEP.

1. Angelo CORSARO, Ph.D.
Chief Technology Officer
OMG DDS Sig Co-Chair
PrismTech
angelo.corsaro@prismtech.com!

2. Defining "
Stream Processing

3. Stream Processing
[1/3]
¨ Stream Processing can be seen as an architectural style
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
for building systems that operate over continuous
(theoretically infinite) streams of data
¨ Stream Processing is often reified under one of its many
declinations, such as:
¨ Reactive Systems
¨ Signal Processing Systems
¨ Functional Stream Programming
¨ Data Flow Systems

4. Stream Processing
[2/3]
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ Stream Processing Architectures are very natural for
modeling systems needing to react to streams of
data produced by the external world, such as the
data produced by sensors, a camera or even the
data produced by the stock exchange.
¨ Stream Processing Systems usually operate in real-
time over streams and generate in turns other
streams of data providing information on what is
happening or suggesting actions to perform, such
as by stock X, raise alarm Y, or detected spatial
violation, etc.

5. Stream Processing
[3/3]
¨ Stream Processing Systems are
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
typically modeled as collection of
modules communicating via
typed data channels
Filter
¨ Modules usually play one of the Filter
following roles:
¨ Sources: Injecting data into the System
Filter
¨ Filters/Actors: Performing some
computation over sources Sink
¨ Sinks: Consuming the data produced Channel
by the system source

6. Implementing"
Stream Processing

7. Stream Processing Alternatives
¨ Different possibilities exist to
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
implement channels as well s
filters
Filter
¨ DDS is a very good fit for
Filter
channels due to its strong typing
as well as performance and Filter
scalability
¨ Filter can be implemented by DDS Reader
custom business logic or by DDS Topic
leveraging CEP DDS Writer
CEP | Custom

8. Stream Processing "
with DDS

9. Data Distribution in a
Nutshell

10. Data Distribution in a Nutshell
struct TempSensor {!
¨ Data distribution is about long Id; !
float temp;!
making application defined float hum;!
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
}!
data available where needed #pragma keylist TempSensor id!
and when needed, while
preserving and end-to-end DR
type contract DW
Id Temp Scale
¨ Data is a first class concept, it
can be Created, Read,
101 25 C DR
Updated, and eventually DW 202 78 F
303 299 K
Disposed (CRUD)
¨ The last value (or last N-values)
DR
of a Data is always available to DW Fully Distributed
applications DW: DataWriter
Global Data Space
DR: DataReader

11. DDS Topics
[1/2]
“org.opensplice.demo.TTempSensor”
A Topic defines the subject of
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨
Name
publications and subscriptions
¨ A Topic has associated a user
defined type and QoS
¨ The Topic name, type and Topic
QoS have a well defined role
in matching subscriptions
Type QoS
¨ Topics can be discovered or
locally defined struct TempSensor {!
long Id; ! DURABILITY,
float temp;! DEADLINE,
float hum;! PRIORITY,
}! …
#pragma keylist TempSensor id!

12. DDS Topics
[2/2]
“org.opensplice.demo.TTempSensor”
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Name
¨ DDS Topic types can have
associated keys
¨ Each unique key-value Topic
identify a Topic Instance –
a specific stream of values Type QoS
struct TempSensor {!
long Id; ! DURABILITY,
float temp;! DEADLINE,
float hum;! PRIORITY,
}! …
#pragma keylist TempSensor id!

13. Anatomy of a DDS Application
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Domain val dp = DomainParticipant(0) ! Domain
Participant
// Create a Publisher / Subscriber!
Publisher p = dp.create_publisher();!
Session Subscriber s = dp.create_subscriber();! Publisher Topic Subscriber
// Create a Topic! Gives access to a
Topic<TempSensor> t = !
dp.create_topic<TempSensor>(“com.myco.TSTopic”)!
DDS Domain
Reader/Writers
User Defined for Types
DataWrter DataReader
// Create a DataWriter/DataWriter!
DataWriter<TempSensor> dw = pub.create_datawriter(t);!
DataReader<TempSensor> dr = sub.create_datareader(t);!

14. Anatomy of a DDS Application
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Domain val dp = DomainParticipant(0) !
Domain
Participant
// Create a Publisher / Subscriber!
val pub = Publisher(dp)!
Session val sub = Subscriber(dp)! Publisher Topic Subscriber
// Create a Topic!
val topic = Topic[TempSensor](dp, !
“org.opensplice.demo.TTempSensor”)!
Reader/Writers
User Defined for Types Pub/Sub
DataWrter DataReader
Abstractions
// Create a DataWriter/DataWriter!
DataWriter<TempSensor> dw = pub.create_datawriter(t);!
DataReader<TempSensor> dr = sub.create_datareader(t);!

15. Anatomy of a DDS Application
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Domain val dp = DomainParticipant(0) !
Domain
Participant
// Create a Publisher / Subscriber!
val pub = Publisher(dp)!
Session val sub = Subscriber(dp)! Publisher Topic Subscriber
// Create a Topic!
val topic = Topic[TempSensor](dp, !
“org.opensplice.demo.TTempSensor”)!
Reader/Writers for User Defined for Types
// Create a DataWriter/DataWriter!
DataWrter DataReader
val writer = DataWriter[TempSensor](pub, topic)!
Reader/Writer for
val reader = DataReader[TempSensor](sub, topic) !
! application
// Write data! defined Topic
val t = new TempSensor ts(101, 25, 40)! Types
writer write ts;!

16. Domain & Partitions
Domain (e.g. Domain 123) Domain
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Participant
Topic
Partition (e.g. Partition “Telemetry”)
Publisher Subscriber
Topic Instances/Samples
DataWrter DataReader

17. Stream Processing in DDS
DDS naturally supports stream processing through three
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨
main concepts:
¨ Topics
¨ Data Writers
¨ Data Readers
¨ In addition DDS’ dynamic discovery, high availability and
high performance make it very easy to architect, deploy
and upgrade stream processing systems

18. DDS Topics as Streams
¨ In Stream Processing,
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Streams are typed data
channel
Filter
¨ A DDS Topic provide an Filter
elegant way of defining the
Streams that make up a Filter
Streaming System while
capturing their type, as well
as their Non-Functional DDS Topic
Constraints

19. DDS Data Writers as Sources
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ DDS Data Writers
make ideal sources for Filter
Stream Processing Filter
systems due to their: Filter
¨ Dynamic Discovery
¨ Durability
¨ Fault-Tolerance
DDS Writer

20. DDS Data Readers as Sinks
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ DDS Data Readers
make ideal sinks for Filter
Filter
Stream Processing
systems due to their: Filter
¨ Dynamic Discovery
DDS Reader
¨ Durability

21. DDS & Filters
¨ Filters/Actors perform
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
transformations from one or
more input streams to one or
more output streams Filter
Filter
¨ The combination of DDS
ContentFilteredTopics with a Filter
powerful language like Scala
can make it relatively easy to
build custom filters
¨ Let’s explore this path…
DDS+Application Logic

22. Content Filtered Topics
¨ Content Filtered Topics
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
provide a way of defining a Example:
local projection of the
// Create a Topic (on default domain)!
stream of data associated val topic = Topic[TempSensor](“TTempSensor”)!
val ftopic = !
with a given topic ContentFilteredTopic[TempSensor](“CFTempSensor”,!
topic,!
¨ Filters are expressed as the filter,!
params)!
“WHERE” clause of an SQL !
// - filter is a WHERE-like clause, such as: !
statement // “temp > 20 AND hum > 50”!
// “temp > %0”!
¨ Filters can operate on any //
//
“temp > hum”!
“temp BETWEEN (%0 AND %1)!
attribute of the type //!
// - params is the list of parameters to pass to the !
associated with the topic // filter expression – if any!

23. Filter Expression Syntax
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ DDS Filters are condition
over a topic type attributes
¨ Temporal properties or
causality cannot be
captured via DDS filter
expression

24. History
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Data older than “n samples ago”
get’s out the window
¨ DDS provides a way of
controlling data future past
windows through the
History QoS now
The window keeps
the last n data samples

25. [Putting it All Together]"
TempSensor Moving Average
object MovingAverageFilter {!
def main(args: Array[String]) {!
if (args.length < 2) {!
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
println("USAGE:ntMovingAverageFilter <window> <filter-expression>")!
}!
!
val topic = Topic[TempSensor]("TTempSensor")!
val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1))!
!
val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt)!
val reader = DataReader[TempSensor](ftopic, rqos)!
!
reader.reactions += {!
case e: DataAvailable[TempSensor] => {!
var average: Float = 0!
val window = e.reader.history!
window foreach (average += _.temp)!
average = average / window.length!
println("+--------------------------------------------------------")!
println("Moving Average: " + average)!
}!
}!
}!

26. Key Points So Far
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ DDS provides some event processing capabilities that
facilitate the development of Stream Processing Filters
¨ Higher Level programming languages like Scala can make it
very easy to concisely express complex filters
¨ Note: Scala targets the JVM as well as the .Net CLR

27. Stream Processing "
with DDS & CEP

28. CEP In Brief
[1/2]
¨ Complex Event Processing (CEP)
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
engines provide a declarative
way of transforming a set of input
streams into one or more output
streams CEP
¨ The declarative language - select *
from TempSensor(temp < 30)
provided by CEP is usually based
on some extension of SQL to - select avg(temp)
include time, pattern matching from TempSensor.win:time(1 sec)
and causality

29. CEP In Brief
[1/2]
Filter
CEP emerged from Stream Filter
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨
Processing as a way to
Filter
facilitate the development of
“Filters/Actors” and make it
“declarative”
CEP
¨ In line with the Stream
- select *
Processing principles, CEP from TempSensor(temp < 30)
operate on typed data streams - select avg(temp)
from TempSensor.win:time(1 sec)

30. DDS + CEP = Stream Processing
¨ DDS provides a powerful Stream
abstraction that is:
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ Type Safe
¨ High Performance
¨ Highly Available CEP
¨ Decoupled in Time/Space CEP
¨ Dynamically Discoverable
¨ CEP provide a powerful abstraction for CEP
processing streams
¨ The combination of DDS + CEP is not DDS Reader
only natural but a perfect fit for building DDS Topic
high performance, highly available DDS Writer
Stream Processing Systems

31. DDS + CEP in Action

32. Esper CEP
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ Esper is an Open Source, pure Java, CEP engine widely used
in a wide variety of applications ranging from the Capital
Market to Defense and Aerospace
¨ http://www.espertech.com
¨ Esper Rules are expressed EPL (Esper Processing
Language) which can be seen as an extension of SQL
with support for time, causality and pattern matching

33. OpenSplice + Esper
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ Esper Provides an EsperIO framework for plugging-in
new stream transports
¨ Plugging OpenSplice into Esper is trivial even w/o
relying on the EsperIO framework
¨ Let’s have a look…

34. Demo Application

35. iShapes Application
Spotted shapes represent subscriptions
¨ To explore play with OpenSplice Pierced shapes represent publications
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
and Esper, we’ll use the simd-cxx
ishapes application
¨ Three Topics
¨ Circle, Square, Triangle
¨ One Type:
struct ShapeType {!
string color;!
long x;!
long y;!
long shapesize;!
};!
#pragma keylist Shapetype color!

36. Esper Setup
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Step 1: Register Topic Types
val config = new Configuration!
val ddsConf = new ConfigurationEventTypeLegacy!
!
ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)!
!
config.addEventType("ShapeType", !
classOf[org.opensplice.demo.ShapeType].getName, !
ddsConf)!
!
val cep: EPServiceProvider = !
EPServiceProviderManager.getDefaultProvider(config)!

37. Esper Setup
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Step 2: Register a Listener for receiving Esper Events
val listener = new UpdateListener {!
def update(ne: Array[EventBean], oe: Array[EventBean]) {!
ne foreach(e => {!
"// Handle the event!
})!
}!
}!

38. Esper Setup
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
Step 3: Hook-up DDS to Esper
reader.reactions += {!
case e: DataAvailable[ShapeType] => {!
(e.reader read) foreach(runtime sendEvent _)!
!
}!
}!

39. iShapes FrameRate
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ Let’s suppose that we wanted to keep under
control the iShapes Frame rate for ech given color
¨ In Esper this can be achieved with the following
expression:
insert into ShapesxSec !
select color, count(*) as cnt !
from ShapeType.win:time_batch(1 second) !
group by color!

40. iShapes Center of Mass
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ Suppose that we wanted to compute the center of
mass of all the shapes currently displayed over the
last second
¨ The Esper expression for this would be:
select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg
(y),int), shapesize) as NewShape !
from ShapeType.win:time(10 sec)!

41. References
OpenSplice DDS
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¥ #1 OMG DDS Implementation ¥ #1 Java-Based CEP Engine
¥ Open Source ¥ Open Source
¥ www.opensplice.org ¥ www.espertech.com
¥ Fastest growing JVM Language ¥ Scala API for OpenSplice DDS
¥ Open Source ¥ Open Source
¥ www.scala-lang.org ¥ code.google.com/p/escalier

42. Summing Up

43. Concluding Remarks
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
¨ DDS provides a very good
foundation to build high
performance, highly available and
scalable streaming systems
¨ DDS + CEP are the most natural,
effective and efficient way of
building next generation Stream
Processing Systems!

44. OpenSplice DDS
Delivering Performance, Openness, and Freedom
http://www.opensplice.com/
http://www.opensplice.org/ http://www.slideshare.net/angelo.corsaro
emailto:opensplicedds@prismtech.com
Copyright
2010,
PrismTech
–
All
Rights
Reserved.
http://bit.ly/1Sreg
http://twitter.com/acorsaro/
http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com