Getting Started with OpenSplice and Esper

Getting StartedwithOpenSplice and Esper Angelo Corsaro, Ph.D. Thomas Bernhardt Chief Technology Ofﬁcer Chief Technology Ofﬁcer / Funder OMG DDS Sig Co-Chair EsperTech PrismTech tom@espertech.com angelo.corsaro@prismtech.com

Agenda Welcome & Intro !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;!! DDS Overview! CEP Overview! DDS + CEP in Action! Q&A

DDS Overview...from a Stream Processing PerspectiveAngelo CORSARO, Ph.D.Chief Technology OfﬁcerOMG DDS Sig Co-ChairPrismTechangelo.corsaro@prismtech.com

Stream Processing [1/3]! Stream Processing is an architectural style for building !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; systems that operate over continuous (theoretically infinite) streams of data! Stream Processing is often reified under one of its many declinations, such as: ! Reactive Systems ! Signal Processing Systems ! Functional Stream Programming ! Data Flow Systems

Stream Processing [2/3]! The Stream Processing Architecture very !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; naturally models systems reacting to streams of data and events produced by the external world, such as the data produced by sensors, a camera or even the data produced by the stock exchange.! Stream Processing Systems usually operate in real-time over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc.

Stream Processing [3/3]! Stream Processing Systems are typically !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; modeled as collection of modules communicating via typed data channels called usually streams Filter! Modules usually play one of the Filter following roles: Filter ! Sources: Injecting data into the System ! Filters/Actors: Performing some computation over sources Sink ! Sinks: Consuming the data produced by Stream the system source

More on Streams

Deﬁning Streams! In abstract terms, a stream is an infinite sequence of !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; data samples of a given type T! Streams can be further classified in continuous and discrete streams. Sometimes referred as Behaviors/ Signals and Events! In this presentation we’ll refer to Continuous Streams as Data Streams and to Discrete Streams as Event Streams

Data StreamsData Streams Temp !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! The value of a Data Stream is always defined, i.e. continuous.! Good examples of a Data Stream are the value assumed by a real-world entity, such as temperature, time pressure, a price, etc.

Event StreamsEvent Streams OverheatAlarm !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! The value of the stream is defined at precise point in time, i.e. it is discrete! Good examples of Event Streams are events in the real world, such a violation of a regulatory compliance, the time temperature higher than a given value, etc.

[A Stream Perspective]What is DDS?! DDS is a high-performance, real-time, highly- !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; available, fully-distributed, messaging technology that allows you to define data/event streams and make them dynamically discoverable! DDS is equipped with a rich set of QoS providing control on the key temporal and availability properties of data

Deﬁning Streams in DDS

DDS Topics [1/2] “org.opensplice.demo.TTempSensor”! A Topic defines a stream class/ !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; category! A Topic has associated a user defined type and QoS! The Topic name, type and QoS defines the key functional and non-functional invariants! Topics can be discovered or locally defined struct TempSensor { long Id; DURABILITY, float temp; DEADLINE, float hum; } PRIORITY, #pragma keylist TempSensor id …

DDS Topics [2/2] “org.opensplice.demo.TTempSensor” !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! DDS Topic types can have associated keys! Each unique key-value identifies a unique sub- stream of values -- called Topic Instance struct TempSensor { long id; DURABILITY, float temp; DEADLINE, float hum; } PRIORITY, #pragma keylist TempSensor id …

“Seeing” Streams !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; id =701 struct TempSensor {id =809 @key long id; Topic float temp; float hum; };id =977 Instances Instances

QoS Policies!"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;

Anatomy of a DDS Application !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;Domain val dp = DomainParticipant(0) Domain Participant // Create a Publisher / Subscriber Publisher p = dp.create_publisher(); Publisher Topic SubscriberSession Subscriber s = dp.create_subscriber(); // Create a Topic Topic<TempSensor> t = Gives access to a dp.create_topic<TempSensor>(“com.myco.TSTopic”) DDS Domain DataWrter DataReaderReader/WritersUser Defined for Types // Create a DataWriter/DataWriter DataWriter<TempSensor> dw = pub.create_datawriter(t); DataReader<TempSensor> dr = sub.create_datareader(t);

Anatomy of a DDS Application !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;Domain val dp = DomainParticipant(0) Domain Participant // Create a Publisher / Subscriber val pub = Publisher(dp) Publisher Topic SubscriberSession val sub = Subscriber(dp) // Create a Topic val topic = Topic[TempSensor](dp, “org.opensplice.demo.TTempSensor”) DataWrter DataReaderReader/WritersUser Defined for Types Pub/Sub // Create a DataWriter/DataWriter Abstractions DataWriter<TempSensor> dw = pub.create_datawriter(t); DataReader<TempSensor> dr = sub.create_datareader(t);

Anatomy of a DDS Application !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;Domain val dp = DomainParticipant(0) Domain Participant // Create a Publisher / Subscriber val pub = Publisher(dp) Publisher Topic SubscriberSession val sub = Subscriber(dp) // Create a Topic val topic = Topic[TempSensor](dp, “org.opensplice.demo.TTempSensor”) DataWrter DataReaderReader/Writers for User Defined for Types // Create a DataWriter/DataWriter val writer = DataWriter[TempSensor](pub, topic) val reader = DataReader[TempSensor](sub, topic) Reader/Writer for application // Write data defined Topic val t = new TempSensor ts(101, 25, 40) writer write ts; Types

Data & Event Streams! DDS does not provide different types for Data/Event Streams. !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; The difference between the two can be made through the DataReader API by using properly using read/take operations! DataReader::read ! Reads the value of the stream w/o removing it from the stream. As a result multiple read can see the last known value of the stream! DataReader::take ! Takes the value available on the stream (if any yet) and removes it from the stream

Stream Processing

Content Filtered Topics! Content Filtered Topics !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; provide a way of defining a Example: filters over an incoming // Create a Topic (on default domain) stream associated with a val topic = Topic[TempSensor](“TTempSensor”) val ftopic = given topic ContentFilteredTopic[TempSensor](“CFTempSensor”, topic, filter,! Filters are expressed as the params) “WHERE” clause of an SQL // - filter is a WHERE-like clause, such as: // “temp > 20 AND hum > 50” statement // “temp > %0” // “temp > hum” // “temp BETWEEN (%0 AND %1)! Filters can operate on any // // - params is the list of parameters to pass to the attribute of the type // filter expression – if any associated with the topic

Filter Expression Syntax !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! DDS Filters are condition over a topic type attributes! Temporal properties or causality cannot be captured via DDS filter expression

History !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; Data older than “n samples ago” get’s out the window! DDS provides a way of controlling data future past windows through the History QoS now The window keeps the last n data samples

[Putting it All Together]TempSensor Moving Average object MovingAverageFilter { def main(args: Array[String]) { if (args.length < 2) { !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; println("USAGE:ntMovingAverageFilter <window> <filter-expression>") } val topic = Topic[TempSensor]("TTempSensor") val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1)) val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt) val reader = DataReader[TempSensor](ftopic, rqos) reader.reactions += { case e: DataAvailable[_] => { var average: Float = 0 val window = e[TempSensor].reader.history window foreach (average += _.temp) average = average / window.length println("+--------------------------------------------------------") println("Moving Average: " + average) } } }

Product Organization

Product Organization Commercial Edition ! No Cost Runtime Licenses !"#$%&()*+,--.*/%&01234(*5**677*8&()0*8303%93: ! Your choice of licensing ! LGPL or Commercial ! Subscription or Perpetual ! Complete DDS Implementation Commercial Edition ! Comprehensive Developer and Deployment Support Options with a range of Service Level Agreements Commercial Add-Ons ! Individually licensable technologies Commercial Add-ons ! Rich ecosystem covering tools, integration, testing, etc.

Key Points So Far !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! DDS key abstractions for building stream processing architectures! DDS provides some event processing capabilities that facilitate the development of Stream Processing Filters! What else can you use for “Stream Processing” in combination with DDS? ! Let’s have Tom introduce us to the world of CEP and Esper!

! ! "! #$ %!# &!!&(

! *!+!! # & , ! - !-# !+!! # ./&# 0# !

33 455 ! # 2 !6 # ! 27$82

1$9( !## - /& !## ! :!#!-# 2 !!!-# 0# ##2!#;! )

#$ 4#$ ! ! ! ! <#, !!/# = #! () ! ! #! (

0 ! ! /#<"! !8> , ##, -?!@<" ! !> A 8 ! B8#!!-8# A 1

! 0CD% 0CD% % ! !!# ( % # :! #E :! # ! ! %F&!- ! ( G &4 &!! E 5 %--53 6!!# H IJ EJK

<!# #! !#4 #$ 2 #! 2! - L !# #! !# #! ! !5! . !# - ! $# #$ #!#4 ! ! #!+

5 0C5C , !#-!= ( !#;%F&! ! +!!! #!! # - ! ( ! - =+

0C! C ! -# &, M!# #5:!#!-# , !# - !#!-##! ! # -N !

&%# #! %# #! [insert into insert_into_def] select select_list from stream_def [as name] [, stream_def [as name]] [,...] [where search_conditions] [group by grouping_expression_list] [having grouping_search_conditions] [output output_specification] [order by order_by_expression_list] %!# select acctId, sum(amount) from Withdrawal.win:time(1 minute) group by acctId having sum(amount) > 1000 order by acctId asc 9

! $O P.! ! $ ! ! ! ! ! 4C!-# -#= M# $ .- -! #-$ !# 2!.% ,= .! &!#.! 4

.! ! ###4

#! ! $-! $ !# QM##,R!!4!S !$ !#! 4 7

&!#.! .! $!# M##, - select * from pattern [w=Withdrawal -> Deposit(acctId = w.acctId)] ! $ &!# w=Withdrawal -> (pin=PinChange(acctId = w.acctId) and pwd=PasswordChange(acctId = w.acctId) ) ## ,

every ( (w=Withdrawal -> Deposit(acctId = w.acctId)) or (d=Deposit -> Withdrawal(acctId = d.acctId)) ) T

&!#.! .! $!# ! ! !# .-

! ! !# every login=LoginEvent -> (timer:interval(20 sec) and not LogoutEvent(userId=login.userId)) /!

-$! ! !# every login=LoginEvent -> logout=LogOutEvent where timer:within(2 sec) ! every [2] login=LoginEvent ! every [2..] login=LoginEvent until logout=LogoutEvent

4 !! 2!# !!#;! ! ! EH4D# #! ! !! Q ! - !! H

"4+0C#! # ! !-#

"#$%H, $ &!!H E2!! 2N2!!# ! 2N !# ! !2 #, #2H! # ! # H: 2#,#! H% ! !#- 22+!! ! !

H. ! !2# # =2#!#! H# ! -!=!!-# ! ! ! H043 !Q% ! ##F%! H%#! !#E Q$ -#2

,#*-"./ # ( 01)2

: # % ( (

)*+!# ! ! ! % ! .+. 6 % ! .+. . ! ! %--

! ! 3! " , : !# ! !

: !# ! ! F6!!# ! &!! !#! !

# ! 2#!#!2 ! !2# ! -!=

F6F U ! V )

!= 455,,, V

DDS + CEP in Action

OpenSplice + Esper !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! Esper Provides an EsperIO framework for plugging-in new stream transports! Plugging OpenSplice into Esper is trivial even w/o relying on the EsperIO framework! Let’s have a look…

Demo Application

iShapes Application Spotted shapes represent subscriptions! To explore play with OpenSplice and Pierced shapes represent publications Esper, we’ll use the simd-cxx ishapes !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; application! Three Topics ! Circle, Square, Triangle! One Type: struct ShapeType { string color; long x; long y; long shapesize; }; #pragma keylist Shapetype color

Esper Setup !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;Step 1: Register Topic Typesval config = new Configurationval ddsConf = new ConfigurationEventTypeLegacyddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)config.addEventType("ShapeType", classOf[org.opensplice.demo.ShapeType].getName, ddsConf)val cep: EPServiceProvider = EPServiceProviderManager.getDefaultProvider(config)

Esper Setup !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;Step 2: Register a Listener for receiving Esper Events val listener = new UpdateListener { def update(ne: Array[EventBean], oe: Array[EventBean]) { ne foreach(e => { ! // Handle the event }) } }

Esper Setup !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;Step 3: Hook-up DDS to Esper reader.reactions += { case e: DataAvailable[ShapeType] => { (e.reader read) foreach(runtime sendEvent _) } }

iShapes FrameRate !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! Let’s suppose that we wanted to keep under control the iShapes Frame rate for ech given color! In Esper this can be achieved with the following expression: insert into ShapesxSec select color, count(*) as cnt from ShapeType.win:time_batch(1 second) group by color

iShapes Center of Mass !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;! Suppose that we wanted to compute the center of mass of all the shapes currently displayed over the last second! The Esper expression for this would be: select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg (y),int), shapesize) as NewShape from ShapeType.win:time(10 sec)

References !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:;" #1 OMG DDS Implementation " #1 Java-Based CEP Engine" Open Source " Open Source" www.opensplice.org " www.espertech.com" Fastest growing JVM Language " Scala API for OpenSplice DDS" Open Source " Open Source" www.scala-lang.org " code.google.com/p/escalier

:: Connect with Us :: !"#$%&()*+,-,.*/%&01234(*5**677*8&()0*8303%93:; ! opensplice.com ! forums.opensplice.org ! @acorsaro ! opensplice.org ! opensplicedds@prismtech.com ! @prismtech ! crc@prismtech.com ! sales@prismtech.com! youtube.com/opensplicetube ! slideshare.net/angelo.corsaro

http://www.scoop.it	62
http://icorsaro.net	4
http://opensplice.blogspot.com	2
http://www.icorsaro.net	2
http://webcache.googleusercontent.com	1

Getting Started with OpenSplice and Esper

by Angelo Corsaro

Accessibility

Categories

Upload Details

Usage Rights

5 Embeds 71

Statistics

Getting Started with OpenSplice and Esper Presentation Transcript