Your SlideShare is downloading. ×
Open splicedds espercep-webinar
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Open splicedds espercep-webinar

128
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
128
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech angelo.corsaro@prismtech.com! 海 賊 天 使
  • 2. Defining " Stream Processing
  • 3. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Stream Processing ¨  Stream Processing can be seen as an architectural style for building systems that operate over continuous (theoretically infinite) streams of data ¨  Stream Processing is often reified under one of its many declinations, such as: ¨  Reactive Systems ¨  Signal Processing Systems ¨  Functional Stream Programming ¨  Data Flow Systems [1/3]
  • 4. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Stream Processing ¨  Stream Processing Architectures are very natural for modeling systems needing to react to streams of data produced by the external world, such as the data produced by sensors, a camera or even the data produced by the stock exchange. ¨  Stream Processing Systems usually operate in real- time over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc. [2/3]
  • 5. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Stream Processing ¨  Stream Processing Systems are typically modeled as collection of modules communicating via typed data channels ¨  Modules usually play one of the following roles: ¨  Sources: Injecting data into the System ¨  Filters/Actors: Performing some computation over sources ¨  Sinks: Consuming the data produced by the system [3/3] Filter Filter Filter source Sink Channel
  • 6. Implementing" Stream Processing
  • 7. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Stream Processing Alternatives ¨  Different possibilities exist to implement channels as well s filters ¨  DDS is a very good fit for channels due to its strong typing as well as performance and scalability ¨  Filter can be implemented by custom business logic or by leveraging CEP Filter Filter Filter DDS Writer DDS Reader DDS Topic CEP | Custom
  • 8. Stream Processing " with DDS
  • 9. Data Distribution in a Nutshell
  • 10. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Data Distribution in a Nutshell ¨  Data distribution is about making application defined data available where needed and when needed, while preserving and end-to-end type contract ¨  Data is a first class concept, it can be Created, Read, Updated, and eventually Disposed (CRUD) ¨  The last value (or last N-values) of a Data is always available to applications Id Temp Scale 101 25 C 202 78 F 303 299 K DW DW DW DR DR DR struct TempSensor {! long Id; ! float temp;! float hum;! }! #pragma keylist TempSensor id! Fully Distributed Global Data SpaceDW: DataWriter DR: DataReader
  • 11. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   DDS Topics ¨  A Topic defines the subject of publications and subscriptions ¨  A Topic has associated a user defined type and QoS ¨  The Topic name, type and QoS have a well defined role in matching subscriptions ¨  Topics can be discovered or locally defined Topic Name QoSType DURABILITY, DEADLINE, PRIORITY, … [1/2] “org.opensplice.demo.TTempSensor” struct TempSensor {! long Id; ! float temp;! float hum;! }! #pragma keylist TempSensor id!
  • 12. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   DDS Topics ¨  DDS Topic types can have associated keys ¨  Each unique key-value identify a Topic Instance – a specific stream of values [2/2] Topic Name QoSType “org.opensplice.demo.TTempSensor” DURABILITY, DEADLINE, PRIORITY, … struct TempSensor {! long Id; ! float temp;! float hum;! }! #pragma keylist TempSensor id!
  • 13. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Anatomy of a DDS Application Domain Reader/Writers User Defined for Types Session val dp = DomainParticipant(0) ! // Create a Publisher / Subscriber! Publisher p = dp.create_publisher();! Subscriber s = dp.create_subscriber();! // Create a Topic! Topic<TempSensor> t = ! dp.create_topic<TempSensor>(“com.myco.TSTopic”)! Domain Participant Publisher DataWrter Topic Subscriber DataReader // Create a DataWriter/DataWriter! DataWriter<TempSensor> dw = pub.create_datawriter(t);! DataReader<TempSensor> dr = sub.create_datareader(t);! Gives access to a DDS Domain
  • 14. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Anatomy of a DDS Application Domain Reader/Writers User Defined for Types Session val dp = DomainParticipant(0) ! // Create a Publisher / Subscriber! val pub = Publisher(dp)! val sub = Subscriber(dp)! // Create a Topic! val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”)! Domain Participant Publisher DataWrter Topic Subscriber DataReader // Create a DataWriter/DataWriter! DataWriter<TempSensor> dw = pub.create_datawriter(t);! DataReader<TempSensor> dr = sub.create_datareader(t);! Pub/Sub Abstractions
  • 15. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Anatomy of a DDS Application Domain Reader/Writers for User Defined for Types Session Domain Participant Publisher DataWrter Topic Subscriber DataReader // Create a DataWriter/DataWriter! val writer = DataWriter[TempSensor](pub, topic)! val reader = DataReader[TempSensor](sub, topic) ! ! // Write data! val t = new TempSensor ts(101, 25, 40)! writer write ts;! Reader/Writer for application defined Topic Types // Create a Publisher / Subscriber! val pub = Publisher(dp)! val sub = Subscriber(dp)! // Create a Topic! val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”)! val dp = DomainParticipant(0) !
  • 16. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Domain & Partitions Topic Instances/Samples Domain (e.g. Domain 123) Partition (e.g. Partition “Telemetry”) Domain Participant Topic Publisher DataWrter Subscriber DataReader
  • 17. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Stream Processing in DDS ¨  DDS naturally supports stream processing through three main concepts: ¨  Topics ¨  Data Writers ¨  Data Readers ¨  In addition DDS’ dynamic discovery, high availability and high performance make it very easy to architect, deploy and upgrade stream processing systems
  • 18. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   DDS Topics as Streams ¨  In Stream Processing, Streams are typed data channel ¨  A DDS Topic provide an elegant way of defining the Streams that make up a Streaming System while capturing their type, as well as their Non-Functional Constraints Filter Filter Filter DDS Topic
  • 19. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   DDS Data Writers as Sources ¨  DDS Data Writers make ideal sources for Stream Processing systems due to their: ¨  Dynamic Discovery ¨  Durability ¨  Fault-Tolerance Filter Filter Filter DDS Writer
  • 20. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   DDS Data Readers as Sinks ¨  DDS Data Readers make ideal sinks for Stream Processing systems due to their: ¨  Dynamic Discovery ¨  Durability Filter Filter Filter DDS Reader
  • 21. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   DDS & Filters ¨  Filters/Actors perform transformations from one or more input streams to one or more output streams ¨  The combination of DDS ContentFilteredTopics with a powerful language like Scala can make it relatively easy to build custom filters ¨  Let’s explore this path… DDS+Application Logic Filter Filter Filter
  • 22. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Content Filtered Topics ¨  Content Filtered Topics provide a way of defining a local projection of the stream of data associated with a given topic ¨  Filters are expressed as the “WHERE” clause of an SQL statement ¨  Filters can operate on any attribute of the type associated with the topic Example: // Create a Topic (on default domain)! val topic = Topic[TempSensor](“TTempSensor”)! val ftopic = ! ContentFilteredTopic[TempSensor](“CFTempSensor”,! topic,! filter,! params)! ! // - filter is a WHERE-like clause, such as: ! // “temp > 20 AND hum > 50”! // “temp > %0”! // “temp > hum”! // “temp BETWEEN (%0 AND %1)! //! // - params is the list of parameters to pass to the ! // filter expression – if any!
  • 23. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Filter Expression Syntax ¨  DDS Filters are condition over a topic type attributes ¨  Temporal properties or causality cannot be captured via DDS filter expression
  • 24. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   History ¨  DDS provides a way of controlling data windows through the History QoS The window keeps the last n data samples Data older than “n samples ago” get’s out the window pastfuture now
  • 25. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   [Putting it All Together]" TempSensor Moving Average object MovingAverageFilter {! def main(args: Array[String]) {! if (args.length < 2) {! println("USAGE:ntMovingAverageFilter <window> <filter-expression>")! }! ! val topic = Topic[TempSensor]("TTempSensor")! val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1))! ! val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt)! val reader = DataReader[TempSensor](ftopic, rqos)! ! reader.reactions += {! case e: DataAvailable[TempSensor] => {! var average: Float = 0! val window = e.reader.history! window foreach (average += _.temp)! average = average / window.length! println("+--------------------------------------------------------")! println("Moving Average: " + average)! }! }! }!
  • 26. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Key Points So Far ¨  DDS provides some event processing capabilities that facilitate the development of Stream Processing Filters ¨  Higher Level programming languages like Scala can make it very easy to concisely express complex filters ¨  Note: Scala targets the JVM as well as the .Net CLR
  • 27. Stream Processing " with DDS & CEP
  • 28. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   CEP In Brief ¨  Complex Event Processing (CEP) engines provide a declarative way of transforming a set of input streams into one or more output streams ¨  The declarative language provided by CEP is usually based on some extension of SQL to include time, pattern matching and causality -  select * from TempSensor(temp < 30) -  select avg(temp) from TempSensor.win:time(1 sec) CEP [1/2]
  • 29. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   CEP In Brief ¨  CEP emerged from Stream Processing as a way to facilitate the development of “Filters/Actors” and make it “declarative” ¨  In line with the Stream Processing principles, CEP operate on typed data streams -  select * from TempSensor(temp < 30) -  select avg(temp) from TempSensor.win:time(1 sec) CEP [1/2] Filter Filter Filter
  • 30. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   DDS + CEP = Stream Processing ¨  DDS provides a powerful Stream abstraction that is: ¨  Type Safe ¨  High Performance ¨  Highly Available ¨  Decoupled in Time/Space ¨  Dynamically Discoverable ¨  CEP provide a powerful abstraction for processing streams ¨  The combination of DDS + CEP is not only natural but a perfect fit for building high performance, highly available Stream Processing Systems CEP CEP CEP DDS Writer DDS Reader DDS Topic
  • 31. DDS + CEP in Action
  • 32. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Esper CEP ¨  Esper is an Open Source, pure Java, CEP engine widely used in a wide variety of applications ranging from the Capital Market to Defense and Aerospace ¨  http://www.espertech.com ¨  Esper Rules are expressed EPL (Esper Processing Language) which can be seen as an extension of SQL with support for time, causality and pattern matching
  • 33. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   OpenSplice + Esper ¨  Esper Provides an EsperIO framework for plugging-in new stream transports ¨  Plugging OpenSplice into Esper is trivial even w/o relying on the EsperIO framework ¨  Let’s have a look…
  • 34. Demo Application
  • 35. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   iShapes Application ¨  To explore play with OpenSplice and Esper, we’ll use the simd-cxx ishapes application ¨  Three Topics ¨  Circle, Square, Triangle ¨  One Type: struct ShapeType {! string color;! long x;! long y;! long shapesize;! };! #pragma keylist Shapetype color! Spotted shapes represent subscriptions Pierced shapes represent publications
  • 36. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Esper Setup Step 1: Register Topic Types val config = new Configuration! val ddsConf = new ConfigurationEventTypeLegacy! ! ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)! ! config.addEventType("ShapeType", ! classOf[org.opensplice.demo.ShapeType].getName, ! ddsConf)! ! val cep: EPServiceProvider = ! EPServiceProviderManager.getDefaultProvider(config)!
  • 37. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Esper Setup Step 2: Register a Listener for receiving Esper Events val listener = new UpdateListener {! def update(ne: Array[EventBean], oe: Array[EventBean]) {! ne foreach(e => {! "// Handle the event! })! }! }!
  • 38. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Esper Setup Step 3: Hook-up DDS to Esper reader.reactions += {! case e: DataAvailable[ShapeType] => {! (e.reader read) foreach(runtime sendEvent _)! ! }! }!
  • 39. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   iShapes FrameRate ¨  Let’s suppose that we wanted to keep under control the iShapes Frame rate for ech given color ¨  In Esper this can be achieved with the following expression: insert into ShapesxSec ! select color, count(*) as cnt ! from ShapeType.win:time_batch(1 second) ! group by color!
  • 40. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   iShapes Center of Mass ¨  Suppose that we wanted to compute the center of mass of all the shapes currently displayed over the last second ¨  The Esper expression for this would be: select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg (y),int), shapesize) as NewShape ! from ShapeType.win:time(10 sec)!
  • 41. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   ¥  Fastest growing JVM Language ¥  Open Source ¥  www.scala-lang.org References OpenSplice DDS ¥  #1 OMG DDS Implementation ¥  Open Source ¥  www.opensplice.org ¥  Scala API for OpenSplice DDS ¥  Open Source ¥  code.google.com/p/escalier ¥  #1 Java-Based CEP Engine ¥  Open Source ¥  www.espertech.com
  • 42. Summing Up
  • 43. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Concluding Remarks ¨  DDS provides a very good foundation to build high performance, highly available and scalable streaming systems ¨  DDS + CEP are the most natural, effective and efficient way of building next generation Stream Processing Systems!
  • 44. Copyright  2010,  PrismTech  –    All  Rights  Reserved.   http://www.opensplice.com/ http://www.opensplice.org/ emailto:opensplicedds@prismtech.com http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com http://bit.ly/1Sreg http://www.slideshare.net/angelo.corsaro http://twitter.com/acorsaro/ D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m OpenSplice DDS