Stream Processing with DDS and CEP


Published on

This presentations explains the foundations of Stream Processing and shows how elegant Stream Processing Architectures can be built by using in synergy DDS and CEP.

Published in: Technology, Business
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Stream Processing with DDS and CEP

  1. 1. Angelo CORSARO, Ph.D.Chief Technology Officer OMG DDS Sig!
  2. 2. Defining "Stream Processing
  3. 3. Stream Processing [1/3]¨  Stream Processing can be seen as an architectural style Copyright  2010,  PrismTech  –    All  Rights  Reserved.   for building systems that operate over continuous (theoretically infinite) streams of data¨  Stream Processing is often reified under one of its many declinations, such as: ¨  Reactive Systems ¨  Signal Processing Systems ¨  Functional Stream Programming ¨  Data Flow Systems
  4. 4. Stream Processing [2/3] Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  Stream Processing Architectures are very natural for modeling systems needing to react to streams of data produced by the external world, such as the data produced by sensors, a camera or even the data produced by the stock exchange.¨  Stream Processing Systems usually operate in real- time over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc.
  5. 5. Stream Processing [3/3]¨  Stream Processing Systems are Copyright  2010,  PrismTech  –    All  Rights  Reserved.   typically modeled as collection of modules communicating via typed data channels Filter¨  Modules usually play one of the Filter following roles: ¨  Sources: Injecting data into the System Filter ¨  Filters/Actors: Performing some computation over sources Sink ¨  Sinks: Consuming the data produced Channel by the system source
  6. 6. Implementing"Stream Processing
  7. 7. Stream Processing Alternatives¨  Different possibilities exist to Copyright  2010,  PrismTech  –    All  Rights  Reserved.   implement channels as well s filters Filter¨  DDS is a very good fit for Filter channels due to its strong typing as well as performance and Filter scalability¨  Filter can be implemented by DDS Reader custom business logic or by DDS Topic leveraging CEP DDS Writer CEP | Custom
  8. 8. Stream Processing " with DDS
  9. 9. Data Distribution in a Nutshell
  10. 10. Data Distribution in a Nutshell struct TempSensor {!¨  Data distribution is about long Id; ! float temp;! making application defined float hum;! Copyright  2010,  PrismTech  –    All  Rights  Reserved.   }! data available where needed #pragma keylist TempSensor id! and when needed, while preserving and end-to-end DR type contract DW Id Temp Scale¨  Data is a first class concept, it can be Created, Read, 101 25 C DR Updated, and eventually DW 202 78 F 303 299 K Disposed (CRUD)¨  The last value (or last N-values) DR of a Data is always available to DW Fully Distributed applications DW: DataWriter Global Data Space DR: DataReader
  11. 11. DDS Topics [1/2] “org.opensplice.demo.TTempSensor” A Topic defines the subject of Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  Name publications and subscriptions¨  A Topic has associated a user defined type and QoS¨  The Topic name, type and Topic QoS have a well defined role in matching subscriptions Type QoS¨  Topics can be discovered or locally defined struct TempSensor {! long Id; ! DURABILITY, float temp;! DEADLINE, float hum;! PRIORITY, }! … #pragma keylist TempSensor id!
  12. 12. DDS Topics [2/2] “org.opensplice.demo.TTempSensor” Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Name¨  DDS Topic types can have associated keys¨  Each unique key-value Topic identify a Topic Instance – a specific stream of values Type QoS struct TempSensor {! long Id; ! DURABILITY, float temp;! DEADLINE, float hum;! PRIORITY, }! … #pragma keylist TempSensor id!
  13. 13. Anatomy of a DDS Application Copyright  2010,  PrismTech  –    All  Rights  Reserved.  Domain val dp = DomainParticipant(0) ! Domain Participant // Create a Publisher / Subscriber! Publisher p = dp.create_publisher();!Session Subscriber s = dp.create_subscriber();! Publisher Topic Subscriber // Create a Topic! Gives access to a Topic<TempSensor> t = ! dp.create_topic<TempSensor>(“com.myco.TSTopic”)! DDS DomainReader/WritersUser Defined for Types DataWrter DataReader // Create a DataWriter/DataWriter! DataWriter<TempSensor> dw = pub.create_datawriter(t);! DataReader<TempSensor> dr = sub.create_datareader(t);!
  14. 14. Anatomy of a DDS Application Copyright  2010,  PrismTech  –    All  Rights  Reserved.  Domain val dp = DomainParticipant(0) ! Domain Participant // Create a Publisher / Subscriber! val pub = Publisher(dp)!Session val sub = Subscriber(dp)! Publisher Topic Subscriber // Create a Topic! val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”)!Reader/WritersUser Defined for Types Pub/Sub DataWrter DataReader Abstractions // Create a DataWriter/DataWriter! DataWriter<TempSensor> dw = pub.create_datawriter(t);! DataReader<TempSensor> dr = sub.create_datareader(t);!
  15. 15. Anatomy of a DDS Application Copyright  2010,  PrismTech  –    All  Rights  Reserved.  Domain val dp = DomainParticipant(0) ! Domain Participant // Create a Publisher / Subscriber! val pub = Publisher(dp)!Session val sub = Subscriber(dp)! Publisher Topic Subscriber // Create a Topic! val topic = Topic[TempSensor](dp, ! “org.opensplice.demo.TTempSensor”)!Reader/Writers for User Defined for Types // Create a DataWriter/DataWriter! DataWrter DataReader val writer = DataWriter[TempSensor](pub, topic)! Reader/Writer for val reader = DataReader[TempSensor](sub, topic) ! ! application // Write data! defined Topic val t = new TempSensor ts(101, 25, 40)! Types writer write ts;!
  16. 16. Domain & PartitionsDomain (e.g. Domain 123) Domain Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Participant Topic Partition (e.g. Partition “Telemetry”) Publisher Subscriber Topic Instances/Samples DataWrter DataReader
  17. 17. Stream Processing in DDS DDS naturally supports stream processing through three Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  main concepts: ¨  Topics ¨  Data Writers ¨  Data Readers¨  In addition DDS’ dynamic discovery, high availability and high performance make it very easy to architect, deploy and upgrade stream processing systems
  18. 18. DDS Topics as Streams¨  In Stream Processing, Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Streams are typed data channel Filter¨  A DDS Topic provide an Filter elegant way of defining the Streams that make up a Filter Streaming System while capturing their type, as well as their Non-Functional DDS Topic Constraints
  19. 19. DDS Data Writers as Sources Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  DDS Data Writers make ideal sources for Filter Stream Processing Filter systems due to their: Filter ¨  Dynamic Discovery ¨  Durability ¨  Fault-Tolerance DDS Writer
  20. 20. DDS Data Readers as Sinks Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  DDS Data Readers make ideal sinks for Filter Filter Stream Processing systems due to their: Filter ¨  Dynamic Discovery DDS Reader ¨  Durability
  21. 21. DDS & Filters¨  Filters/Actors perform Copyright  2010,  PrismTech  –    All  Rights  Reserved.   transformations from one or more input streams to one or more output streams Filter Filter¨  The combination of DDS ContentFilteredTopics with a Filter powerful language like Scala can make it relatively easy to build custom filters¨  Let’s explore this path… DDS+Application Logic
  22. 22. Content Filtered Topics¨  Content Filtered Topics Copyright  2010,  PrismTech  –    All  Rights  Reserved.   provide a way of defining a Example: local projection of the // Create a Topic (on default domain)! stream of data associated val topic = Topic[TempSensor](“TTempSensor”)! val ftopic = ! with a given topic ContentFilteredTopic[TempSensor](“CFTempSensor”,! topic,!¨  Filters are expressed as the filter,! params)! “WHERE” clause of an SQL ! // - filter is a WHERE-like clause, such as: ! statement // “temp > 20 AND hum > 50”! // “temp > %0”!¨  Filters can operate on any // // “temp > hum”! “temp BETWEEN (%0 AND %1)! attribute of the type //! // - params is the list of parameters to pass to the ! associated with the topic // filter expression – if any!
  23. 23. Filter Expression Syntax Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  DDS Filters are condition over a topic type attributes¨  Temporal properties or causality cannot be captured via DDS filter expression
  24. 24. History Copyright  2010,  PrismTech  –    All  Rights  Reserved.   Data older than “n samples ago” get’s out the window¨  DDS provides a way of controlling data future past windows through the History QoS now The window keeps the last n data samples
  25. 25. [Putting it All Together]"TempSensor Moving Average object MovingAverageFilter {! def main(args: Array[String]) {! if (args.length < 2) {! Copyright  2010,  PrismTech  –    All  Rights  Reserved.   println("USAGE:ntMovingAverageFilter <window> <filter-expression>")! }! ! val topic = Topic[TempSensor]("TTempSensor")! val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1))! ! val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt)! val reader = DataReader[TempSensor](ftopic, rqos)! ! reader.reactions += {! case e: DataAvailable[TempSensor] => {! var average: Float = 0! val window = e.reader.history! window foreach (average += _.temp)! average = average / window.length! println("+--------------------------------------------------------")! println("Moving Average: " + average)! }! }! }!
  26. 26. Key Points So Far Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  DDS provides some event processing capabilities that facilitate the development of Stream Processing Filters¨  Higher Level programming languages like Scala can make it very easy to concisely express complex filters¨  Note: Scala targets the JVM as well as the .Net CLR
  27. 27. Stream Processing " with DDS & CEP
  28. 28. CEP In Brief [1/2]¨  Complex Event Processing (CEP) Copyright  2010,  PrismTech  –    All  Rights  Reserved.   engines provide a declarative way of transforming a set of input streams into one or more output streams CEP¨  The declarative language -  select * from TempSensor(temp < 30) provided by CEP is usually based on some extension of SQL to -  select avg(temp) include time, pattern matching from sec) and causality
  29. 29. CEP In Brief [1/2] Filter CEP emerged from Stream Filter Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  Processing as a way to Filter facilitate the development of “Filters/Actors” and make it “declarative” CEP¨  In line with the Stream -  select * Processing principles, CEP from TempSensor(temp < 30) operate on typed data streams -  select avg(temp) from sec)
  30. 30. DDS + CEP = Stream Processing¨  DDS provides a powerful Stream abstraction that is: Copyright  2010,  PrismTech  –    All  Rights  Reserved.   ¨  Type Safe ¨  High Performance ¨  Highly Available CEP ¨  Decoupled in Time/Space CEP ¨  Dynamically Discoverable¨  CEP provide a powerful abstraction for CEP processing streams¨  The combination of DDS + CEP is not DDS Reader only natural but a perfect fit for building DDS Topic high performance, highly available DDS Writer Stream Processing Systems
  31. 31. DDS + CEP in Action
  32. 32. Esper CEP Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  Esper is an Open Source, pure Java, CEP engine widely used in a wide variety of applications ranging from the Capital Market to Defense and Aerospace ¨¨  Esper Rules are expressed EPL (Esper Processing Language) which can be seen as an extension of SQL with support for time, causality and pattern matching
  33. 33. OpenSplice + Esper Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  Esper Provides an EsperIO framework for plugging-in new stream transports¨  Plugging OpenSplice into Esper is trivial even w/o relying on the EsperIO framework¨  Let’s have a look…
  34. 34. Demo Application
  35. 35. iShapes Application Spotted shapes represent subscriptions¨  To explore play with OpenSplice Pierced shapes represent publications Copyright  2010,  PrismTech  –    All  Rights  Reserved.   and Esper, we’ll use the simd-cxx ishapes application¨  Three Topics ¨  Circle, Square, Triangle¨  One Type: struct ShapeType {! string color;! long x;! long y;! long shapesize;! };! #pragma keylist Shapetype color!
  36. 36. Esper Setup Copyright  2010,  PrismTech  –    All  Rights  Reserved.  Step 1: Register Topic Typesval config = new Configuration!val ddsConf = new ConfigurationEventTypeLegacy!!ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)!!config.addEventType("ShapeType", ! classOf[org.opensplice.demo.ShapeType].getName, ! ddsConf)!!val cep: EPServiceProvider = ! EPServiceProviderManager.getDefaultProvider(config)!
  37. 37. Esper Setup Copyright  2010,  PrismTech  –    All  Rights  Reserved.  Step 2: Register a Listener for receiving Esper Events val listener = new UpdateListener {! def update(ne: Array[EventBean], oe: Array[EventBean]) {! ne foreach(e => {! "// Handle the event! })! }! }!
  38. 38. Esper Setup Copyright  2010,  PrismTech  –    All  Rights  Reserved.  Step 3: Hook-up DDS to Esper reader.reactions += {! case e: DataAvailable[ShapeType] => {! (e.reader read) foreach(runtime sendEvent _)! ! }! }!
  39. 39. iShapes FrameRate Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  Let’s suppose that we wanted to keep under control the iShapes Frame rate for ech given color¨  In Esper this can be achieved with the following expression: insert into ShapesxSec ! select color, count(*) as cnt ! from second) ! group by color!
  40. 40. iShapes Center of Mass Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  Suppose that we wanted to compute the center of mass of all the shapes currently displayed over the last second¨  The Esper expression for this would be: select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg (y),int), shapesize) as NewShape ! from sec)!
  41. 41. ReferencesOpenSplice DDS Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¥  #1 OMG DDS Implementation ¥  #1 Java-Based CEP Engine¥  Open Source ¥  Open Source¥ ¥¥  Fastest growing JVM Language ¥  Scala API for OpenSplice DDS¥  Open Source ¥  Open Source¥ ¥
  42. 42. Summing Up
  43. 43. Concluding Remarks Copyright  2010,  PrismTech  –    All  Rights  Reserved.  ¨  DDS provides a very good foundation to build high performance, highly available and scalable streaming systems¨  DDS + CEP are the most natural, effective and efficient way of building next generation Stream Processing Systems!
  44. 44. OpenSplice DDSDelivering Performance, Openness, and Freedom Copyright  2010,  PrismTech  –    All  Rights  Reserved.