Building Reactive Applications with DDS


Published on

Reactive architectures are emerging as the way to build systems that are responsive, scalable, resilient and event-driven. In other terms, systems that deliver highly responsive user experiences with a real-time feel, that are scalable, resilient, and ready to be deployed on multicore and cloud computing architectures. The Reactive Manifesto (see captures the key traits that characterize reactive architectures.

The Data Distribution Service (DDS) incarnates the principles enumerated by the reactive manifesto and provides a very good platform for building reactive systems. In this webcast I will (1) introduce the key principles of Reactive Architectures, (2) explain the DDS features that are essential to build reactive systems, and (3) introduce some programming techniques that remove inversion of control while maintaining applications even-driven.

Published in: Technology, Business

Building Reactive Applications with DDS

  1. 1. Building Reactive Applications with DDS Angelo Corsaro, PhD Chief Technology Officer OMG DDS SIG Co-Chair PrismTech
  2. 2. • Reactive Systems were characteristic of “niche” applications domains such as Industrial, Military, Telco, Finance and Aerospace • To address the challenges imposed contemporary Internet Applications an increasing movement of architects and developers is recognising the importance of the techniques and technologies used for building Reactive Systems PrismTech Copyright PrismTech, 2014 Reactive Systems Go Mainstream
  3. 3. Integrated Modular Vetronics Air Traffic Control & Management PrismTech Training & Simulation Systems Unmanned Air Vehicles Naval Combat Systems Aerospace Applications Copyright PrismTech, 2014 Defense and Aerospace
  4. 4. Autonomous Vehicle Systems Train Control / Management Systems PrismTech Smart Grids / Power Generation Smart Cities Connected Medical Devices Financial Services Copyright PrismTech, 2014 Industrial Internet
  5. 5. Reactive Manifesto • The techniques and Responsive Event-Driven Resilient • Recently, the Reactive Manifesto was defined to summarise the key traits of “Reactive Applications” Scalable PrismTech Copyright PrismTech, 2014 technologies traditionally used in a class of Reactive and Interactive Systems is gaining quite a bit of attention from the mainstream developers community
  6. 6. Reactive Manifesto • Event-Driven: modular, • Responsive: Timely react to stimuli • Resilient: Tolerate failures functionally and ideally temporally • Scalable: Gracefully scale up and down depending on load demands Event-Driven Resilient Scalable PrismTech Copyright PrismTech, 2014 Responsive asynchronous
  7. 7. Reactive Systems
  8. 8. Reactive Systems 1 • A Reactive System designates a permanently operating system that responds, reacts, to external stimuli at a speed determined by its environment, e.g. an industrial control system. [1] Nicolas Halbwachs, Synchronous Programming of Reactive Systems PrismTech Copyright PrismTech, 2014 Reactive Systems
  9. 9. Transformational Systems • A Transformational System designates a system that terminates after transforming its input into its output, e.g. a compiler PrismTech Copyright PrismTech, 2014 Transformational Systems
  10. 10. Interactive Systems • An Interactive System designates a systems that permanently communicates with its environment at its own speed, e.g. a computer game PrismTech Copyright PrismTech, 2014 Interactive Systems
  11. 11. Reactive vs. Interactive Systems Interactive • react to an environment which • react to an environment which • response time needs to be • response time needs to be • distributed concurrent systems • distributed concurrent systems • data-centric • data-centric “cannot wait”, e.g. reacting late has harsh consequences deterministic “doesn’t like to wait”, e.g. reacting late induces QoS degradation acceptable The term reactive systems is increasingly used to denote interactive systems with stringent response time, performance and scalability constraints. PrismTech Copyright PrismTech, 2014 Reactive
  12. 12. Reactive Systems Key Traits Concurrency • Aside from the concurrency between the system and its environment, it is natural to Strict Temporal Requirements • Reactive Systems have strict requirements with respect to the rate at which the need to process input as well as the response time for their outputs Determinism • The output of these systems are generally determined by the input and the occurrence time, e.g. no scheduling effects on the temporal properties of the output Reliability • Reactive Systems are often life/mission critical, as such reliability is of utmost importance PrismTech Copyright PrismTech, 2014 decompose these systems as an ensemble of concurrent components that cooperate to achieve the intended behaviour
  13. 13. Architectural Styles for Reactive Systems
  14. 14. Stream Processing • Stream Processing is a commonly adopted architectural style for building • Stream Processing is often applied in: - Interactive Systems - Signal Processing Systems - Functional Stream Programming - PrismTech Reactive Systems Data Flow Systems Copyright PrismTech, 2014 systems that operate over continuous (theoretically infinite) streams of data
  15. 15. Stream Processing for modelling systems that react to streams of data produced by the cyber-phisical world, such as data produced by sensors, the stock market, etc. • Stream Processing Systems operate in realtime over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc. PrismTech Copyright PrismTech, 2014 • Stream Processing Architectures are ideal
  16. 16. Stream Processing • Stream Processing Systems are typically modelled • Modules usually play one of the following roles: - - PrismTech Filter Sources: Injecting data into the System Filter Filter Filters/Actors: Performing some computation over sources Sinks: Consuming the data produced by the system Copyright PrismTech, 2014 as collection of concurrent modules communicating via typed data channels Source Stream Sink
  17. 17. Stream Processing with DDS
  18. 18. Streams in DDS infinite sequence of data samples of a given type T Filter • As such, in DDS, a stream can be naturally represented with a Topic • Example: - PrismTech Topic<PressureType> Topic<MarketData> Filter Filter Stream Copyright PrismTech, 2014 • A stream represents a potentially
  19. 19. Topic • A Topic defines a domain-wide information’s class Copyright PrismTech, 2014 • A Topic is defined by means of a (name, type, qos) tuple, where type: is the programming language type associated with the topic. Types are extensible and evolvable Name - qos: is a collection of policies that express the nonfunctional properties of this topic, e.g. reliability, persistence, etc. Topic Qo S - PrismTech name: identifies the topic within the domain e Typ -
  20. 20. •As explained in the previous slide a topic defines a class/type of information •Topics can be defined as Singleton or can have multiple Instances •Topic Instances are identified by means of the topic key •A Topic Key is identified by a tuple of attributes -- like in databases •Remarks: • A Singleton topic has a single domain-wide instance • A “regular” Topic can have as many instances as the number of different key values, e.g., if the key is an 8-bit character then the topic can have 256 different instances PrismTech Copyright PrismTech, 2014 Topic and Instances
  21. 21. • Sources in DDS are mapped to Filter DataWriters for a given Topic • Sinks in DDS are mapped to Filter Filter DataReaders for a given Topic Source PrismTech Copyright PrismTech, 2014 Source and Sinks in DDS Sink
  22. 22. Filters in DDS application business logic • Each Filter has one or more input stream and one or more output streams • Obviously, a Filter consumes data from an input stream through a DDS data reader and pushed data into an output stream through a DDS data writer PrismTech Filter Filter Filter Copyright PrismTech, 2014 • Filters implement bit and pieces of the
  23. 23. DDS Architectural Benefits
  24. 24. Loose Coupling • Anonymous Communication => No Spatial Coupling • Non-Blocking and Asynchronous Interaction => No control or temporal coupling In DDS the only thing that matters is the kind of information a module is interested in consuming or producing — who is producing the data or when the data has been produced is not relevant PrismTech Copyright PrismTech, 2014 Stream Processing Architectures in general and DDS-based architecture in particular promote loose coupling
  25. 25. Location Transparency • Location Transparency is an important architectural property as it makes it possible to completely decouple the logical decomposition of a system from its actual physical deployment • DDS Automatic Discovery brings location transparency to the next level by enabling systems that are zero-conf, self-forming and self-healing Module Logical Deployment Unit Physical Deployment Unit Module Module PrismTech Module Module Copyright PrismTech, 2014 Module
  26. 26. Temporal Decoupling Volatile  Durability decoupling by means of its Durability Policy • As such the data produced by a Source can be: - - - PrismTech Source t Sink t Sink t Volatile: available for those that are around at the time at production Transient: available as far as the system/source is running Durable: available forever Transient  Durability Source t Sink t Sink t Copyright PrismTech, 2014 • DDS provides full control over temporal
  27. 27. Failure Detection traditional faults as well as performance failures • The Fault-Detection mechanism is Source TD Sink t controlled by means of the DDS Liveliness policy • Performance Failures can be detected using the Deadline Policy which allows to receive notification when data is not received within the expected delays Fault  Notification Source t Sink P P P Performance  Failure  Notification PrismTech t Copyright PrismTech, 2014 • DDS provides mechanism for detecting
  28. 28. Fault-Masking mechanism that allow to replicate Sources and transparently switch over when a failure occurs • At any point in time the “active” source is the one with the highest strength. Where the strength is an integer parameter controller by the user PrismTech Source Source t Sink t Copyright PrismTech, 2014 • DDS provides a built-in fault-masking
  29. 29. Fault-Masking mechanism that allow to replicate Sources and transparently switch over when a failure occurs • At any point in time the “active” source is the one with the highest strength. Where the strength is an integer parameter controller by the user PrismTech Source Source t Sink t Copyright PrismTech, 2014 • DDS provides a built-in fault-masking
  30. 30. Performance - 15-20 usec Inter-Core Latency - 45-75 usec over GBps Ethernet • Throughput - PrismTech Up to 10+M msg/sec inter-core Up to 5M msg/sec inter node Copyright PrismTech, 2014 • Latency
  31. 31. Standard and Interoperable introduced as an OMG standard in 2006 to address Real-time and interoperable Data Sharing in Network Centric Systems • Key requirements for the standard were high and predictable performance, and scalability to ultra-large-scale deployments • DDS is recommended by key administration worldwide, e.g. DoD, MoD, EUROCAE, etc. and widely adopted across application domains, e.g., Automated Trading, Simulations, SCADA, Telemetry, etc. PrismTech Copyright PrismTech, 2014 • The Data Distribution Service (DDS) was
  32. 32. Escape from Callback-Hell
  33. 33. Dealing with Events environment • Stimuli coming from the cyber-physical world are often represented as events to be handled by the Reactive/Interactive System - e.g. think about the event from a GUI, or the data available event in DDS • Events are often handled through some form of callbacks, such as listeners, functors, etc. This stile of event handling leads to inversion of control PrismTech Copyright PrismTech, 2014 • Reactive and Interactive Systems continuously interact with their
  34. 34. The Callback Hell • The Inversion of Control induced by callback-based code in known to • As an mental exercise, think about the call-back style code you’d have to write to in Java for drawing when the mouse is being dragged. Although the logic is simple, it is far cry from having a “declarative style” • The problem of callback management, infamously known as Callback Hell, has become even more important due to the surge of Reactive Systems… • Can we escape the Callback Hell? PrismTech Copyright PrismTech, 2014 make applications brittle and harder to understand
  35. 35. Reactive Programming • Reactive Programming is a paradigm based on a relaxed form of • The Reactive Programming is built around the notion of continuous timevarying values and propagation of change • Reactive Programming facilitates the declarative development of non- blocking event driven applications — in essence you express what should be done and the runtime decides when to do it • Reactive Programming was popularised in the context of Functional Programming Languages by the seminal done by Elliot and Hudak in 1997 as part of Fran — a framework for composing richly interactive multimedia animations PrismTech Copyright PrismTech, 2014 Synchronous Dataflow Programming
  36. 36. execute typical instruction fetch from L1 cache memory branch misprediction fetch from L2 cache memory Mutex lock/unlock fetch from main memory send 2K bytes over 1Gbps network read 1MB sequentially from memory fetch from new disk location (seek) read 1MB sequentially from disk send packet US to Europe and back 1/1,000,000,000 sec = 1 nanosec 0.5 nanosec 5 nanosec 7 nanosec 25 nanosec 100 nanosec 20,000 nanosec 250,000 nanosec 8,000,000 nanosec 20,000,000 nanosec 150 milliseconds = 150,000,000 nanosec­‐days.html PrismTech Copyright PrismTech, 2014 Time for Common Operations
  37. 37. Reactive Programming Libraries There is an increasing number of Reactive Programming framework. The Rx framework introduced by Erik Meijer is becoming a reference .NET Rx • Arguably the first framework that brought Reactive Programming to the masses. Available for the .NET platform RxJava ( • An Open Source implementation of Rx for the JVM contributed by NetFlix, it supports, Java, Scala, Kotlin, Clojure, Groovy and JRuby PrismTech Copyright PrismTech, 2014 !
  38. 38. RxJava Overview for composing asynchronous and event-based programs by using observable sequences. • RxJava allow you to compose streams of data declaratively while abstracting away low level concerns such as threading, synchronization, concurrent data structures, and non-blocking I/O • The two main concepts at the foundation of RxJava are Observables and Observers PrismTech Copyright PrismTech, 2014 • RxJava is a Java implementation of .NET Reactive Extensions — a library
  39. 39. RxJava Observables • RxJava Observables allow the composition of flows and sequences of • You can think of Observables as some kinds of asynchronous collections that push data to you • In essence an observable can be created from any synchronous or asynchronous stream of data. - - PrismTech you can create an observables that represents the data coming into a DDS Data Reader, the events from a Button, or even time you can also create an observable that represent an actual container Copyright PrismTech, 2014 asynchronous data
  40. 40. Observable Examples 0 1 2 3 4 5 t n val  evens:  Observable[Long]  =  ticks.filter(n  =>  n  %  2  ==  0) x 0 1 x 2 3 4 x 5 t n val  bufs:  Observable[Seq[Long]]  =  evens.buffer(2,1) 0 PrismTech 2 4 6 8 n t Copyright PrismTech, 2014 val  ticks:  Observable[Long]  =  Observable.interval(1  seconds)
  41. 41. • The Observer receives notification concerning new values, errors or the completion of the stream — e.g. no more data PrismTech trait  Observer[-­‐T]  {    def  onNext(value:  T):  Unit    def  onError(error:  Throwable):  Unit    def  onCompleted():  Unit   } Copyright PrismTech, 2014 RxJava Observers
  42. 42. • Beside from the standard RxJava/RxScala Observable, each framework that wants to integrate with RxJava/RxScala has to provide its own factory methods to create observables • For DDS you have to use the DdsObservable defined as follows: object  DdsObservable  {      def  fromDataReaderData[T](dr:  DataReader[T]):  Observable[T]      def  fromDataReaderEvents[T](dr:  DataReader[T]):  Observable[ReaderEvent[T]]      //  more  methods  which  we’ll  ignore  for  the  time  being   } PrismTech Copyright PrismTech, 2014 DDS Observables
  43. 43. Coding Lab
  44. 44. Shapes Application Application to illustrate the benefits of using Reactive Programming with DDS • Three Topics - Circle, Square, Triangle • One Type: struct ShapeType { string color; Pierced shapes represent publications long x; long y; long shapesize; Spotted shapes represent subscriptions }; #pragma keylist ShapeType color PrismTech Copyright PrismTech, 2014 • We will use the omnipresent Shapes
  45. 45. • To appreciate the joys of Reactive lets look first at how to implement a Shapes Logger using traditional “callback” programming PrismTech Copyright PrismTech, 2014 Logging Shapes
  46. 46. Step 1: Implement Listener class  ShapesListener  extends  DataReaderListener[ShapeType]  {   !    def  onRequestedDeadlineMissed(p1:  RequestedDeadlineMissedEvent[ShapeType]):  Unit      def  onRequestedIncompatibleQos(p1:  RequestedIncompatibleQosEvent[ShapeType]):  Unit   !    def  onSampleRejected(p1:  SampleRejectedEvent[ShapeType]):  Unit   !    def  onLivelinessChanged(p1:  LivelinessChangedEvent[ShapeType]):  Unit   !    def  onDataAvailable(p1:  DataAvailableEvent[ShapeType]):  Unit   !    def  onSubscriptionMatched(p1:  SubscriptionMatchedEvent[ShapeType]):  Unit   !    def  onSampleLost(p1:  SampleLostEvent[ShapeType]):  Unit   } PrismTech Copyright PrismTech, 2014 !
  47. 47. Step 2: Read and Log Data • From the listener code you’d retrieve the data reader (dr) and do something like this: def  onDataAvailable(p1:  DataAvailableEvent[ShapeType]):  Unit  =  {  =>  println("("  +  s.color  +  ",  "  +  s.x  +  ",  "  +  s.y))     }   ! PrismTech Copyright PrismTech, 2014 • To keep things simple and more compact we’ll use the Scala binding
  48. 48. Molière Makes it Better write: dr  listen  {      case  DataAvailable(_)    =>  {          dr  read()  foreach  (s  =>  println("("  +  s.color  +  ",  "  +  s.x  +  ",  "  +  s.y))      }   } PrismTech Copyright PrismTech, 2014 • To be fair, if you use Moliere, the Scala API for OpenSplice you could simply
  49. 49. val  shapes  =  DdsObservable.fromDataReaderData(dr)   shapes.subscribe(s  =>            println("("  +  s.color  +  ",  "  +  s.x  +  ",  "  +  s.y)   ) PrismTech Copyright PrismTech, 2014 Reactive Logger
  50. 50. To show a bit more the expressive power of Reactive programming, let’s see how we can solve the following problems PrismTech Copyright PrismTech, 2014 “Shapes Problems”
  51. 51. • We want to show a triangle that positioned midway between every single pair of circle and square of the same colour PrismTech Copyright PrismTech, 2014 Problem 1: Shape in the Middle
  52. 52. Shapes Marble Diagrams t • Conceptually, we need to select on stream, look at it one element at the time and match it with the corresponding item in the other stream • Then, we can produce the average PrismTech Copyright PrismTech, 2014 t
  53. 53. Shapes Marble Diagrams x x t t PrismTech Copyright PrismTech, 2014 t
  54. 54. Shapes Marble Diagrams x x x x t t P.S.: The dropping may seem a bit counter-intuitive but it is in part due to how the DDS streams work PrismTech Copyright PrismTech, 2014 t
  55. 55.        val  circles  =  DdsObservable.fromDataReaderData  {              DataReader[ShapeType](Topic[ShapeType](circle))          }          val  squares  =  DdsObservable.fromDataReaderData  {              DataReader[ShapeType](Topic[ShapeType](square))          }          val  ttopic  =  Topic[ShapeType](triangle)          val  tdw  =  DataWriter[ShapeType](ttopic)   ! !        //  Compute  the  average  between  circle  and  square  of  matching  color  with  flatMap          val  triangles  =  circles.flatMap  {              c  =>  squares.dropWhile(_.color  !=  c.color).take(1).map  {                  s  =>  new  ShapeType(s.color,  (s.x  +  c.x)/2,  (s.y  +  c.y)/2,  (s.shapesize  +  c.shapesize)/4)              }          }   !        triangles.subscribe(tdw.write(_))   PrismTech Copyright PrismTech, 2014 Midpoint Triangle
  56. 56.            //  Compute  the  average  between  circle  and  square  of  matching  colour  with  flatMap          val  triangles  =  circles.flatMap  {              c  =>  squares.dropWhile(_.color  !=  c.color).take(1).map  {                  s  =>  new  ShapeType(s.color,  (s.x  +  c.x)/2,  (s.y  +  c.y)/2,  (s.shapesize  +  c.shapesize)/4)              }          }              //  Compute  the  average  between  circle  and  square  of  matching  colour  with  for  comprehension          val  triangles  =  for  {                c  <-­‐  circles;                s  <-­‐  squares.dropWhile(_.color  !=  c.color).take(1)          }  yield  new  ShapeType(s.color,  (s.x  +  c.x)/2,  (s.y  +  c.y)/2,  (s.shapesize  +  c.shapesize)/4)   PrismTech Copyright PrismTech, 2014 Using for-comprehension
  57. 57. Summing Up
  58. 58. Concluding Remarks • Many of the requirements of traditional Reactive Systems are applicable to • Stream Processing Architecture and Reactive Programming can greatly help in the design and implementation of Reactive Systems that are efficient, responsive, resilient, scalable and highly available • As proven through years of deployments DDS provides the ideal infrastructure technology for Reactive Systems • The combination of DDS with Reactive Programming makes for a very powerful platform! PrismTech Copyright PrismTech, 2014 many mainstream application domains
  59. 59. • OpenSplice DDS [] • RxJava [] (OpenSouce Rx Implementation from NetFlix) • Moliere [] (This includes Rx Binding for DDS) PrismTech Copyright PrismTech, 2014 References