Building Reactive Applications with DDS

1. Building Reactive Applications with DDS Angelo Corsaro, PhD Chief Technology Oﬃcer OMG DDS SIG Co-Chair angelo.corsaro@prismtech.com PrismTech

2. • Reactive Systems were characteristic of “niche” applications domains such as Industrial, Military, Telco, Finance and Aerospace • To address the challenges imposed contemporary Internet Applications an increasing movement of architects and developers is recognising the importance of the techniques and technologies used for building Reactive Systems PrismTech Copyright PrismTech, 2014 Reactive Systems Go Mainstream

3. Integrated Modular Vetronics Air Trafﬁc Control & Management PrismTech Training & Simulation Systems Unmanned Air Vehicles Naval Combat Systems Aerospace Applications Copyright PrismTech, 2014 Defense and Aerospace

4. Autonomous Vehicle Systems Train Control / Management Systems PrismTech Smart Grids / Power Generation Smart Cities Connected Medical Devices Financial Services Copyright PrismTech, 2014 Industrial Internet

5. Reactive Manifesto • The techniques and Responsive Event-Driven Resilient • Recently, the Reactive Manifesto was defined to summarise the key traits of “Reactive Applications” Scalable http://www.reactivemanifesto.org/ PrismTech Copyright PrismTech, 2014 technologies traditionally used in a class of Reactive and Interactive Systems is gaining quite a bit of attention from the mainstream developers community

6. Reactive Manifesto • Event-Driven: modular, • Responsive: Timely react to stimuli • Resilient: Tolerate failures functionally and ideally temporally • Scalable: Gracefully scale up and down depending on load demands Event-Driven Resilient Scalable http://www.reactivemanifesto.org/ PrismTech Copyright PrismTech, 2014 Responsive asynchronous

7. Reactive Systems

8. Reactive Systems 1 • A Reactive System designates a permanently operating system that responds, reacts, to external stimuli at a speed determined by its environment, e.g. an industrial control system. [1] Nicolas Halbwachs, Synchronous Programming of Reactive Systems PrismTech Copyright PrismTech, 2014 Reactive Systems

9. Transformational Systems • A Transformational System designates a system that terminates after transforming its input into its output, e.g. a compiler PrismTech Copyright PrismTech, 2014 Transformational Systems

10. Interactive Systems • An Interactive System designates a systems that permanently communicates with its environment at its own speed, e.g. a computer game PrismTech Copyright PrismTech, 2014 Interactive Systems

11. Reactive vs. Interactive Systems Interactive • react to an environment which • react to an environment which • response time needs to be • response time needs to be • distributed concurrent systems • distributed concurrent systems • data-centric • data-centric “cannot wait”, e.g. reacting late has harsh consequences deterministic “doesn’t like to wait”, e.g. reacting late induces QoS degradation acceptable The term reactive systems is increasingly used to denote interactive systems with stringent response time, performance and scalability constraints. PrismTech Copyright PrismTech, 2014 Reactive

12. Reactive Systems Key Traits Concurrency • Aside from the concurrency between the system and its environment, it is natural to Strict Temporal Requirements • Reactive Systems have strict requirements with respect to the rate at which the need to process input as well as the response time for their outputs Determinism • The output of these systems are generally determined by the input and the occurrence time, e.g. no scheduling effects on the temporal properties of the output Reliability • Reactive Systems are often life/mission critical, as such reliability is of utmost importance PrismTech Copyright PrismTech, 2014 decompose these systems as an ensemble of concurrent components that cooperate to achieve the intended behaviour

13. Architectural Styles for Reactive Systems

14. Stream Processing • Stream Processing is a commonly adopted architectural style for building • Stream Processing is often applied in: - Interactive Systems - Signal Processing Systems - Functional Stream Programming - PrismTech Reactive Systems Data Flow Systems Copyright PrismTech, 2014 systems that operate over continuous (theoretically infinite) streams of data

15. Stream Processing for modelling systems that react to streams of data produced by the cyber-phisical world, such as data produced by sensors, the stock market, etc. • Stream Processing Systems operate in realtime over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc. PrismTech Copyright PrismTech, 2014 • Stream Processing Architectures are ideal

16. Stream Processing • Stream Processing Systems are typically modelled • Modules usually play one of the following roles: - - PrismTech Filter Sources: Injecting data into the System Filter Filter Filters/Actors: Performing some computation over sources Sinks: Consuming the data produced by the system Copyright PrismTech, 2014 as collection of concurrent modules communicating via typed data channels Source Stream Sink

17. Stream Processing with DDS

18. Streams in DDS infinite sequence of data samples of a given type T Filter • As such, in DDS, a stream can be naturally represented with a Topic • Example: - PrismTech Topic<PressureType> Topic<MarketData> Filter Filter Stream Copyright PrismTech, 2014 • A stream represents a potentially

19. Topic • A Topic defines a domain-wide information’s class Copyright PrismTech, 2014 • A Topic is defined by means of a (name, type, qos) tuple, where type: is the programming language type associated with the topic. Types are extensible and evolvable Name - qos: is a collection of policies that express the nonfunctional properties of this topic, e.g. reliability, persistence, etc. Topic Qo S - PrismTech name: identifies the topic within the domain e Typ -

20. •As explained in the previous slide a topic defines a class/type of information •Topics can be defined as Singleton or can have multiple Instances •Topic Instances are identified by means of the topic key •A Topic Key is identified by a tuple of attributes -- like in databases •Remarks: • A Singleton topic has a single domain-wide instance • A “regular” Topic can have as many instances as the number of different key values, e.g., if the key is an 8-bit character then the topic can have 256 different instances PrismTech Copyright PrismTech, 2014 Topic and Instances

21. • Sources in DDS are mapped to Filter DataWriters for a given Topic • Sinks in DDS are mapped to Filter Filter DataReaders for a given Topic Source PrismTech Copyright PrismTech, 2014 Source and Sinks in DDS Sink

22. Filters in DDS application business logic • Each Filter has one or more input stream and one or more output streams • Obviously, a Filter consumes data from an input stream through a DDS data reader and pushed data into an output stream through a DDS data writer PrismTech Filter Filter Filter Copyright PrismTech, 2014 • Filters implement bit and pieces of the

23. DDS Architectural Beneﬁts

24. Loose Coupling • Anonymous Communication => No Spatial Coupling • Non-Blocking and Asynchronous Interaction => No control or temporal coupling In DDS the only thing that matters is the kind of information a module is interested in consuming or producing — who is producing the data or when the data has been produced is not relevant PrismTech Copyright PrismTech, 2014 Stream Processing Architectures in general and DDS-based architecture in particular promote loose coupling

25. Location Transparency • Location Transparency is an important architectural property as it makes it possible to completely decouple the logical decomposition of a system from its actual physical deployment • DDS Automatic Discovery brings location transparency to the next level by enabling systems that are zero-conf, self-forming and self-healing Module Logical Deployment Unit Physical Deployment Unit Module Module PrismTech Module Module Copyright PrismTech, 2014 Module

26. Temporal Decoupling Volatile Durability decoupling by means of its Durability Policy • As such the data produced by a Source can be: - - - PrismTech Source t Sink t Sink t Volatile: available for those that are around at the time at production Transient: available as far as the system/source is running Durable: available forever Transient Durability Source t Sink t Sink t Copyright PrismTech, 2014 • DDS provides full control over temporal

27. Failure Detection traditional faults as well as performance failures • The Fault-Detection mechanism is Source TD Sink t controlled by means of the DDS Liveliness policy • Performance Failures can be detected using the Deadline Policy which allows to receive notification when data is not received within the expected delays Fault Notiﬁcation Source t Sink P P P Performance Failure Notiﬁcation PrismTech t Copyright PrismTech, 2014 • DDS provides mechanism for detecting

28. Fault-Masking mechanism that allow to replicate Sources and transparently switch over when a failure occurs • At any point in time the “active” source is the one with the highest strength. Where the strength is an integer parameter controller by the user PrismTech Source Source t Sink t Copyright PrismTech, 2014 • DDS provides a built-in fault-masking

29. Fault-Masking mechanism that allow to replicate Sources and transparently switch over when a failure occurs • At any point in time the “active” source is the one with the highest strength. Where the strength is an integer parameter controller by the user PrismTech Source Source t Sink t Copyright PrismTech, 2014 • DDS provides a built-in fault-masking

30. Performance - 15-20 usec Inter-Core Latency - 45-75 usec over GBps Ethernet • Throughput - PrismTech Up to 10+M msg/sec inter-core Up to 5M msg/sec inter node Copyright PrismTech, 2014 • Latency

31. Standard and Interoperable introduced as an OMG standard in 2006 to address Real-time and interoperable Data Sharing in Network Centric Systems • Key requirements for the standard were high and predictable performance, and scalability to ultra-large-scale deployments • DDS is recommended by key administration worldwide, e.g. DoD, MoD, EUROCAE, etc. and widely adopted across application domains, e.g., Automated Trading, Simulations, SCADA, Telemetry, etc. PrismTech Copyright PrismTech, 2014 • The Data Distribution Service (DDS) was

32. Escape from Callback-Hell

33. Dealing with Events environment • Stimuli coming from the cyber-physical world are often represented as events to be handled by the Reactive/Interactive System - e.g. think about the event from a GUI, or the data available event in DDS • Events are often handled through some form of callbacks, such as listeners, functors, etc. This stile of event handling leads to inversion of control PrismTech Copyright PrismTech, 2014 • Reactive and Interactive Systems continuously interact with their

34. The Callback Hell • The Inversion of Control induced by callback-based code in known to • As an mental exercise, think about the call-back style code you’d have to write to in Java for drawing when the mouse is being dragged. Although the logic is simple, it is far cry from having a “declarative style” • The problem of callback management, infamously known as Callback Hell, has become even more important due to the surge of Reactive Systems… • Can we escape the Callback Hell? PrismTech Copyright PrismTech, 2014 make applications brittle and harder to understand

35. Reactive Programming • Reactive Programming is a paradigm based on a relaxed form of • The Reactive Programming is built around the notion of continuous timevarying values and propagation of change • Reactive Programming facilitates the declarative development of non- blocking event driven applications — in essence you express what should be done and the runtime decides when to do it • Reactive Programming was popularised in the context of Functional Programming Languages by the seminal done by Elliot and Hudak in 1997 as part of Fran — a framework for composing richly interactive multimedia animations PrismTech Copyright PrismTech, 2014 Synchronous Dataflow Programming

36. execute typical instruction fetch from L1 cache memory branch misprediction fetch from L2 cache memory Mutex lock/unlock fetch from main memory send 2K bytes over 1Gbps network read 1MB sequentially from memory fetch from new disk location (seek) read 1MB sequentially from disk send packet US to Europe and back 1/1,000,000,000 sec = 1 nanosec 0.5 nanosec 5 nanosec 7 nanosec 25 nanosec 100 nanosec 20,000 nanosec 250,000 nanosec 8,000,000 nanosec 20,000,000 nanosec 150 milliseconds = 150,000,000 nanosec http://norvig.com/21-‐days.html PrismTech Copyright PrismTech, 2014 Time for Common Operations

37. Reactive Programming Libraries There is an increasing number of Reactive Programming framework. The Rx framework introduced by Erik Meijer is becoming a reference .NET Rx • Arguably the first framework that brought Reactive Programming to the masses. Available for the .NET platform RxJava (http://bit.ly/rxjava) • An Open Source implementation of Rx for the JVM contributed by NetFlix, it supports, Java, Scala, Kotlin, Clojure, Groovy and JRuby PrismTech Copyright PrismTech, 2014 !

38. RxJava Overview for composing asynchronous and event-based programs by using observable sequences. • RxJava allow you to compose streams of data declaratively while abstracting away low level concerns such as threading, synchronization, concurrent data structures, and non-blocking I/O • The two main concepts at the foundation of RxJava are Observables and Observers PrismTech Copyright PrismTech, 2014 • RxJava is a Java implementation of .NET Reactive Extensions — a library

39. RxJava Observables • RxJava Observables allow the composition of flows and sequences of • You can think of Observables as some kinds of asynchronous collections that push data to you • In essence an observable can be created from any synchronous or asynchronous stream of data. - - PrismTech you can create an observables that represents the data coming into a DDS Data Reader, the events from a Button, or even time you can also create an observable that represent an actual container Copyright PrismTech, 2014 asynchronous data

40. Observable Examples 0 1 2 3 4 5 t n val evens: Observable[Long] = ticks.filter(n => n % 2 == 0) x 0 1 x 2 3 4 x 5 t n val bufs: Observable[Seq[Long]] = evens.buffer(2,1) 0 PrismTech 2 4 6 8 n t Copyright PrismTech, 2014 val ticks: Observable[Long] = Observable.interval(1 seconds)

41. • The Observer receives notification concerning new values, errors or the completion of the stream — e.g. no more data PrismTech trait Observer[-‐T] { def onNext(value: T): Unit def onError(error: Throwable): Unit def onCompleted(): Unit } Copyright PrismTech, 2014 RxJava Observers

42. • Beside from the standard RxJava/RxScala Observable, each framework that wants to integrate with RxJava/RxScala has to provide its own factory methods to create observables • For DDS you have to use the DdsObservable defined as follows: object DdsObservable { def fromDataReaderData[T](dr: DataReader[T]): Observable[T] def fromDataReaderEvents[T](dr: DataReader[T]): Observable[ReaderEvent[T]] // more methods which we’ll ignore for the time being } PrismTech Copyright PrismTech, 2014 DDS Observables

43. Coding Lab

44. Shapes Application Application to illustrate the benefits of using Reactive Programming with DDS • Three Topics - Circle, Square, Triangle • One Type: struct ShapeType { string color; Pierced shapes represent publications long x; long y; long shapesize; Spotted shapes represent subscriptions }; #pragma keylist ShapeType color PrismTech Copyright PrismTech, 2014 • We will use the omnipresent Shapes

45. • To appreciate the joys of Reactive lets look first at how to implement a Shapes Logger using traditional “callback” programming PrismTech Copyright PrismTech, 2014 Logging Shapes

46. Step 1: Implement Listener class ShapesListener extends DataReaderListener[ShapeType] { ! def onRequestedDeadlineMissed(p1: RequestedDeadlineMissedEvent[ShapeType]): Unit def onRequestedIncompatibleQos(p1: RequestedIncompatibleQosEvent[ShapeType]): Unit ! def onSampleRejected(p1: SampleRejectedEvent[ShapeType]): Unit ! def onLivelinessChanged(p1: LivelinessChangedEvent[ShapeType]): Unit ! def onDataAvailable(p1: DataAvailableEvent[ShapeType]): Unit ! def onSubscriptionMatched(p1: SubscriptionMatchedEvent[ShapeType]): Unit ! def onSampleLost(p1: SampleLostEvent[ShapeType]): Unit } PrismTech Copyright PrismTech, 2014 !

47. Step 2: Read and Log Data • From the listener code you’d retrieve the data reader (dr) and do something like this: def onDataAvailable(p1: DataAvailableEvent[ShapeType]): Unit = { dr.read.foreach(s => println("(" + s.color + ", " + s.x + ", " + s.y)) } ! PrismTech Copyright PrismTech, 2014 • To keep things simple and more compact we’ll use the Scala binding

48. Molière Makes it Better write: dr listen { case DataAvailable(_) => { dr read() foreach (s => println("(" + s.color + ", " + s.x + ", " + s.y)) } } PrismTech Copyright PrismTech, 2014 • To be fair, if you use Moliere, the Scala API for OpenSplice you could simply

49. val shapes = DdsObservable.fromDataReaderData(dr) shapes.subscribe(s => println("(" + s.color + ", " + s.x + ", " + s.y) ) PrismTech Copyright PrismTech, 2014 Reactive Logger

50. To show a bit more the expressive power of Reactive programming, let’s see how we can solve the following problems PrismTech Copyright PrismTech, 2014 “Shapes Problems”

51. • We want to show a triangle that positioned midway between every single pair of circle and square of the same colour PrismTech Copyright PrismTech, 2014 Problem 1: Shape in the Middle

52. Shapes Marble Diagrams t • Conceptually, we need to select on stream, look at it one element at the time and match it with the corresponding item in the other stream • Then, we can produce the average PrismTech Copyright PrismTech, 2014 t

53. Shapes Marble Diagrams x x t t PrismTech Copyright PrismTech, 2014 t

54. Shapes Marble Diagrams x x x x t t P.S.: The dropping may seem a bit counter-intuitive but it is in part due to how the DDS streams work PrismTech Copyright PrismTech, 2014 t

55. val circles = DdsObservable.fromDataReaderData { DataReader[ShapeType](Topic[ShapeType](circle)) } val squares = DdsObservable.fromDataReaderData { DataReader[ShapeType](Topic[ShapeType](square)) } val ttopic = Topic[ShapeType](triangle) val tdw = DataWriter[ShapeType](ttopic) ! ! // Compute the average between circle and square of matching color with flatMap val triangles = circles.flatMap { c => squares.dropWhile(_.color != c.color).take(1).map { s => new ShapeType(s.color, (s.x + c.x)/2, (s.y + c.y)/2, (s.shapesize + c.shapesize)/4) } } ! triangles.subscribe(tdw.write(_)) PrismTech Copyright PrismTech, 2014 Midpoint Triangle

56. // Compute the average between circle and square of matching colour with flatMap val triangles = circles.flatMap { c => squares.dropWhile(_.color != c.color).take(1).map { s => new ShapeType(s.color, (s.x + c.x)/2, (s.y + c.y)/2, (s.shapesize + c.shapesize)/4) } } // Compute the average between circle and square of matching colour with for comprehension val triangles = for { c <-‐ circles; s <-‐ squares.dropWhile(_.color != c.color).take(1) } yield new ShapeType(s.color, (s.x + c.x)/2, (s.y + c.y)/2, (s.shapesize + c.shapesize)/4) PrismTech Copyright PrismTech, 2014 Using for-comprehension

57. Summing Up

58. Concluding Remarks • Many of the requirements of traditional Reactive Systems are applicable to • Stream Processing Architecture and Reactive Programming can greatly help in the design and implementation of Reactive Systems that are efficient, responsive, resilient, scalable and highly available • As proven through years of deployments DDS provides the ideal infrastructure technology for Reactive Systems • The combination of DDS with Reactive Programming makes for a very powerful platform! PrismTech Copyright PrismTech, 2014 many mainstream application domains

59. • OpenSplice DDS [http://prismtech.com] • RxJava [http://bit.ly/rxjava] (OpenSouce Rx Implementation from NetFlix) • Moliere [http://bit.ly/nuvo-moliere] (This includes Rx Binding for DDS) PrismTech Copyright PrismTech, 2014 References

Building Reactive Applications with DDS

You are reading a preview.

Check these out next

Building Reactive Applications with DDS

More Related Content

Slideshows for you (20)

Viewers also liked (20)

Similar to Building Reactive Applications with DDS (20)

More from Angelo Corsaro (20)

Recently uploaded (20)

Building Reactive Applications with DDS