High Performance Distributed Computing with DDS and Scala
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

High Performance Distributed Computing with DDS and Scala

on

  • 1,432 views

This presentation provides a DDS tutorial and an introduction to programming DDS with the Scala Programming Language.

This presentation provides a DDS tutorial and an introduction to programming DDS with the Scala Programming Language.

Statistics

Views

Total Views
1,432
Views on SlideShare
1,396
Embed Views
36

Actions

Likes
3
Downloads
64
Comments
0

9 Embeds 36

https://twitter.com 17
http://icorsaro.net 8
http://www.icorsaro.net 4
https://si0.twimg.com 2
https://twimg0-a.akamaihd.net 1
http://translate.googleusercontent.com 1
http://dds4u.blogspot.com.es 1
http://www.linkedin.com 1
https://www.linkedin.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

High Performance Distributed Computing with DDS and Scala Presentation Transcript

  • 1. High Performance Distributed Computing with DDS and ScalaOpenSplice DDS Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech angelo.corsaro@prismtech.com
  • 2. OpenSplice DDS Background
  • 3. Data Distribution Service For Real-Time Systems Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Introduced in 2004 to address the Data Distribution Real-Timeliness Scale challenges faced by a wide class of Defense and Aerospace ApplicationsOpenSplice DDS Key requirement for the Parallelism Determinism Throughput, Availability Scalability, Persistence, Security ☐ standard were to deliver very Systemic Signal Data Real-Time Information Near Real-Time Fault- Tolerant Information Complex Information high and predictable Processing Processing Management Processing Processing performance while scaling from embedded to ultra- Parallel Systems Distributed Systems large-scale deployments
  • 4. Data Distribution Service For Real-Time Systems Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Recommended by key Real-Timeliness Scale administration worldwide, e.g. DoD, MoD, EUROCAE, etc.OpenSplice DDS Widely adopted across Parallelism Determinism Throughput, Availability Scalability, Persistence, Security ☐ several different domains, Systemic Signal Data Real-Time Information Near Real-Time Fault- Tolerant Information Complex Information e.g., Automated Trading, Processing Processing Management Processing Processing Simulations, SCADA, Telemetry, etc. Parallel Systems Distributed Systems
  • 5. Core Standard Copyright  2011,  PrismTech  –    All  Rights  Reserved. Application ☐ DDS is nowadays a family Data Centric Publish Subscriber (DCPS) of standards Content Ownership Durability SubscriptionOpenSplice DDS ☐ The two standards at the Minimum Profile core are: ☐ DDS v1.2 -- API DDS Interoperability Wire Protocol - DDSI-RTPS ☐ DDSI v2.1 -- Wire Protocol UDP/IP Application
  • 6. Standards: What For? Copyright  2011,  PrismTech  –    All  Rights  Reserved. Common “Language” Data Model +OpenSplice DDS QoS Requirements Interoperability DDSI Wire Interoperability
  • 7. OpenSplice DDS DDS Application API Standard Standards: What For? Portability Copyright  2011,  PrismTech  –    All  Rights  Reserved.
  • 8. Defense and Aerospace Copyright  2011,  PrismTech  –    All  Rights  Reserved.OpenSplice DDS Integrated Modular Vetronics Training & Simulation Systems Naval Combat Systems Air Traffic Control & Management Unmanned Air Vehicles Aerospace Applications
  • 9. Commercial Applications Copyright  2011,  PrismTech  –    All  Rights  Reserved.OpenSplice DDS Agricultural Vehicle Systems Large Scale SCADA Systems Smart Cities Train Control Systems Complex Medical Devices High Frequency Auto-Trading
  • 10. OpenSplice DDS D D BaS ics
  • 11. Data Distribution Service For Real-Time Systems DDS provides a Topic-Based Publish/ Copyright  2011,  PrismTech  –    All  Rights  Reserved. Subscribe abstraction based on: Data Reader Data Writer ☐ Topics: data distribution subject’sOpenSplice DDS Data Reader Data TopicD Writer DataWriters: data producers TopicA ☐ Data TopicB Reader Data Writer ☐ DataReaders: data consumers TopicC ... Data Data Writer Reader DDS Global Data Space
  • 12. Data Distribution Service For Real-Time Systems Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ DataWriters and DataReaders are automatically and Data Reader dynamically matched by the Data Writer DDS Dynamic DiscoveryOpenSplice DDS Data Reader Data TopicD Writer TopicA ☐ A rich set of QoS allows to Data Reader TopicB control existential, temporal, Data Writer TopicC ... and spatial properties of data Data Data Writer Reader DDS Global Data Space
  • 13. OpenSplice DDS DDS Entities
  • 14. Domain Participant Domain (e.g. Domain 123) Domain Participant Copyright  2011,  PrismTech  –    All  Rights  Reserved. Topic Partition (e.g. “Telemetry”, “Shapes”, ) T1 Publisher T1 T3 Subscriber TxOpenSplice DDS Topic Instances/Samples Ty Tb Ta DataWrter DataReader Tc ☐ A DomainParticipant provides access to a DDS Domain ☐ DDS Domains are identified by natural numbers, such as 0, 1, 2, ... ☐ Each DDS Domain has associated some unique communication and discovery endpoints, automatically deduced by the Domain ID
  • 15. OpenSplice DDS Topics
  • 16. DDS Topics “Circle”, “Square”, “Triangle”, ... ☐ A Topic defines a class of streams, Copyright  2011,  PrismTech  –    All  Rights  Reserved. also called topic instances Name ☐ A Topic has associated a unique Topic name, a user defined extensible Typ SOpenSplice DDS type and a set of QoS policies DURABILITY, Qo e DEADLINE, ShapeType ☐ QoS Policies capture the Topic PRIORITY, … non-functional invariants struct ShapeType { ☐ Topics can be discovered or @Key string color; locally defined long x; long y; long shapesize; };
  • 17. Topic Instances ☐ Each unique key value Copyright  2011,  PrismTech  –    All  Rights  Reserved. Instances Instances identifies a unique stream of data color =”Green” DDS not only struct ShapeType { ☐ color =”red” @Key string color; TopicOpenSplice DDS long x; long y; demultiplexes “streams” color = “Blue” long shapesize;}; but provides also lifecycle information ☐ A DDS DataWriter can write multiple instances
  • 18. OpenSplice DDS Topic Types
  • 19. Topic Types Type System ☐ DDS features a structural type system and the sub-type Copyright  2011,  PrismTech  –    All  Rights  Reserved. relationship ( <: ) defined by this type system is used to guide subscription matching ☐ In general, a DataReader[X] will match a DataWriter[Y], whereOpenSplice DDS X and Y are DDS Topic types, iff Y <: X ☐ Annotations available to: ☐ Constraint the subtype relationship ☐ Control the set of attributes considered when performing the subtype test
  • 20. Type-Level Annotations ☐ The nature of a DDS Topic Type can be controlled through Copyright  2011,  PrismTech  –    All  Rights  Reserved. annotations applicable at a type or attribute level ☐ Specifically the @Extensibility annotation controls the allowed “extensibility”, in the sense of sub-typing for the given typeOpenSplice DDS ☐ Possible values are: ☐ Final: the type is sealed and can’t be changed ☐ Extensible: Only monotonic extensions are allowed, e.g. “adding attributes to the end” ☐ Mutable: Arbitrary extensions are allowed such as attribute re-ordering, removal, etc.
  • 21. Attribute-Level Annotations ☐ Attribute semantics can also be controlled via an extensible Copyright  2011,  PrismTech  –    All  Rights  Reserved. annotation system ☐ Some of the most commonly used annotations are: @Optional: This attribute is allowed to be “nil”OpenSplice DDS ☐ ☐ @Shared: This attribute behaves like a pointer ☐ @MustUnderstand: This attribute must be present in order to match subscriptions ☐ @ID: Ordinal value associated to the type ☐ @Key: This attribute is part of the topic key
  • 22. Publishers and SubscribersOpenSplice DDS
  • 23. Publishers / Subscribers Domain (e.g. Domain 123) Domain Participant Copyright  2011,  PrismTech  –    All  Rights  Reserved. Topic Partition (e.g. “Telemetry”, “Shapes”, ) T1 Publisher T1 T3 Subscriber TxOpenSplice DDS Topic Instances/Samples Ty Tb Ta DataWrter DataReader Tc Publishers and Subscribers in DDS play two roles: ☐ Coordinate actions across DataReaders/DataWriters ☐ e.g. Atomic writes across data-writers ☐ Control access to DDS partitions, a mechanism provided by DDS to further
  • 24. DDS Partitions Domain Partitions DDS Publishers and Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Subscribers can specify a M:0:0 M:0:1 M:0:2 list of regular expressions representing the partitionsOpenSplice DDS to join M:1:0 M:1:1 M:1:2 ☐ Actual partition can be any string, but some M:2:0 M:2:1 M:2:2 structure can be given in order mimic hierarchy
  • 25. DDS Partitions Matching Domain Partitions Copyright  2011,  PrismTech  –    All  Rights  Reserved. M:0:0 M:0:1 M:0:2 M:0:*OpenSplice DDS M:1:0 M:1:1 M:1:2 M:2:0 M:2:1 M:2:2
  • 26. DDS Partitions Matching Domain Partitions Copyright  2011,  PrismTech  –    All  Rights  Reserved. M:0:0 M:0:1 M:0:2OpenSplice DDS M:1:0 M:1:1 M:1:2 M:2:0 M:2:1 M:2:2 M:*:2
  • 27. DDS Partitions Matching Domain Partitions Copyright  2011,  PrismTech  –    All  Rights  Reserved. M:0:0 M:0:1 M:0:2OpenSplice DDS M:1:0 M:1:1 M:1:2 M:2:0 M:2:1 M:2:2 M:0:0, M:1:1, M:2:2
  • 28. DataReaders & DataWritersOpenSplice DDS
  • 29. DataReader / DataWriter Domain (e.g. Domain 123) Domain Participant Copyright  2011,  PrismTech  –    All  Rights  Reserved. Topic Partition (e.g. “Telemetry”, “Shapes”, ) T1 Publisher T1 T3 Subscriber TxOpenSplice DDS Topic Instances/Samples Ty Tb Ta DataWrter DataReader Tc ☐ DataWriters allow to write samples for a specific Topic with a given QoS ☐ DataReaders allow to read samples for a specific Topic with a given QoS ☐ The Samples written/read are those belonging to the partitions associated with the Publisher/Subscriber
  • 30. Subscription MatchingOpenSplice DDS
  • 31. Subscription Matching For a DataWriter and a DataReader to match the following Copyright  2011,  PrismTech  –    All  Rights  Reserved. conditions have to hold: ☐ The Topic Name has to be the identicalOpenSplice DDS ☐ Said X the Topic Type associated with the DataWriter and Y the Topic Type associated with the Data reader, then X <: Y (recall that annotations control the result subtype test...) ☐ The QoS with which the DataWriter produces data matches or exceeds the DataReader QoS
  • 32. Type Matching struct ShapeType3D { struct ShapeType3D { @Key @Key string color; string color; long x; long x; long y; long y; Copyright  2011,  PrismTech  –    All  Rights  Reserved. long shapesize; DW long shapesize; DR1 long z; long z; }; }; CircleOpenSplice DDS Circle ☐ As ShapeType3D <: ShapeType DW and DR2 match Circle struct ShapeType { DDS will take care of projecting @Key ☐ DR2 string color; ShapeType3D into ShapeType long x; long y; long shapesize; };
  • 33. Type Matching @Extensibility(MUTABLE_EXTENSIBILITY) @Extensibility(MUTABLE_EXTENSIBILITY) struct ShapeTypeC { struct ShapeTypeA { @Key @ID(0) @Key @ID(0) string color; string color; @ID(10) @MustUnderstand @ID(10) @MustUnderstand DW1 long x; DR1 Copyright  2011,  PrismTech  –    All  Rights  Reserved. long x; @ID(11) @MustUnderstand @ID(11) @MustUnderstand long y; }; long y; @ID(12) @Optional long z; @ID(20) @Optional ☐ In this example, as a long shapesize; }; Circle Circle consequence ofOpenSplice DDS annotations, DR1 Circle Circle and DR2 match @Extensibility(MUTABLE_EXTENSIBILITY) struct ShapeTypeB { DW1 and DW2 @Key @ID(0) @Extensibility(MUTABLE_EXTENSIBILITY) string color; struct ShapeTypeA { @ID(10) @MustUnderstand @Key @ID(0) long x; DR2 string color; @ID(11) @MustUnderstand DW2 @ID(10) @MustUnderstand long y; }; long x; @ID(11) @MustUnderstand long y; @ID(12) @Optional long z; @ID(20) @Optional long shapesize; };
  • 34. Anatomy of a DDS ApplicationOpenSplice DDS
  • 35. Stepping into ScalaOpenSplice DDS Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech angelo.corsaro@prismtech.com
  • 36. What is Scala Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Scala (pronounced Skah-lah) stands for “Scalable language” ☐ It is a language that carefully and creatively blends Object Oriented and Functional constructs into a statically typedOpenSplice DDS language with sophisticated type inference ☐ Scala targets the JVM and .NET runtime and is 100% compatible with Java
  • 37. Why Should You Care? Scala is simple to write, extremely compact and easy to Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ understand ☐ Scala is strongly typed with a structural type systemOpenSplice DDS ☐ Scala is an extensible language (many construct are build in the standard library) ☐ Scala makes it easy to design Domain Specific Language
  • 38. Case Study: Complex NumbersOpenSplice DDS
  • 39. Complex Numbers Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ To explore some of the nice features of Scala, let’s see how we might design a Complex number class ☐ What we expect to be able to do is all mathematical operationsOpenSplice DDS between complex numbers as well as scalar multiplications and division ☐ [(1+i2)+2*(3-i5)*(i4)]/(1+i3) ☐ ~(1+i2) [conjugate] ☐ !(3+i4) [Modulo]
  • 40. Constructor ☐ Scala allows to implicitly create constructors and attributes starting from Copyright  2011,  PrismTech  –    All  Rights  Reserved. a simple argument list associated with the class declarationOpenSplice DDS class Complex(val re: Float, val im: Float)
  • 41. In Java public class Complex { Copyright  2011,  PrismTech  –    All  Rights  Reserved. ! private final float re; ! private final float im; ! public Complex(float re, float im) { ! ! this.re = re;OpenSplice DDS ! ! this.im = im; ! } ! public Complex(float f) { ! ! this.re = f; ! ! this.im = 0; ! } ! public float re() { return re;} ! public float im() { return im;}
  • 42. Methods ☐ Everything in Scala is a method even operators Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Methods name can be symbols such as *, !, +, etc.OpenSplice DDS def + (c: Complex) : Complex = Complex(re + c.re, im + c.im) or, taking advantage of type inference.... def + (c: Complex) = Complex(re + c.re, im + c.im)
  • 43. In Java... Copyright  2011,  PrismTech  –    All  Rights  Reserved. public Complex add(Complex that) { return new Complex(this.re() + that.re(),OpenSplice DDS this.im() + that.im()); }
  • 44. As a result... val result = Complex(1,2) + Complex(2,3) Copyright  2011,  PrismTech  –    All  Rights  Reserved.OpenSplice DDS Complex c1 = new Complex(1, 2); Complex c2 = new Complex (3,4); Complex c3 = c1.add(c2); or... Complex c3 = (new Complex(1, 2).add(new Complex (3,4));
  • 45. Companion Object ☐ Scala does not have the concept of static methods/attributes ☐ On the other hand it provides built-in support for Singletons, Copyright  2011,  PrismTech  –    All  Rights  Reserved. which are specified with the “object” keyword as opposed to the “class” keywordOpenSplice DDS object Complex { The companion object, is def apply(real: Float, img: Float) = new Complex(real, img) the object associated with def apply(real: Float) = new Complex(real, 0) a class, which shares the same name and provides implicit def floatToReComplex (f: Float) = new ReComplex(f) typically helper methods implicit def intToReComplex(i : Int) = new ReComplex(i) }
  • 46. “apply” Magic ☐ When an instance of a class is followed by parentheses with a list of zero Copyright  2011,  PrismTech  –    All  Rights  Reserved. or more parameters, the compiler invokes the apply method for that instance ☐ This is true for an object with a defined apply method (such as a companion object), as well as an instance of a class that defines anOpenSplice DDS apply method val result = Complex(1,2) is the same as.... val result = Complex.apply(1,2)
  • 47. Negation and Scalar Multiplication Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ In order to design a Complex class that is well integrated in our type system we should be able to support the following cases: ☐ -(a+ib) ☐ c*(a+ib)OpenSplice DDS ☐ (a+ib)*c ☐ How can we supporting something like -(a+ib) and c*(a+ib)?
  • 48. Scala Unary Operators ☐ Scala allows to define unary operators for the following method Copyright  2011,  PrismTech  –    All  Rights  Reserved. identifiers +, -, !, ~ def unary_-() = Complex(-re, -im)OpenSplice DDS ! def unary_!() = Math.sqrt(re*re + im*im) ! def unary_~() = Complex(re, -im) as a result we can write: val result = -Complex(1,2) + ~Complex(2,3)
  • 49. Scala Implicit Conversions Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The expression: val c3 = 3*Complex(5, 7) ☐ Is equivalent to:OpenSplice DDS val c3 = 3.*(Complex(5, 7)) ☐ Yet, the method to multiply a Integer to a Complex is not present in the Scala Int class ☐ What can we do to make the trick?
  • 50. Scala Implicit Conversions Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Scala does not support Open Classes, thus allowing to add new methods to existing classes ☐ Yet Scala supports implicit conversions that can be used to achieve the same resultOpenSplice DDS ☐ Lets see how...
  • 51. Scala Implicit Conversion object Complex { Copyright  2011,  PrismTech  –    All  Rights  Reserved. implicit def floatToReComplex (f: Float) = new ReComplex(f) implicit def intToReComplex(i : Int) = new ReComplex(i) }OpenSplice DDS class ReComplex(re: Float) { def * (that: Complex) = Complex(re*that.re,re*that.im) }
  • 52. The Result is... val c3 = 3*Complex(5, 7) Copyright  2011,  PrismTech  –    All  Rights  Reserved. is converted automatically into: val c3 = ReComplex(3).*(Complex(5, 7))OpenSplice DDS
  • 53. Putting it all together case class Complex(val re: Float, val im: Float) { // Unary Operators Copyright  2011,  PrismTech  –    All  Rights  Reserved. def unary_-() = Complex(-re, -im) // Binary Operators def unary_!() = Math.sqrt(re*re + im*im) def + (c: Complex) = Complex(re + c.re, im + c.im) def unary_~() = Complex(re, -im) def - (c: Complex) = Complex(re - c.re, im - c.im) // Formatting def * (f: Float) = Complex(f*re, f*im) override def toString() : String = {OpenSplice DDS def * (c: Complex) = Complex((re*c.re) - (im*c.im), if (im > 0) re + "+i" + im ((re*c.im) + (im*c.re))) else if (im < 0) re + "-i" + (-im) else re.toString def / (c: Complex) = { val d = c.re*c.re + c.im*c.im } Complex(((re*c.re) + (im + c.im))/d, } ((im*c.re) - (re*c.im))/d ) }
  • 54. Functions, Closures and CurryingOpenSplice DDS
  • 55. Functions Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Scala has first-class functions ☐ Functions can be defined and called, but equally functions can be defined as unnamed literals and passed as valuesOpenSplice DDS def inc(x: Int) = x + 1 val vinc = (x: Int) => x+1 inc(5) vinc(5) Notice once again the uniform access principle
  • 56. Playing with Functions val list = List(1,2,3,4,5,6,7,8,9) Copyright  2011,  PrismTech  –    All  Rights  Reserved. val g5 = list.filter((x: Int) => x > 5) g5: List[Int] = List(6, 7, 8, 9)OpenSplice DDS Or with placeholder syntax val list = List(1,2,3,4,5,6,7,8,9) val g5 = list.filter(_ > 5) g5: List[Int] = List(6, 7, 8, 9)
  • 57. Closures Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Scala allows you to define functions which include free variables meaning variables whose value is not bound to the parameter list ☐ Free variable are resolved at runtime considering the closure of visible variableOpenSplice DDS ☐ Example: def mean(e : Array[Float]) : Float = { def mean(e : Array[Float]) : Float = { var sum = 0.0F var sum = 0.0F e.foreach((x: Int) => sum += x) e.foreach(sum += _) return sum/e.length return sum/e.length } }
  • 58. Currying ☐ Scala provides support for curried functions which are applied to Copyright  2011,  PrismTech  –    All  Rights  Reserved. multiple argument lists, instead of just one ☐ Currying is the mechanism Scala provides for introducing new control abstractionOpenSplice DDS def curriedSum(x: Int)(y: Int) = x + y curriedSum(1) { 3 +5 }
  • 59. OpenSplice DDS Traits
  • 60. Traits ☐ Scala supports single inheritance from classes but can mix-in multiple Copyright  2011,  PrismTech  –    All  Rights  Reserved. traits ☐ A trait is the unit of code reuse for Scala. It encapsulate methods and field definitions Traits usually expect a class to implement an abstract method, whichOpenSplice DDS ☐ constitutes the “narrow” interface that allows to implement a rich behaviour ☐ Traits are also very useful for dependency injection
  • 61. Ordered Complex Numbers Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Our complex numbers are not comparable ☐ Let’s assume that we wanted to make them comparable, and lets supposed that we define the total order as based on the module of the complex numberOpenSplice DDS ☐ How can we implement this behavior?
  • 62. Ordered Trait Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The Ordered[T] traits encapsulates the set of methods that allow to define a total ordering over a type ☐ All the behaviour is defined in terms of an abstract method, namely “compare”OpenSplice DDS ☐ Classes that mix-in this trait have to implement the “compare” method class Complex(val re: Float, val im: Float) extends Ordering[Complex] { def compare(x: Complex, y: Complex) = { if (x == y) 0 else if (!x > !y) 1 else -1 }
  • 63. Case Classes &OpenSplice DDS Pattern Matching
  • 64. Case Classes and Pattern Matching ☐ Case Classes and Pattern Matching are twin constructs that are pretty Copyright  2011,  PrismTech  –    All  Rights  Reserved. useful when dealing with tree-like recursive data structures ☐ These constructs allow to match patterns in an expression and reconstruct the object graph that makes it upOpenSplice DDS ☐ Lets see an example...
  • 65. Case Classes and Pattern abstract class Expr case class Var(name: String) extends Expr Copyright  2011,  PrismTech  –    All  Rights  Reserved. case class Number(num: Float) extends Expr case class UnOp(operator: String, arg: Expr) extends Expr case class BinOp(operator: String, left: Expr, right: Expr)OpenSplice DDS def simplifyExpr(expr: Expr) : Expr = expr match { case UnOp("-", UnOp("-", e)) => e case BinOp("+", e, Number("0")) => e case BinOp("*", e, Number("1")) => e case _ => expr }
  • 66. Type ParametrizationOpenSplice DDS
  • 67. Type Parametrization Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Scala provides support for type parametrization and makes it available for both classes as well as traits trait Queue[T] { def head: TOpenSplice DDS def tail: Queue[T] def append(x: T) : Queue[T] } ☐ Scala allows to annotate the parametrized type to control the resulting type variance
  • 68. Type Variance ☐ If S <: T is Queue[S] <: Queue[T]? Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ By default Scala makes generic types nonvariant. This behaviour can be changed using the following annotations: ☐ Queue[+T] indicates that the the sub-typing is covariant in the parameter TOpenSplice DDS ☐ Queue[-T] indicates that the the sub-typing is contravariant in the parameter T
  • 69. Anatomy of a DDS ApplicationOpenSplice DDS
  • 70. Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Publisher Topic SubscriberOpenSplice DDS DataWriter DataReader
  • 71. Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Session // Create a Topic Publisher Topic SubscriberOpenSplice DDS val topic = Topic[ShapeType](dp, “Circle”) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) DataWriter DataReader
  • 72. Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Session // Create a Topic Publisher Topic SubscriberOpenSplice DDS val topic = Topic[ShapeType](dp, “Circle”) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types DataWriter DataReader // Create a DataWriter/DataWriter val writer = DataWriter[ShapeType](pub, topic) Reader/Writer for val reader = DataReader[ShapeType](sub, topic) application defined Topic Types
  • 73. Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Session // Create a Topic Publisher Topic SubscriberOpenSplice DDS val topic = Topic[ShapeType](dp, “Circle”) // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types DataWriter DataReader // Write data val data = new ShapeType(“RED”, 131, 107, 75) writer write data Reader/Writer for // But you can also write like this... application defined writer ! data Topic Types // Read new data and print it on the screen (reader read) foreach (prinln)
  • 74. Data Reader/Writer CachesOpenSplice DDS
  • 75. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 76. Reader/Writer Caches write( ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 77. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Topic Instance Network
  • 78. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 79. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Topic Instance Topic Instance Network
  • 80. Reader/Writer Caches write( ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 81. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Writer History Network
  • 82. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 83. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Reader History Reader History Network
  • 84. Reader/Writer Caches write( ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 85. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 86. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 87. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
  • 88. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache QoS Policies ‣ History ‣ Destination Order Network ‣ Presentation ‣ Partition QoS Policies ‣ Time Based Filter ‣ Reliability ‣ Durability ‣ Ownership ‣ History ‣ Transport Priority ‣ Latency Budget ‣ Time Based Filter
  • 89. Writer Cache Properties ☐ An entry for each topic instance Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Each cache entry might contain one or more samples, the number of DataWriter samples is controlled by the History QoS PolicyOpenSplice DDS ... ☐ Depending on the setting of the Durability QoS Policy, the content of the cache may be made available DataWriter Cache to late joiners. The depth of the cache controls how many samples per instance will be delivered to late joiners
  • 90. Reader Cache Properties ☐ An entry for each topic instance Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Each cache entry might contain one or more samples, the number of samples is controlled by the History QoS Policy DataReader ☐ History.KeepLast(n) keeps the last n samplesOpenSplice DDS per instance ☐ History.KeepAll maintains all the samples, ... modulo resource constraints ☐ For a given instance: DataReader Cache ☐ Samples from the same writer are inserted in write-order ☐ Samples from multiple-writers are ordered using time-stamps (source or reception depending on configurable a QoS Policy)
  • 91. Eventual Consistency DDS caches provide eventual consistency semantics Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ ☐ This means that a read will see the effect of a preceding write eventuallyOpenSplice DDS ☐ Furthermore, given a data-writer that is currently matching N readers, we can think of DDS as providing eventual consistency with W=0 and R=1 ☐ W: the number of Acks expected in order to return from a write ☐ R: the number of sources from which a read access data
  • 92. Understanding HistoryOpenSplice DDS
  • 93. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 94. Reader/Writer Caches write( 1 ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 95. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 96. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache 1 Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 97. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 98. Reader/Writer Caches write( 2 ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 99. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 100. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache 2 Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 101. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 ...OpenSplice DDS 2 2 DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 102. Reader/Writer Caches write( 3 ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 ...OpenSplice DDS 2 2 DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 103. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 3 1 1 ...OpenSplice DDS 2 2 DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 104. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 3 1 1 ...OpenSplice DDS 2 2 DataWriter Cache DataReader Cache DataReader Cache 3 Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 105. Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 3 2 2 ...OpenSplice DDS 3 3 DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
  • 106. Caches or Streams?OpenSplice DDS
  • 107. Dynamic View of a Stream Copyright  2011,  PrismTech  –    All  Rights  Reserved.OpenSplice DDS ... Stream: Set of samples written over time for a given topic instance.
  • 108. Dynamic View of a Stream Assumptions: Reader History = KeepLast (n) WriterHistory = KeepLast (m) Copyright  2011,  PrismTech  –    All  Rights  Reserved. Writer HistoryOpenSplice DDS ... Stream: Set of samples written over time for a given topic instance.
  • 109. Dynamic View of a Stream Assumptions: Reader History = KeepLast (n) WriterHistory = KeepLast (m) Copyright  2011,  PrismTech  –    All  Rights  Reserved. Samples ‘on Writer History the wire’OpenSplice DDS ... Stream: Set of samples written over time for a given topic instance.
  • 110. Dynamic View of a Stream Assumptions: Reader History = KeepLast (n) WriterHistory = KeepLast (m) Copyright  2011,  PrismTech  –    All  Rights  Reserved. Samples ‘on Writer History the wire’ Reader HistoryOpenSplice DDS ... Stream: Set of samples written over time for a given topic instance.
  • 111. Dynamic View of a Stream Assumptions: Reader History = KeepLast (n) WriterHistory = KeepLast (m) Copyright  2011,  PrismTech  –    All  Rights  Reserved. Samples ‘on Writer History the wire’ Reader History ‘Past’ SamplesOpenSplice DDS ... Stream: Set of samples written over time for a given topic instance.
  • 112. Eventual View of a Stream Assumptions (Default Settings): Reader History = KeepLast (1) WriterHistory = KeepLast (1) Copyright  2011,  PrismTech  –    All  Rights  Reserved. Writer History Reader History ‘Past’ SamplesOpenSplice DDS Stream: Set of samples written over time for a given topic instance.
  • 113. Eventual View of a Stream Assumptions: Reader History = KeepLast (n) with n > 1 WriterHistory = KeepLast (1) Copyright  2011,  PrismTech  –    All  Rights  Reserved. Writer History Reader History ‘Past’ SamplesOpenSplice DDS Stream: Set of samples written over time for a given topic instance.
  • 114. Eventual View of a Stream Assumptions: Reader History = KeepLast (n) with n > 1 WriterHistory = KeepLast (m) with n > m > 1 Copyright  2011,  PrismTech  –    All  Rights  Reserved. Writer History Reader History ‘Past’ SamplesOpenSplice DDS Stream: Set of samples written over time for a given topic instance.
  • 115. Interacting with the DataReader CacheOpenSplice DDS
  • 116. Reading Data Samples ☐ Samples can be read from the Data Reader History Cache Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The action of reading a sample is non-destructive. Samples are not removed from the cacheOpenSplice DDS DataReader DataReader ... read ... DataReader Cache DataReader Cache
  • 117. Taking Data Samples ☐ Samples can be taken from the Data Reader History Cache Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The action of taking a sample is destructive. Samples are removed from the cacheOpenSplice DDS DataReader DataReader ... take ... DataReader Cache DataReader Cache
  • 118. Read vs. Take Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The read operation should always be access the latest know value for topics that represent distributed stateOpenSplice DDS ☐ The take operation should be used to get the last notification from a topic that represent an event
  • 119. Data SelectorsOpenSplice DDS
  • 120. Cherry Picking in DDS Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ DDS provides some very flexible mechanisms for selecting the data to be read:OpenSplice DDS ☐ Data Content ☐ Data Status ☐ These mechanisms are composable
  • 121. Content-Based Data SelectionOpenSplice DDS
  • 122. Filters and Queries Application ☐ DDS Filters allow to control what gets into a DataReader cache Copyright  2011,  PrismTech  –    All  Rights  Reserved. Query ☐ DDS Queries allow to control what gets out of a DataReader cache DataReaderOpenSplice DDS ☐ Filters are defined by means of ... ContentFilteredTopics ... ... ... ☐ Queries operate in conjunction with DataReader Cache read operations Filter ☐ Filters and Queries are expressed as SQL where clauses
  • 123. struct ShapeType { Filters @Key string color; long x; long y; [Scala API] }; long shapesize; /** Copyright  2011,  PrismTech  –    All  Rights  Reserved. * NOTE: The Scala API if not provided with DP/Sub/Pub assumes * default domains and default partition. **/ // Create a Topic val topic = Topic[ShapeType](“Circle”) // Define filter expression and parametersOpenSplice DDS val query = Query(“x < %0 AND y < %1”, List(“200”, “300”)) // Define content filtered topic val cftopic = ContentFilteredTopic[ShapeType](“Circle”, topic, query) // Create a DataReader for the content-filtered Topic val reader = DataReader[ShapeType](cftopic)
  • 124. struct ShapeType { Query @Key string color; long x; long y; [Scala API] }; long shapesize; Copyright  2011,  PrismTech  –    All  Rights  Reserved. // Define filter expression and parametersOpenSplice DDS val query = Query(“x < %0 AND y < %1”, List(“200”, “300”)) dr read (contentQuery = query)
  • 125. Instances ☐ DDS provides a very efficient way of reading data belonging to a Copyright  2011,  PrismTech  –    All  Rights  Reserved. specific Topic Instance ☐ Obviously, one could use queries to match the key’s value, but this is not as efficient as the special purpose instance selectorOpenSplice DDS val data = reader read(instanceHandle = handle)
  • 126. State-Based SelectionOpenSplice DDS
  • 127. Sample, Instance, and View State ☐ The samples included in the DataReader cache have associated Copyright  2011,  PrismTech  –    All  Rights  Reserved. some meta-information which, among other things, describes the status of the sample and its associated stream/instance ☐ The Sample State (READ, NOT_READ) allows to distinguish betweenOpenSplice DDS new samples and samples that have already been read ☐ The View State (NEW, NOT_NEW) allows to distinguish a new instance from an existing one ☐ The Intance State (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of the instance to which a sample belongs
  • 128. State Selector in Action Copyright  2011,  PrismTech  –    All  Rights  Reserved. Scala // Read only new samples val data = reader read // Read any samples from live instancesOpenSplice DDS val data = reader read(stateFilter = SampleSelector.AnyData)
  • 129. Putting all Together Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Selectors can be composed in a flexible and expressive mannerOpenSplice DDS reader read ( Scala instanceHandle = handle, filterState = SampleSelector.AllData filterContent = query )
  • 130. List Comprehension TextOpenSplice DDS
  • 131. List Comprehension in DDS Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ When using the SCALA API, the DDS readers expose data as LinearSeqOptimized thus making possible list comprehension to address quite a few taskOpenSplice DDS ☐ Example ☐ dr read foreach (println) ☐ dr read map (transformer) ☐ ...
  • 132. Examples Compute the average shape and write it as a Square sample: Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ val data = dr history val s0 = new ShapeType(“GRAY”, 0, 0 ,0)OpenSplice DDS dw ! scaleShape((data : s0) (sumShape), data.size) ☐ Filter samples: (dr read) filter (_.x <= 100 && _.y <= 200)
  • 133. OpenSplice DDS Communication Statuses Copyright  2011,  PrismTech  –    All  Rights  Reserved.
  • 134. OpenSplice DDS Liveliness Changed Status Copyright  2011,  PrismTech  –    All  Rights  Reserved.
  • 135. OpenSplice DDS QoS
  • 136. QoS Model ☐ QoS-Policies control local and end-to-end properties of DDS Copyright  2011,  PrismTech  –    All  Rights  Reserved. Type Matching entities QoS matching QoS QoS QoS QoS QoS QoS QoS ☐ Local properties controlled by Topic Name QoS are related resource usage Publisher Subscriber ... DataWriter writes Type reads DataReader ...OpenSplice DDS ... ☐ End-to-end properties DomainParticipant DataWriter writes Type reads DataReader DomainParticipant controlled by QoS are related Topic Name to temporal and spatial aspects QoS QoS QoS of data distribution ☐ Some QoS-Policies are matched based on a Request vs. Offered Model thus QoS-enforcement
  • 137. QoS Policies [T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant] QoS Policy Applicability RxO Modifiable Copyright  2011,  PrismTech  –    All  Rights  Reserved. USER_DATA DP, DR, DW N Y TOPIC_DATA T N Y Configuration GROUP_DATA P, S N Y DURABILITY T, DR, DW Y NOpenSplice DDS DURABILITY T, DW N N SERVICE Data Availability HISTORY T, DR, DW N N PRESENTATION P, S Y N RELIABILITY T, DR, DW Y N PARTITION P, S N Y Data Delivery DESTINATION T, DR, DW Y N ORDER LIFESPAN T, DW N Y
  • 138. QoS Policies [T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant] QoS Policy Applicability RxO Modifiable Copyright  2011,  PrismTech  –    All  Rights  Reserved. DEADLINE T, DR, DW Y Y LATENCY T, DR, DW Y Y BUDGET Temporal/ TRANSPORT T, DW N Y Importance PRIORITY CharacteristicsOpenSplice DDS TIME BASED DR N Y FILTER OWNERSHIP T, DR, DW Y N OWNERSHIP DW N Y Replication STRENGTH LIVELINESS T, DR, DW Y N Fault-Detection
  • 139. OpenSplice DDS Partition Data Delivery Reliability Presentation Data Delivery Order Destination Copyright  2011,  PrismTech  –    All  Rights  Reserved.
  • 140. Reliability QoS Policy QoS Policy Applicability RxO Modifiable RELIABILITY T, DR, DW Y N Copyright  2011,  PrismTech  –    All  Rights  Reserved. The Reliability Policy controls the level of guarantee offered by the DDS in delivering data to subscribers ☐ Reliable. In steady-state, and no data writer crashes, theOpenSplice DDS middleware guarantees that all samples in the DataWriter history will eventually be delivered to all the DataReader ☐ Best Effort. Indicates that it is acceptable to not retry propagation of any samples
  • 141. Partition QoS Policy QoS Policy Applicability RxO Modifiable ☐ The Partition QoS Policy can PARTITION P, S N Y Copyright  2011,  PrismTech  –    All  Rights  Reserved. be used as subjects for organizing the flows of data ☐ The Partition QoS Policy is Subscriber used to connect Publishers/ Publisher "tracks.kfo" "tracks.ufo"OpenSplice DDS Subscribers to a Partitions’ List which might also contain wildcards, e.g. Subscriber Publisher tracks.* ☐ Topics instances are published and subscribed Publisher Subscriber across one or more Partition Partitions
  • 142. Data Availability History Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataOpenSplice DDS Lifespan Durability Availability Ownership Ownership Strength
  • 143. Durability QoS Policy QoS Policy Applicability RxO Modifiable DURABILITY T, DR, DW Y N Copyright  2011,  PrismTech  –    All  Rights  Reserved. The DURABILITY QoS controls the data availability w.r.t. late joiners, specifically the DDS provides the following variants: ☐ Volatile. No need to keep data instances for late joining data readersOpenSplice DDS ☐ Transient Local. Data instance availability for late joining data reader is tied to the data writer availability ☐ Transient. Data instance availability outlives the data writer ☐ Persistent. Data instance availability outlives system restarts
  • 144. History QoS Policy QoS Policy Applicability RxO Modifiable HISTORY T, DR, DW N N Copyright  2011,  PrismTech  –    All  Rights  Reserved. For DataWriters, the HISTORY QoS policy controls the amount of data that can be made available to late joining DataReaders under TRANSIENT_LOCAL DurabilityOpenSplice DDS For DataReader, the HISTORY QoS policy controls how many samples will be kept on the reader cache ☐ Keep Last. DDS will keep the most recent “depth” samples of each instance of data identified by its key ☐ Keep All. The DDS keep all the samples of each instance of data identified by its key -- up to reaching some configurable resource limits
  • 145. Ownership QoS Policy QoS Policy Applicability RxO Modifiable OWNERSHIP T, DR, DW Y N STRENGTH DW N Y Copyright  2011,  PrismTech  –    All  Rights  Reserved. Availability of data producers can be controlled via two QoS Policies ☐ OWNERSHIP (SHARED vs. EXCLUSIVE)OpenSplice DDS ☐ OWNERSHIP STRENGTH ☐ Instances of exclusively owned Topics can be modified (are owned) by the higher strength writer ☐ Writer strength is used to coordinate replicated writers
  • 146. Temporal Properties Copyright  2011,  PrismTech  –    All  Rights  Reserved. TimeBasedFilter Deadline [Inbound]OpenSplice DDS Throughput LatencyBudget Latency [Outbound] TransportPriority
  • 147. Latency Budget QoS Policy QoS Policy Applicability RxO Modifiable LATENCY T, DR, DW Y Y ☐ The LATENCY_BUDGET QoS BUDGET Copyright  2011,  PrismTech  –    All  Rights  Reserved. policy specifies the maximum acceptable delay from the time the data is written until the dataOpenSplice DDS is inserted in the receivers application-cache Latency = T1+T2+T3 DataWriter T3 DataReader ☐ A non-zero latency-budget T1 allows a DDS Batching implementation to batch T2 samples and improve CPU/ Network utilization
  • 148. Deadline QoS Policy QoS Policy Applicability RxO Modifiable DEADLINE T, DR, DW Y Y Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The DEADLINE QoS policy allows to define the maximum inter-arrival time between data samples ☐ DataWriter indicates that the application commits to write a newOpenSplice DDS value at least once every deadline period ☐ DataReaders are notified by the DDS when the DEADLINE QoS contract is violated DataWriter Deadline Deadline Deadline Deadline Deadline DataReader Deadline Violation
  • 149. Transport Priority QoS Policy QoS Policy Applicability RxO Modifiable TRANSPORT T, DW N Y PRIORITY Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The TRANSPORT_PRIORITY QoS policy is a hint to the infrastructure asOpenSplice DDS to how to set the priority of the underlying transport used to send the data.
  • 150. Time-Based Filter QoS Policy QoS Policy Applicability RxO Modifiable TIME BASED DR N Y FILTER Copyright  2011,  PrismTech  –    All  Rights  Reserved. mit mit ☐ The Time Based Filter allows to control the throughput at which Latency = T1+T2+T3 data is received by a data readerOpenSplice DDS DataWriter T3 DataReader ☐ Samples produced more often T2 than the minimum inter-arrival time are not delivered to the data reader mit mit mit = minimum inter-arrival time produced sample delivered sample discarded sample
  • 151. Setting QoS Policies Copyright  2011,  PrismTech  –    All  Rights  Reserved. C++ // Setting Partition QoS-Policy on Publisher val pubQos = PublisherQos + Partition("FooPartition");OpenSplice DDS // Setting various QoS-Policy on a DataReader val drQos = DataReaderQos() + History.KeepLast(10) + Durability.Transient() val dr = DataReader[ShapeType](topic, sub, drQos);
  • 152. Let’s Experiment!OpenSplice DDS
  • 153. OpenSplice DDS
  • 154. Advanced TopicsOpenSplice DDS
  • 155. Advanced Topics Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Depending on time and attendees interest, I’ll be covering a set of advanced topics such as:OpenSplice DDS ☐ Distributed State and Events ☐ Advanced Distributed Algorithms with DDS, such as Leader Election, Mutual Exclusion, etc.
  • 156. Distributed Events vs.OpenSplice DDS Distributed State
  • 157. OpenSplice DDS State in DDS
  • 158. Distributed State with DDS Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The “public” state of the elements making the distributed system can easily be captured via topic definitions ☐ Representing state with topics is more a matter of discipline w.r.t. toOpenSplice DDS the QoS being used and the way in which it is accessed
  • 159. State’s DDS QoS Topics representing state should have the following QoS Settings Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ RELIABILITY = RELIABLE ☐ HISTORY = KEEP_LAST(1)OpenSplice DDS ☐ DURABILITY = (TRANSIENT |PERSISTENT) ☐ OWNERSHIP = EXCLUSIVE ☐ DESTINATION_ORDER = SOURCE_TIMESTAMP
  • 160. Soft-State’s DDS QoS Topics representing soft-state, meaning state that is periodically Copyright  2011,  PrismTech  –    All  Rights  Reserved. updated, should have the following QoS Settings ☐ RELIABILITY = BEST_EFFORTOpenSplice DDS ☐ HISTORY = KEEP_LAST(1) ☐ DURABILITY = VOLATILE ☐ OWNERSHIP = EXCLUSIVE ☐ DESTINATION_ORDER = SOURCE_TIMESTAMP
  • 161. Accessing State in DDS ☐ The DataReader.read operation should be used to access topics Copyright  2011,  PrismTech  –    All  Rights  Reserved. representing state ☐ This ensures that the last value for the state will be kept in DDS and will be readable again and againOpenSplice DDS ☐ The DataReader data should be accessed with the following flags: ☐ ANY_SAMPLE_STATE ☐ ALIVE_INSTANCE_STATE ☐ ANY_VIEW_STATE
  • 162. Example [1/3] A Robot Position in 2D is an example of state Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ ☐ Let’s assume that the Robot only update position when it moves ☐ Topic Type:OpenSplice DDS struct RobotPosition { @key long rid; long x; long y; };
  • 163. Example [2/3] ☐ The Topic and DataReader would be constructed as follows Copyright  2011,  PrismTech  –    All  Rights  Reserved. // Create Topic Qos val tQos =OpenSplice DDS TopicQos() <= KeepLastHistory(1) <= Reliable() <= TransientDurability() <= ExclusiveOwnership() <= SourceTimestamp(); // Create Topic val rpt = Topic[RobotPosition](“RobotPosition”,topicQos) // Create DataReader val rpdr = DataReader[RobotPosition](rpt, DataReaderQos(tqos))
  • 164. Example [3/3] ☐ Data can be read as follows Copyright  2011,  PrismTech  –    All  Rights  Reserved. // Read data val data = rpdr.read(ReadState.AllData)OpenSplice DDS // Or specific to Escalier val data = rpdr.history
  • 165. OpenSplice DDS Events in DDS
  • 166. Distributed Events with DDS ☐ Events raised by a distributed system can be easily captured via Copyright  2011,  PrismTech  –    All  Rights  Reserved. topic definitions ☐ Representing events with topics is more a matter of discipline w.r.t. to the QoS being used and the way in which it is accessedOpenSplice DDS ☐ Event topics are often keyless
  • 167. Events’ DDS QoS Events should have the following QoS Settings Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ RELIABILITY = RELIABLE ☐ HISTORY = KEEP_ALLOpenSplice DDS ☐ DURABILITY = VOLATILE ☐ OWNERSHIP = SHARED ☐ DESTINATION_ORDER = SOURCE_TIMESTAMP
  • 168. Events’ DDS QoS Events should have the following QoS Settings Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ RELIABILITY = RELIABLE ☐ HISTORY = KEEP_ALLOpenSplice DDS ☐ DURABILITY = VOLATILE ☐ OWNERSHIP = SHARED ☐ DESTINATION_ORDER = SOURCE_TIMESTAMP
  • 169. Accessing Events in DDS Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The DataReader.take operation should be used to access topics representing events ☐ This ensures that the DDS cache is always freed by available eventsOpenSplice DDS ☐ The DataReader data should be accessed with the following flags: ☐ NEW_SAMPLE_STATE ☐ ALIVE_INSTANCE_STATE ☐ ANY_VIEW_STATE
  • 170. Example [1/3] Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ A CollisionEvent could be raised by a Robot when it is colliding (or about to collide) with something ☐ Topic Type:OpenSplice DDS struct CollisionEvent { long detectingRobotId; long collidingRobotId; long xe; long ye; };
  • 171. Example [2/3] ☐ The Topic and DataReader would be constructed as follows Copyright  2011,  PrismTech  –    All  Rights  Reserved. // Create Topic Qos val tQos = TopicQos() <= KeepAll()OpenSplice DDS <= Reliable() <= VolatileDurability() <= SharedOwnership() <= SourceTimestamp(); // Create Topic val cet = Topic[CollisionEvent](“CollisionEvent”,topicQos) // Create DataReader val cedr = DataReader[CollisionEvent](cet, DataReaderQos(tqos))
  • 172. OpenSplice DDS ☐ Example // Take data [3/3] val data = cedr.take() Data can be read as follows Copyright  2011,  PrismTech  –    All  Rights  Reserved.
  • 173. Distributed MutexOpenSplice DDS
  • 174. Lamport’s Distributed Mutex ☐ A relatively simple Distributed Mutex Algorithm was proposed by Leslie Copyright  2011,  PrismTech  –    All  Rights  Reserved. Lamport as an example application of Lamport’s Logical Clocks ☐ The basic protocol (with Agrawala optimization) works as follows (sketched):OpenSplice DDS ☐ When a process needs to enter a critical section sends a MUTEX request by tagging it with its current logical clock ☐ The process obtains the Mutex only when he has received ACKs from all the other process in the group ☐ When process receives a Mutex requests he sends an ACK only if he has not an outstanding Mutex request timestamped with a smaller logical clock
  • 175. Mutex Abstraction ☐ A base class defines the Copyright  2011,  PrismTech  –    All  Rights  Reserved. Mutex Protocol abstract class Mutex  {   def acquire() ☐ The Mutex companion def release() uses dependency injection }OpenSplice DDS to decide which concrete mutex implementation to use
  • 176. Foundation Abstractions Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The mutual exclusion algorithm requires essentially: ☐ FIFO communication channels between group members ☐ Logical ClocksOpenSplice DDS ☐ MutexRequest and MutexAck Messages These needs, have now to be translated in terms of topic types, topics, readers/writers and QoS Settings
  • 177. Topic Types Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ For implementing the Mutual Exclusion Algorithm it is sufficient to define the following topic types:OpenSplice DDS struct TLogicalClock { long ts; long mid; }; #pragma keylist LogicalClock mid struct TAck { long amid; // acknowledged member-id LogicalClock ts; }; #pragma keylist TAck ts.mid
  • 178. Topics We need essentially two topics: Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ One topic for representing the Mutex Requests, and ☐ Another topic for representing AcksOpenSplice DDS This leads us to: ☐ Topic(name = MutexRequest, type = TLogicalClock, QoS = {Reliability.Reliable, History.KeepAll}) ☐ Topic(name = MutexAck, type = TAck, QoS = {Reliability.Reliable, History.KeepAll})
  • 179. Distinguishing Groups ☐ To distinguish between members belonging to different groups we Copyright  2011,  PrismTech  –    All  Rights  Reserved. introduce a group-id gid that is used to uniquely identify a group ☐ At a DDS-level, the gid is used to name the partition in which all the group related traffic will take placeOpenSplice DDS Partition associated to the group with gid=2 “2” “1” “3” DDS Domain
  • 180. Show me the Code! Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ All the algorithms presented were implemented using DDS and ScalaOpenSplice DDS ☐ Specifically we’ve used the OpenSplice Escalier language mapping for Scala ☐ The resulting library has been baptized “dada” (DDS Advanced Distributed Algorithms) and is available under LGPL-v3
  • 181. LCMutex ☐ The LCMutex is one of the possible Mutex protocol, implementing the Agrawala variation of the classical Lamport’s Algorithm Copyright  2011,  PrismTech  –    All  Rights  Reserved. class LCMutex(val mid: Int, val gid: Int, val n: Int)(implicit val logger: Logger) extends Mutex { private var group = Group(gid) private var ts = LogicalClock(0, mid)OpenSplice DDS private var receivedAcks = new AtomicLong(0) private var pendingRequests = new SynchronizedPriorityQueue[LogicalClock]() private var myRequest = LogicalClock.Infinite private val reqDW = DataWriter[TLogicalClock](LCMutex.groupPublisher(gid), LCMutex.mutexRequestTopic, LCMutex.dwQos) private val reqDR = DataReader[TLogicalClock](LCMutex.groupSubscriber(gid), LCMutex.mutexRequestTopic, LCMutex.drQos) private val ackDW = DataWriter[TAck](LCMutex.groupPublisher(gid), LCMutex.mutexAckTopic, LCMutex.dwQos) private val ackDR = DataReader[TAck](LCMutex.groupSubscriber(gid), LCMutex.mutexAckTopic, LCMutex.drQos) private val ackSemaphore = new Semaphore(0)
  • 182. LCMutex.acquire Copyright  2011,  PrismTech  –    All  Rights  Reserved. def acquire() { ts = ts.inc() myRequest = ts Notice that as the LCMutex reqDW ! myRequest is single-threaded we can’t ackSemaphore.acquire() issue concurrent acquire.OpenSplice DDS }
  • 183. LCMutex.release Copyright  2011,  PrismTech  –    All  Rights  Reserved. def release() { myRequest = LogicalClock.Infinite Notice that as the LCMutex (pendingRequests dequeueAll) foreach { req => is single-threaded we can’t ts = ts inc() issue a new request before ackDW ! new TAck(req.id, ts) } we release.OpenSplice DDS
  • 184. LCMutex.onACK ackDR.reactions += { case DataAvailable(dr) => { Copyright  2011,  PrismTech  –    All  Rights  Reserved. // Count only the ACK for us val acks = ((ackDR take) filter (_.amid == mid)) val k = acks.length // Set the local clock to the max (tsi, tsj) + 1 synchronized { val maxTs = math.max(ts.ts, (acks map (_.ts.ts)).max) + 1 ts = LogicalClock(maxTs, ts.id)OpenSplice DDS } val ra = receivedAcks.addAndGet(k) val groupSize = group.size // If received sufficient many ACKs we can enter our Mutex! if (ra == groupSize - 1) { receivedAcks.set(0) ackSemaphore.release() } } }
  • 185. LCMutex.onReq reqDR.reactions += { case DataAvailable(dr) => { val requests = (reqDR take) filter (_.mid != mid) Copyright  2011,  PrismTech  –    All  Rights  Reserved. synchronized { val maxTs = math.max((requests map (_.ts)).max, ts.ts) + 1 ts = LogicalClock(maxTs, ts.id) } requests foreach (r => { if (r < myRequest) { ts = ts inc()OpenSplice DDS val ack = new TAck(r.mid, ts) ackDW ! ack } else { (pendingRequests find (_ == r)).getOrElse( { pendingRequests.enqueue(r) r }) } }) } }
  • 186. Dealing with Faults...OpenSplice DDS
  • 187. How to deal with Faults? ☐ The algorithm presented here intentionally ignores failures to keep the presentation simple Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The failure of a single group member would violate progress ☐ It is not hard to extend the algorithm to deal with failures,OpenSplice DDS especially under the assumption of eventual synchrony ☐ If you want to learn more attend the following RTWS-2012 presentation this Thursday: Classical Distributed Algorithms with DDS Sara Tucci-Piergiovanni, Research Engineer, CEA LIST Angelo Corsaro, Chief Technology Officer, PrismTech
  • 188. Concluding RemarksOpenSplice DDS
  • 189. DDS Standard Ecosystem Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. 2012 API 2012 DDS RMI DDS RMI 2012 2010 2010 2012 ANSI C ISO C++ Java-5 ScalaOpenSplice DDS 2004 2010 2010 201x Security Security X-Types X-Types DDS 2004 Wire Protocol DDSI-RTPS network DDSI-RTPS 2006 2006
  • 190. Concluding Remarks ☐ The DDS provides a powerful and feature- rich topic-based publish/subscribe Copyright  2011,  PrismTech  –    All  Rights  Reserved. abstraction ☐ This technology is widely used in mission and business critical systems and it beingOpenSplice DDS swiftly adopted in data-centric/big-data systems ☐ Differently from what some people think, DDS is very simple to get-started with ☐ Very good Open Source implementation are available... Good Hacking!
  • 191. References ¥ DDS-based Advanced Distributed Algorithms Toolkit ¥ Open Source ¥ github.com/kydos/dada Copyright  2011,  PrismTech  –    All  Rights  Reserved. OpenSplice | DDS Escalier ¥ #1 OMG DDS Implementation ¥ Fastest growing JVM Language ¥ Scala API for OpenSplice DDS ¥ Open Source ¥ Open Source ¥ Open Source ¥ www.opensplice.org ¥ www.scala-lang.org ¥ github.com/kydos/escalierOpenSplice DDS [C++] [Java] DDS-PSM-Cxx 2010 ¥ Simple C++ API for DDS ¥ DDS-PSM-Java for OpenSplice DDS ¥ DDS-PSM-Cxx API Standard ¥ Open Source ¥ Open Source ¥ Open Source ¥ code.google.com/p/simd-cxx ¥ github.com/kydos/simd-java ¥ github.com/kydos/dds-psm-cxx
  • 192. :: Connect with Us :: ¥opensplice.com ¥forums.opensplice.org ¥@acorsaro ¥opensplice.org ¥opensplicedds@prismtech.com ¥@prismtechOpenSplice DDS ¥ crc@prismtech.com ¥sales@prismtech.com ¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro
  • 193. OpenSplice DDS