• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Extensible and Efficient Data Centric Architectures
 

Extensible and Efficient Data Centric Architectures

on

  • 1,633 views

This presentation provides first an abstract definition of Data Centric Architecture. Then it shows the mechanism and techniques that can be used to architect Data Centric Architectures with DDS.

This presentation provides first an abstract definition of Data Centric Architecture. Then it shows the mechanism and techniques that can be used to architect Data Centric Architectures with DDS.

Statistics

Views

Total Views
1,633
Views on SlideShare
1,628
Embed Views
5

Actions

Likes
2
Downloads
49
Comments
2

3 Embeds 5

http://icorsaro.net 3
http://a0.twimg.com 1
http://opensplice.blogspot.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

12 of 2 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Hello Dan,

    OpenSplice currently partial support for the X-Types. In essence, you can already use annotations in the IDL, but not all of those are taken into account.

    That means, that if you are starting to use OpenSplice you can still design extensible data models. Once the X-Types will be fully implemented in OpenSplice then you'll have also the correct 'on-the-wire' behaviour.

    A+
    Are you sure you want to
    Your message goes here
    Processing…
  • Does OpenSplice support X-Types?
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Extensible and Efficient Data Centric Architectures Extensible and Efficient Data Centric Architectures Presentation Transcript

    • Extensible and Efficient Data Centric ArchitecturesOpenSplice DDS Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech angelo.corsaro@prismtech.com
    • Agenda Data-Centric Architectures (DCA) Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ ☐ DDS and DCAOpenSplice DDS ☐ Extensible and Efficient DCA with DDS ☐ Concluding Remarks
    • Data-Centric ArchitecturesOpenSplice DDS
    • Data-Centric Architecture (DCA) Abstract Definition A Data Centric Architecture consists of: Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Abstract Data Types: A, B, C, ... ☐ Arrows: f, g, h, ... where arrows satisfy the following properties:OpenSplice DDS Given f: A → B, g: B → C, and h: C → D then the following holds true: ∃ g ∘ f: A → C [Composition] ∃ 1A : A → A [Identity Arrow] h ∘ (g ∘ f) = (h ∘ g) ∘ f [Associativity] f ∘ 1A = f = 1B ∘ f [Unit] A Data Centric Architecture is a Category
    • Data-Centric Architecture (DCA) The abstract definition of a Data-Centric Architecture formally captures its key properties: Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ The Abstraction Barrier is defined in terms of Abstract Data TypesOpenSplice DDS ☐ The System Behaviour is specified in terms of transformations (functions or arrows) over these Abstract Data Types From this emerges a natural relationship between DCA and Functional Programming Languages
    • Example: Space Traveller [1/3] ☐ Suppose we want to create a distributed game in which we have Copyright  2012,  PrismTech  –    All  Rights  Reserved. (some) UFOs that are trying to destroy our spaceship ☐ UFO can be our allies/enemies and can be played by other players orOpenSplice DDS the computer ☐ Some UFOs, might just be space travellers and should not be destroyed ☐ The goal of the game is to destroy the UFOs that have been classified as threats
    • Example: Space Traveller [2/3] ☐ The DCA for this game could be described as follows Copyright  2011,  PrismTech  –    All  Rights  Reserved. (keeping it simple...) Data Types ArrowsOpenSplice DDS ☐ FlyingObject ☐ animate: Dynamics → Dynamics ☐ Dynamics ☐ classify: FlyingObject x Dynamics → Classification ☐ Classification ☐ collide: [FlyingObject x Dynamics] → [Collision] ☐ Collision ☐ pilot: IO → Dynamics x FlyingObject ☐ fire: IO → FlyingObject x Dynamics ☐ display: FlyingObject x Dynamics x Classification x Collision → IO
    • Example: Space Traveller [3/3] Copyright  2012,  PrismTech  –    All  Rights  Reserved. FlyingObject FlyingObject IO pilot Dynamics animate Dynamics classify Classification Dynamics DynamicsOpenSplice DDS FlyingObject FlyingObject [FlyingObject] Dynamics IO fire [Dynamics] collide [Collision] display IO Dynamics Classification Collision
    • Data-Centric vs. Service-Centric Data-Centric and Service-Centric architectures, such as SOA, Distributed Objects, etc., differ in several dimensions: Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Abstraction Barrier: ☐ Data-Centric architectures rely on data abstractions ☐ Service-Centric architectures rely on service abstractionsOpenSplice DDS ☐ Level of Coupling ☐ Data-Centric architectures rely solely on the shared knowledge of abstract data types (agree on the “what”) ☐ Service-Centric architectures rely on the shared knowledge of operational interfaces along with the knowledge of the service provider and of its existence (agree on “what”, “who”, and “when”)
    • DCA Benefits ☐ Loose Coupling Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Anonymicity (e.g. only things to be known are the data types and not who produces/consumes them) Composability (especially when combined with functionalOpenSplice DDS ☐ languages) ☐ Extensibility ☐ Ease of Integration ☐ Performance (easier to parallelize)
    • OpenSplice DDS DDS and DCA
    • DCAs with DDS Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ DDS provides first class support for DCAs ☐ DCAs Abstract Data Types are captured in DDS via TopicsOpenSplice DDS ☐ DDS does not provide primitive support for specifying DCAs Arrows. These are defined by the architect using her/his favourite formalism
    • DDS Topics “net.gva.VehiclePosition” ☐ A Topic defines a typed data Copyright  2012,  PrismTech  –    All  Rights  Reserved. stream Name ☐ A Topic has associated a data type and QoS Topic Typ SOpenSplice DDS Qo ☐ The Topic name, type and QoS DURABILITY, e DEADLINE, defines the key functional and PRIORITY, … non-functional invariants struct Position2D { ☐ Topics can be discovered or long long vid; //@Key x; locally defined }; long y;
    • Extensible and Efficient DCA with DDSOpenSplice DDS
    • Back to the Space Traveller ☐ In the next few slides, Copyright  2012,  PrismTech  –    All  Rights  Reserved. we’ll explore the steps required to design extensible and efficientOpenSplice DDS data models with DDS ☐ We’ll use the Space Traveller as our running example
    • Space Traveller Topic Types //@Nested //@Nested enum Classifier { struct Bounds { struct Vector { UNKNOWN, Copyright  2012,  PrismTech  –    All  Rights  Reserved. long width; long x; FRIEND, long height; long y; THREAT}; }; }; struct Classification { struct FlyingObject { struct Dynamics { long oid; // @key long oid; //@Key long oid; //@Key Classifier kind;OpenSplice DDS long kind; Vector pos; }; Bounds bounds; Vector speed; }; }; struct Collision { long oid; long coid; };
    • From 2D to 3D ☐ Our original data model was designed for a 2D world Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Let’s assume that we want to be able to run our game in both 3D as well as 2D. In addition we don’t want toOpenSplice DDS break interoperability between the 3D and the (older) 2D version ☐ Afterall the 2D visualization is simply a projection of the 3D visualization ☐ How can we do this?
    • 2D to 3D w/o X-Types ☐ If we don’t want to break interoperability with existing games, as we are not going Copyright  2012,  PrismTech  –    All  Rights  Reserved. to be able to upgrade them all, we can’t change the existing topic types ☐ As a result we need to define another topic type to capture the 3rd dimension. This way the older version of the game will not notice the difference, while the new version will be made aware of the additional topic-type providing the 3rdOpenSplice DDS dimension //@Nested struct Vector { long x; long y; }; struct DynamicsEx { long oid; //@Key struct Dynamics { long z; long oid; //@Key long dz; Vector pos; }; Vector speed; };
    • Extensibility w/o X-Types Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Extending and evolving a DDS data model w/o the use of X-Types introduces (1) fragmentation, (2) canOpenSplice DDS potentially introduce update anomalies (e.g. inconsistency), and (3) adds complexity for both the architect and the developer
    • X-Types Background Info Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ The DDS X-Types specification extends DDS with: ☐ A polymorphic structural type systemOpenSplice DDS ☐ A set of concrete syntaxes for expressing DDS Topic Types ☐ Extensible Type Encoding ☐ A Dynamic API for runtime creation and manipulation of Topic Types, DataReaders and Writers
    • Types Assignability ☐ The main mechanism provided by the X-Types specification Copyright  2012,  PrismTech  –    All  Rights  Reserved. to support interoperability among different types is the concept of “assignability” ☐ The rules of assignability depends on the kind of extensions/OpenSplice DDS modifications allowed for the type: ☐ FINAL => Types are identical ☐ EXTENSIBLE => Types are one the monotonic extension of the other ☐ MUTABLE => Types can be projected w/o violating attribute invariants
    • 2D to 3D with X-Types ☐ The first thing to observe from our current data model is that it does not Copyright  2012,  PrismTech  –    All  Rights  Reserved. provide any extensibility annotations ☐ This means that by default it is considered “Extensible”, or to be more precise, monotonically extensibleOpenSplice DDS //@Nested struct Vector { long x; long y; }; struct Dynamics { long oid; //@Key Vector pos; Vector speed; };
    • Extensible X-Types (EXT) Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Monotonically Extensible Types (EXT), as suggested by their name, imply extensions only at the “tail” of the type.OpenSplice DDS ☐ Can we go from 2D to 3D using Monotonic Extensible Types? ☐ Let’s give it a try ...
    • 2D to 3D with EXT? [1/3] ☐ Let’s add explicit extensibility annotations first, so that Copyright  2012,  PrismTech  –    All  Rights  Reserved. we don’t get confused later on //@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY) struct Vector {OpenSplice DDS long x; long y; }; //@Extensibility(EXTENSIBLE_EXTENSIBILITY) struct Dynamics { long oid; //@Key Vector pos; Vector speed; };
    • 2D to 3D with EXT? [2/3] ☐ Let’s now extend the Vector Type: Copyright  2012,  PrismTech  –    All  Rights  Reserved. //@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY) struct Vector { long x; long y;OpenSplice DDS long z; }; ☐ Everything is fine as this extension is “adding to the tail” of our type
    • 2D to 3D with EXT? [3/3] ☐ The resulting types are now: Copyright  2012,  PrismTech  –    All  Rights  Reserved. //@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY) struct Vector { long x; long y; long z; }; //@Extensibility(EXTENSIBLE_EXTENSIBILITY)OpenSplice DDS struct Dynamics { long oid; //@Key Vector pos; Vector speed; }; ☐ Wait a second... The monotonic-extension on the type Vector is leading to a non-monotonic-extension of the type Dynamics! ☐ We need a “more extensible” Dynamics Type!
    • Mutable X-Types (MXT) Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Mutable X-Types (MXT) provide the most genericOpenSplice DDS form of extensibility. Attributes can be added in any order, and they can equally be removed
    • 2D to 3D with MXT? [1/3] ☐ As our extension to the Vector type is monotonic, Copyright  2012,  PrismTech  –    All  Rights  Reserved. we can keep it as a EXTOpenSplice DDS //@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY) struct Vector { long x; long y; };
    • 2D to 3D with MXT? [2/3] ☐ The Dynamics type needs to be MXT so to allow non- Copyright  2012,  PrismTech  –    All  Rights  Reserved. monotonic extensions. //@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY) struct Vector { long x;OpenSplice DDS long y; long z; }; //@Extensibility(MUTABLE_EXTENSIBILITY) struct Dynamics { long oid; //@Key Vector pos; Vector speed; };
    • 2D to 3D with MXT? [3/3] ☐ The resulting type ensures that a DataReader of the old version can safely project the new types on the old types Copyright  2012,  PrismTech  –    All  Rights  Reserved. //@Nested @Extensibility(EXTENSIBLE_EXTENSIBILITY) struct Vector { long x;OpenSplice DDS long y; long z; }; //@Extensibility(MUTABLE_EXTENSIBILITY) struct Dynamics { long oid; //@Key Vector pos; Vector speed; }; ☐ Yet, there are still a few issues. Can you see them?
    • Open Issues ☐ The “legacy” DataWriter will not match DataReaders for the new type as Copyright  2012,  PrismTech  –    All  Rights  Reserved. the Vector type would have to be “widened” as opposed to “projected”. This means that the 3D version of the game won’t see the flying objects on the 2D game. While the 2D will see a projection of the 3D objectsOpenSplice DDS ☐ If we change the extensibility of the Vector type to MUTABLE we still have an issue. How can we figure out, from the 3D application, when a “z” is not available? We need this information to be able to assign a arbitrary flight level to 2D UFOs as well as being able to distinguish when the “z” value is really “0”
    • 2D to 3D with MXT Revised [1/2] ☐ To address the issues just noted, we can simply change the Vector extensibility type, and also tag the z attribute as Optional Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Optional attributes can have a special “nil” value that allow us to detect when the field was not set //@Nested @Extensibility(MUTABLE_EXTENSIBILITY) struct Vector { long x;OpenSplice DDS long y; long z; //@Optional }; //@Extensibility(MUTABLE_EXTENSIBILITY) struct Dynamics { long oid; //@Key Vector pos; Vector speed; }; ☐ As defined the types allow the level of interoperability required by our application. However, our type definition is a bit loose as we are not defining the strong invariants...
    • 2D to 3D with MXT Revised [2/2] ☐ For mutable types it is important to identify which attributes represent the strong invariants of the type. These are the attributes that should always be present to allow assignability. These invariants are described via the Copyright  2012,  PrismTech  –    All  Rights  Reserved. MustUnderstand annotation //@Nested @Extensibility(MUTABLE_EXTENSIBILITY) struct Vector { long x; //@MustUndertand @Id(1) long y; //@MustUndertand @Id(2)OpenSplice DDS long z; //@Optional @Id(3) }; //@Extensibility(MUTABLE_EXTENSIBILITY) struct Dynamics { long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Id(3) }; ☐ To allow attribute re-ordering/removal it is a good practice to explicitly assign attributes Ids, otherwise those will be deduced by the order of declaration thus impacting the extensibility of a type
    • Resulting Types //@Nested @Extensibility(MUTABLE_EXTENSIBILITY) //@Nested @Extensibility(MUTABLE_EXTENSIBILITY) struct Bounds { struct Vector { long width; //@MustUndertand @Id(1) long x; //@MustUndertand @Id(1) long height; //@MustUndertand @Id(2) Copyright  2012,  PrismTech  –    All  Rights  Reserved. long y; //@MustUndertand @Id(2) long depth; //@Optional @Id(3) long z; //@Optional @Id(3) }; }; //@Extensibility(MUTABLE_EXTENSIBILITY) //@Extensibility(MUTABLE_EXTENSIBILITY) struct Dynamics { struct FlyingObject { long oid; //@Key @Id(1) long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) long kind; //@MustUndertand @Id(2)OpenSplice DDS Vector speed; //@MustUndertand @Id(3) Bounds bounds; //@MustUndertand @Id(3) }; }; // @Extensibility(FINAL_EXTENSIBILITY) // @Extensibility(FINAL_EXTENSIBILITY) enum Classifier { enum Classifier { UNKNOWN,FRIEND,THREAT}; UNKNOWN,FRIEND,THREAT}; // @Extensibility(FINAL_EXTENSIBILITY) // @Extensibility(FINAL_EXTENSIBILITY) struct Classification { struct Classification { long oid; // @key long oid; // @key Classifier kind;}; Classifier kind;}; NOTE: Some Final types to demonstrate their use
    • Are we Done? Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Our game has become very popular and friends play with it across the internet However, some people are complaining that in someOpenSplice DDS ☐ cases we are consuming a bit too much bandwith, especially when playing on narrow-band links ☐ How can we fix this problem?
    • Copyright  2012,  PrismTech  –    All  Rights  Reserved. N’ouvre la bouche que lorque tu es sur que ce que tu vas dire est plus beau que le silence.OpenSplice DDS [Proverbe Arabe]
    • Is our Data Model too Chatty? ☐ If we look at the Dynamics type, it includes the position -- Copyright  2012,  PrismTech  –    All  Rights  Reserved. which will change regularly and relatively often -- as well as //@Nested @Extensibility(MUTABLE_EXTENSIBILITY) the speed vector -- which is struct Vector { long x; //@MustUndertand @Id(1) subject to change less oftenOpenSplice DDS long y; //@MustUndertand @Id(2) long z; //@Optional @Id(3) (especially for UFOs) }; //@Extensibility(MUTABLE_EXTENSIBILITY) ☐ As a consequence of mixing struct Dynamics { long oid; //@Key @Id(1) the same topic-type attributes Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Id(3) with different update rates, we }; are sending more data than we really should
    • Frequency Normalization ☐ To address this issue we can split the topic type into as many Copyright  2012,  PrismTech  –    All  Rights  Reserved. topic types as necessary to ensure that attributes are updated at similar rates //@Extensibility(MUTABLE_EXTENSIBILITY) //@Extensibility(MUTABLE_EXTENSIBILITY)OpenSplice DDS struct FOPosition { struct FOSpeed { long oid; //@Key @Id(1) long oid; //@Key @Id(1) Vector pos; //@MustUndertand @Id(2) Vector speed; //@MustUndertand @Id(2) }; }; ☐ Notice that splitting the topic can lead to temporal inconsistencies, thus either (1) these are acceptable to your application or (2) we use coherent updates, or (3) we add sequence numbers to correlate the corresponding positions/speeds
    • Delta Updates ☐ This splitting does not feel Copyright  2012,  PrismTech  –    All  Rights  Reserved. completely right, does it? //@Nested @Extensibility(MUTABLE_EXTENSIBILITY) struct Vector { ☐ It would be nice if the middleware long x; //@MustUndertand @Id(1) long y; //@MustUndertand @Id(2) could figure out what had long z; //@Optional @Id(3) }; changed and only send deltasOpenSplice DDS //@Extensibility(MUTABLE_EXTENSIBILITY) struct Dynamics { ☐ Delta-updates are not currently long oid; Vector pos; //@Key @Id(1) //@MustUndertand @Id(2) available in DDS, these will be Vector speed; //@MustUndertand @Id(3) }; included in future versions of the standard ☐ Is there a way we can emulate delta-updates?
    • Emulating Delta Updates ☐ The @Optional annotation Copyright  2012,  PrismTech  –    All  Rights  Reserved. can be used to emulate //@Nested @Extensibility(MUTABLE_EXTENSIBILITY) delta updates. struct Vector { long x; //@MustUndertand @Id(1) ☐ This works since optional long y; //@MustUndertand @Id(2) long z; //@Optional @Id(3) attributes set to nil are never };OpenSplice DDS sent on the wire //@Extensibility(MUTABLE_EXTENSIBILITY) struct Dynamics { long oid; //@Key @Id(1) ☐ This comes at a cost, as it Vector pos; //@MustUndertand @Id(2) requires some work -- ideally Vector speed; //@MustUndertand @Optional @Id(3) }; embedded into a layer sitting between the middleware and the application
    • Concluding RemarksOpenSplice DDS
    • DCA Benefits ☐ Loose Coupling Copyright  2012,  PrismTech  –    All  Rights  Reserved. ☐ Anonymicity (e.g. only things to be known are data types and not who produces/consumes them) Composability (especially when combined with functionalOpenSplice DDS ☐ languages) ☐ Extensibility ☐ Ease of Integration ☐ Performance (easier to parallelize)
    • My Advice on X-Types Copyright  2012,  PrismTech  –    All  Rights  Reserved. Start using the X-Types annotation system immediately! ☐ DDS vendors are still working on the implementions of this specification, yet comment-style annotations //@ will beOpenSplice DDS ignored by non-compliant IDL compilers
    • My Advice on X-Types When in doubt, make outer types Mutable X-Types. Inner types Copyright  2012,  PrismTech  –    All  Rights  Reserved. are usually better candidates for Extensible X-Types (yet keep in mind our example) ☐ The main trade-off between Mutable and Extensible X-TypesOpenSplice DDS lies in the space/time efficiency. The latter is more efficient than the former.
    • My Advice on X-Types Always identify @MustUnderstand attributes of Mutable X-Types. Copyright  2012,  PrismTech  –    All  Rights  Reserved. Then declare as @Optional whatever is not @MustUnderstand ☐ This will ensure that you’ll always be able to distinguishOpenSplice DDS between default constructor values and nil
    • My Advice on X-Types Copyright  2012,  PrismTech  –    All  Rights  Reserved. Always explicitly define @Id for the attributes of Mutable X-Types ☐ This will ensure you’ll be to handle attribute re-ordering and removalOpenSplice DDS
    • OpenSplice DDS Questions?