Extensible and Dynamic
Topic Types for DDS
MARS – Jacksonville, FL – March 2010
Presenter: Rick Warren, RTI
Submitting POC...
Structure of this Presentation
Overview of specification and process
Specification contents
 Very detailed discussion las...
DDS Types: Assessment and Challenges
Good
Unmatched type safety
=> fewer defects
Unique type awareness
=> faster integrati...
Where We Are
RFP issued: Jun. 2008
Initial submission: Nov 2008
Revised submission extended: Feb. 2009
Revised submission ...
Design Principles
Don’t break anybody.
 Many people depend on DDS
 Don’t modify existing APIs
 Don’t modify existing co...
Overview
Copyright © 2010 RTI - All rights Reserved 6
Type System
Type
Representation
Language
Binding
Data
Representation
Overview: Example
Copyright © 2010 RTI - All rights Reserved 7
Type
Representation
Language
Binding
Data
Representation
ID...
Conformance Changes
Feedback on previous submission: conformance
criteria, relationships to other specs unclear
New submis...
Type System
Extensible, Compatible
 6.5.1: (a) Type System as UML meta-model.
(b) Type substitutability rules.
 6.5.2: T...
Type System Overview
Copyright © 2010 RTI - All rights Reserved 10
class Type System
NamedElement
Type
+ nested: boolean {...
Type System Overview
Entirely familiar from IDL:
 Primitives
 Strings, narrow and wide
 Arrays and sequences
 Aliases ...
Type System Changes
Clearer, more-flexible type extensibility kinds
1. Previous kind (Boolean): “extensible”=true:
 Add/r...
Type Extensibility Example
@Extensibility(
EXTENSIBLE)
struct Coord2D {
long x;
long y;
};
@Extensibility(
EXTENSIBLE)
str...
Type Extensibility Example
@Extensibility(
FINAL)
struct Coord2D {
long x;
long y;
};
@Extensibility(
FINAL)
struct Coord3...
Type Representations
 IDL
 XML, XSD
 TypeObject
Copyright © 2010 RTI - All rights Reserved 15
Static Definition
 6.5.1...
IDL Type Representation Changes
Added optional pragmas to delimit IDL segments
using new extended syntax
#pragma dds_xtopi...
IDL Type Representation Changes
Use struct syntax for inheritance, not valuetype
struct MyStruct : MyBaseStruct { … };
 T...
IDL Type Representation Changes
Cleaner user-defined annotation syntax
 Previously used methods, like Java annotations:
@...
XML Type Representations (No Change)
XML Type Representation
 Terse XML syntax designed ground-up for this Type
System
 ...
TypeObject Changes
Type Representation suitable for DDS discovery
Renamed from “TypeCode”
 Name similarity to CORBA TypeC...
Data Representations
 Extended CDR
 XML
Copyright © 2010 RTI - All rights Reserved 21
CDR Data Representation (No Change)
Requirements:
 Continue supporting existing (compact) CDR used by user-defined
types ...
XML Data Representation (No Change)
Optional Requirement (6.6.1):
Dynamic data access based on XML or JSON
Given: XSD Type...
Language Bindings
 Plain Language Binding
 Dynamic Language Binding
Copyright © 2010 RTI - All rights Reserved 24
Plain Language Binding (No Change)
Defined by existing IDL-to-language mappings:
1. Start with IDL Representation of a typ...
Dynamic Language Binding (No Change)
Classes
 DynamicTypeFactory: Creates new types (6.5.20)
 DynamicType: Introspect ty...
Use by DDS
 Data Representation Compatibility
 Type Compatibility
 Built-in Types
Copyright © 2010 RTI - All rights Res...
Data Representation Compatibility
Reader and writer must agree
on which Data Representation to use
 6.5.14: Encapsulation...
Type Compatibility
If I write type “Foo” and you read type “Bar”, can
we communicate?
New submission clearer, more flexibl...
Type Compatibility Example: What I Want
struct Coord2D
{
long x;
long y;
};
struct Coord3D :
Coord2D
{
long z;
};
Copyrigh...
1. Type Assignability
Structural compatibility:
Can sample serialized according to one type be
deserialized according to a...
2. Type Signature
Do I intend the types to be compatible?
Lists of equivalent types, base types provided at
registration t...
3. Type Consistency Enforcement Policy
QoS policy on Topic, DataReader, DataWriter
governs how (1) and (2) take effect
 R...
Type Compatibility Example: How to Get It
1. Is Coord2D assignable from Coord3D? Yes.
2. Type Signature:
Coord3DTypeSuppor...
Built-in Types (No Change)
Goal: Improve out-of-box experience for new
users, those with simple needs
Background: Usabilit...
Summary
Copyright © 2010 RTI - All rights Reserved 36
AB Feedback
Incorporates feedback from 3 AB reviewers:
Define IDL pragma to delimit extensions: Done
Clarify relationships...
Summary
The proposal:
Satisfies all mandatory and optional requirements and
supports identified real-world use cases
 Wit...
Q & A
Copyright © 2010 RTI - All rights Reserved 39
Upcoming SlideShare
Loading in …5
×

Extensible and Dynamic Topic Types for DDS

2,158 views

Published on

Presentation to the OMG MARS Task Force from March of 2010 on the "Extensible and Dynamic Topic Types for DDS" (DDS-XTypes) specification. Following this presentation, the Task Force voted to recommend the adoption of this specification.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,158
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
41
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Lots of material in a short amount of time, so here’s the format:If I’m not clear, let me know right away.If you have a quick question, go ahead and ask itIf you have a longer question of discussion point, we’ll talk about it at the end.
  • DDS is unique among pub-sub systems in providing interoperable static type safety.Applications that use data already expect certain contents; by making expectations explicit, enable: type-aware integration (e.g. relational mapping) compact network representation.
  • Many systems are already committed to, deployed on DDS 1.2, RTPS 2 (such as, many international navies). Breaking compatibility with those specs makes integration really hard, costs a lot of time and money for users an vendors alike. Don’t do it.People choose DDS for its performance. Don’t break that either.
  • Already, DDS conformance and RTPS conformance are independent. This spec pertains to both programming APIs and network representation, so it makes the same distinction.
  • Remember why we decided to use parameterized encoding for discovery in the first place:We knew it would change in future versions.We knew vendors would add extensions.But we created a special enclave: built-in topic data types are extensible, but no one else.This spec generalizes the model that has proven itself over a decade.
  • Nothing very new here: IDL  WSDL let you do this before. Now it’s official.
  • C, C++, Java covers almost all DDS users.Ada is a tiny fraction.Scripting languages are covered by Web-Enabled.
  • This is one common use case. After we talk about the different pieces, we’ll see how to accomplish it.
  • “DECLARED” is the default for conforming implementations. Call get_default_qos and you will get this.When a conforming implementation talks to a non-conforming implementation, this policy will be missing, so according to the type defaulting rules, it will assume “EXACT_TYPE.”
  • Extensible and Dynamic Topic Types for DDS

    1. 1. Extensible and Dynamic Topic Types for DDS MARS – Jacksonville, FL – March 2010 Presenter: Rick Warren, RTI Submitting POCs:  Angelo Corsaro, PrismTech: angelo.corsaro@prismtech.com  Gerardo Pardo, RTI: gerardo.pardo@rti.com  Robert Poth, PrismTech: robert.poth@prismtech.com  Rick Warren, RTI: rick.warren@rti.com Supporting POCs:  Virginie Watine, Thales: virginie.watine@thalesgroup.com document number: 2010-03-02
    2. 2. Structure of this Presentation Overview of specification and process Specification contents  Very detailed discussion last time  This presentation focuses on what’s changed AB feedback Copyright © 2010 RTI - All rights Reserved 2
    3. 3. DDS Types: Assessment and Challenges Good Unmatched type safety => fewer defects Unique type awareness => faster integration Unique type awareness => higher performance Bad No standard API to define / access types at runtime Hard to change types without rebuilding, redeploying Hard to upgrade systems one subsystem at a time Copyright © 2010 RTI - All rights Reserved 3 Vendor-Specific Solutions OMG-Standard Solutions
    4. 4. Where We Are RFP issued: Jun. 2008 Initial submission: Nov 2008 Revised submission extended: Feb. 2009 Revised submission extended: Aug. 2009 Revised submission extended: Nov. 2009 Revised submission deadline: Feb. 2010  Submitters back on the same page  Responded to comments from 2 (now 3) AB reviewers  Discussed with Task Force in Long Beach meeting Copyright © 2010 RTI - All rights Reserved 4
    5. 5. Design Principles Don’t break anybody.  Many people depend on DDS  Don’t modify existing APIs  Don’t modify existing compliance points  Many people depend on RTPS/DDSI 2  Don’t modify existing protocols or formats  Don’t modify existing compliance points Keep it fast.  New capabilities must permit efficient implementations.  Don’t use them? Don’t pay for them. Keep it orthogonal. Type compatibility doesn’t depend on:  Types’ definition language(s)  Data representation(s)  Programming language(s)  QoS configuration Copyright © 2010 RTI - All rights Reserved 5
    6. 6. Overview Copyright © 2010 RTI - All rights Reserved 6 Type System Type Representation Language Binding Data Representation
    7. 7. Overview: Example Copyright © 2010 RTI - All rights Reserved 7 Type Representation Language Binding Data Representation IDL: Foo.idl struct Foo { string name; long ssn; }; IDL to Language Mapping: Foo.h Foo.c FooTypeSupport.c struct Foo { char *name; int ssn; }; Foo f = {"hello", 2}; IDL to CDR: 00000006 68656C6C 6F000000 00000002
    8. 8. Conformance Changes Feedback on previous submission: conformance criteria, relationships to other specs unclear New submission more precise, better organized:  Conformance (part II, section 3): 2 conformance levels correspond to DDS, RTPS  Programming language conformance: Type system, type representations, language bindings, DDS Minimum Profile  Network interoperability conformance: Type system, data representations, RTPS  Relations to other specs (part III, section 8)  Extensions to DDS:  “Programming language conformance” sections constitute new optional conformance level (“profile”)  No changes to existing DDS profiles  Extensions to IDL:  Normative to impl.’s of this spec only; no change to CORBA conformance Copyright © 2010 RTI - All rights Reserved 8
    9. 9. Type System Extensible, Compatible  6.5.1: (a) Type System as UML meta-model. (b) Type substitutability rules.  6.5.2: Type extensibility, mutability rules  6.5.3: Type compatibility in presence of change  6.5.17: Allow type to evolve without breaking interoperability  6.5.5: Support keys. How does extensibility apply? More Expressive  6.5.4: Map data type  6.5.6, 6.5.19: Sparse data: object contains only subset of fields defined by its type Copyright © 2010 RTI - All rights Reserved 9
    10. 10. Type System Overview Copyright © 2010 RTI - All rights Reserved 10 class Type System NamedElement Type + nested: boolean {readOnly} PrimitiveType «enumeration» TypeKind Collection Namespace Aggregation Namespace Enumeration NamedElement Namespace Module Namespace BitSet Alias ConstructedType «enumeration» ExtensibilityKind FINAL_EXTENSIBILITY {readOnly} EXTENSIBLE_EXTENSIBILITY {readOnly} MUTABLE_EXTENSIBILITY {readOnly} +/container 0..* {frozen} +/containedType * {addOnly} +extensibility_kind 1 {frozen} +element_type 1 {frozen} +kind 1 {frozen} +base_type 1
    11. 11. Type System Overview Entirely familiar from IDL:  Primitives  Strings, narrow and wide  Arrays and sequences  Aliases (typedefs)  Unions  Modules As in IDL, but extended:  Structures—including single inheritance (for substitutability: 6.5.1)  Enumerations—specify bit width and constant values New relative to IDL:  Maps—like std::map or java.util.Map; required by RFP (6.5.4), fundamental OO collection  Annotations—for extensibility (6.5.2, 6.5.3) Copyright © 2010 RTI - All rights Reserved 11 module Foo { struct Bar { long baz; }; typedef Bar Quux; }
    12. 12. Type System Changes Clearer, more-flexible type extensibility kinds 1. Previous kind (Boolean): “extensible”=true:  Add/remove/reorder type members  Renamed to “mutable” 2. Previous kind (Boolean): “extensible”=false:  Add to end, but don’t add elsewhere, remove, or reorder  Renamed to “extensible”  Now allow removing from end as well as adding to end 3. New kind “final”:  Don’t allow any modifications  Explicit annotations identify all levels in IDL  Un-annotated types presumed “extensible” (2)  Can be overridden by IDL compiler option Several concrete examples added Copyright © 2010 RTI - All rights Reserved 12
    13. 13. Type Extensibility Example @Extensibility( EXTENSIBLE) struct Coord2D { long x; long y; }; @Extensibility( EXTENSIBLE) struct Coord3D { long x; long y; long z; }; Copyright © 2010 RTI - All rights Reserved 13 Assignable
    14. 14. Type Extensibility Example @Extensibility( FINAL) struct Coord2D { long x; long y; }; @Extensibility( FINAL) struct Coord3D { long x; long y; long z; }; Copyright © 2010 RTI - All rights Reserved 14 Not Assignable
    15. 15. Type Representations  IDL  XML, XSD  TypeObject Copyright © 2010 RTI - All rights Reserved 15 Static Definition  6.5.11: Programming-language- independent type serialization format(s)  6.5.9: Represent types in IDL and XML  6.5.7: Support IDL subset used by DDS 1.2 Dynamic Definition  6.5.12: Type discovery
    16. 16. IDL Type Representation Changes Added optional pragmas to delimit IDL segments using new extended syntax #pragma dds_xtopics begin [<version_number>] // IDL definitions #pragma dds_xtopics end  Document conformance requirements for human readers  Help non-conforming compilers recognize problems sooner, give more meaningful errors  No particular errors or warnings mandated Copyright © 2010 RTI - All rights Reserved 16
    17. 17. IDL Type Representation Changes Use struct syntax for inheritance, not valuetype struct MyStruct : MyBaseStruct { … };  Type system doesn’t include all valuetype features, so syntax was confusing  Unclear whether valuetype could extend struct Copyright © 2010 RTI - All rights Reserved 17
    18. 18. IDL Type Representation Changes Cleaner user-defined annotation syntax  Previously used methods, like Java annotations: @Annotation local interface MyAnnotation { long my_annotation_member(); };  Now uses attributes: @Annotation local interface MyAnnotation { attribute long my_annotation_member; };  Still uses local interface (alternative: struct)  Emphasizes that no runtime state exists  Allows future support for multiple inheritance, if desired Copyright © 2010 RTI - All rights Reserved 18
    19. 19. XML Type Representations (No Change) XML Type Representation  Terse XML syntax designed ground-up for this Type System  Easy to read, plays nicely with auto-completing editors XSD Type Representation  Based on existing IDL-to-WSDL mapping specification  Extended for new Type System concepts from this spec Copyright © 2010 RTI - All rights Reserved 19
    20. 20. TypeObject Changes Type Representation suitable for DDS discovery Renamed from “TypeCode”  Name similarity to CORBA TypeCode deemed A Bad Thing: this spec doesn’t depend on that one  .Net also has “TypeCode,” something else entirely… Improved string sharing to make representations much more compact Copyright © 2010 RTI - All rights Reserved 20
    21. 21. Data Representations  Extended CDR  XML Copyright © 2010 RTI - All rights Reserved 21
    22. 22. CDR Data Representation (No Change) Requirements:  Continue supporting existing (compact) CDR used by user-defined types (6.5.16)  Continue supporting existing (extensible, parameterized) CDR used by discovery types (6.5.16)  Provide extensibility to user-defined types (6.5.2, 6.5.3, 6.5.17)— all Data Representations should have equivalent expressiveness Solution: Make existing extensible, parameterized CDR available to all  Same rules for every type, built-in or user-defined  100% compatible with existing types and serialization Parameterized CDR: structs & unions marked “mutable”  Each member == parameter  Member ID == parameter ID Compact CDR: everything else Copyright © 2010 RTI - All rights Reserved 22
    23. 23. XML Data Representation (No Change) Optional Requirement (6.6.1): Dynamic data access based on XML or JSON Given: XSD Type Representation Data object == XML document Validated by XSD Representation of its type SAX, DOM-based access follow naturally New RTPS encapsulation ID lets you use XML on the network Copyright © 2010 RTI - All rights Reserved 23 struct OriginalLandData { long x; long y; }; <OriginalLandData> <x>5</x> <y>42</y>; </OriginalLandData>
    24. 24. Language Bindings  Plain Language Binding  Dynamic Language Binding Copyright © 2010 RTI - All rights Reserved 24
    25. 25. Plain Language Binding (No Change) Defined by existing IDL-to-language mappings: 1. Start with IDL Representation of a type 2. Apply mapping to desired programming language Mappings expanded to cover new concepts  e.g. maps, annotations, optional members  For C, C++, Java as required by RFP (6.5.8) Copyright © 2010 RTI - All rights Reserved 25 // IDL struct OrigLandData { long x; long y; }; // Java public class OrigLandData { public int x; public int y; }
    26. 26. Dynamic Language Binding (No Change) Classes  DynamicTypeFactory: Creates new types (6.5.20)  DynamicType: Introspect type definitions (6.5.13)  Informed by CORBA TypeCode API  DynamicData: Introspect objects (6.5.18)  Informed by CORBA DynAny, JMS MapMessage, TIBCO Rendezvous self-describing messages  DynamicTypeSupport: Registers types with DDS (6.5.20)  DynamicDataWriter: Writes objects of any (single) type, each represented by a DynamicData object (6.5.21)  DynamicDataReader: Reads objects of any (single) type, each represented by a DynamicData object (6.5.21) Copyright © 2010 RTI - All rights Reserved 26
    27. 27. Use by DDS  Data Representation Compatibility  Type Compatibility  Built-in Types Copyright © 2010 RTI - All rights Reserved 27
    28. 28. Data Representation Compatibility Reader and writer must agree on which Data Representation to use  6.5.14: Encapsulation specification, negotiation  6.5.15: Is encapsulation request-offer?  6.6.2: Optional: Custom encapsulation API Largely unchanged from previous submission Solution: DataRepresentationQosPolicy  List of representation IDs (standard = XCDR, XML)  Applies to both readers and writers  Included in corresponding built-in topic data types  Request-Offer semantics  Writer offers single representation it will use  Reader requests a list of representations  Compatible iff writer’s representation is in reader’s list  Can describe proposed representations, future additions, and vendor extensions Copyright © 2010 RTI - All rights Reserved 28
    29. 29. Type Compatibility If I write type “Foo” and you read type “Bar”, can we communicate? New submission clearer, more flexible based on feedback on previous submission 3 parts: 1. Physical: type assignability 2. Intentional: type signature 3. Enforcement: QoS policy governs how above are applied (NEW in this submission) Copyright © 2010 RTI - All rights Reserved 29
    30. 30. Type Compatibility Example: What I Want struct Coord2D { long x; long y; }; struct Coord3D : Coord2D { long z; }; Copyright © 2010 RTI - All rights Reserved 30 Read Write Write Read
    31. 31. 1. Type Assignability Structural compatibility: Can sample serialized according to one type be deserialized according to another?  Reminder: Rules designed to require no writer-specific runtime checks at deserialization time Intrinsic property of types’ definitions  Governed in part by extensibility kind  No action required by application programmer Copyright © 2010 RTI - All rights Reserved 31
    32. 32. 2. Type Signature Do I intend the types to be compatible? Lists of equivalent types, base types provided at registration time Clarification to previous submission: both reader and writer declarations taken into account New since previous submission: lists can contain patterns, not just literal names Copyright © 2010 RTI - All rights Reserved 32
    33. 33. 3. Type Consistency Enforcement Policy QoS policy on Topic, DataReader, DataWriter governs how (1) and (2) take effect  RxO: reader and writer must have the same setting Copyright © 2010 RTI - All rights Reserved 33  EXACT_TYPE:  Types must also be equal (as if extensibility were “final”)  Registered names must be equal  Like DDS of today but with extra safety  Assumed during endpoint matching if remote policy is unspecified  EXACT_NAME:  Assignability assumed without inspection  Registered names must be equal  This is what DDS guarantees today.  DECLARED:  Types must be assignable  Type signatures must be compatible  Default for conformant implementations  ASSIGNABLE:  Types must be assignable  Type names, signatures ignored: talk to any compatible type
    34. 34. Type Compatibility Example: How to Get It 1. Is Coord2D assignable from Coord3D? Yes. 2. Type Signature: Coord3DTypeSupport.register_type( my_participant, "Coord3D : Coord2D") 3. Type Consistency Enforcement: DECLARED (default) Copyright © 2010 RTI - All rights Reserved 34 struct Coord2D { long x; long y; }; struct Coord3D : Coord2D { long z; };
    35. 35. Built-in Types (No Change) Goal: Improve out-of-box experience for new users, those with simple needs Background: Usability gap vs. competition  Other middlewares allow sending textual or binary data without explicit type definition, registration Response: Provide simple types built in  Pre-defined, pre-registered  String: single string payload  Bytes: octet sequence payload  KeyedString: string payload + string key  KeyedBytes: octet sequence payload + string key Copyright © 2010 RTI - All rights Reserved 35
    36. 36. Summary Copyright © 2010 RTI - All rights Reserved 36
    37. 37. AB Feedback Incorporates feedback from 3 AB reviewers: Define IDL pragma to delimit extensions: Done Clarify relationships to other specs: Done Provide more detail about modules and namespaces: Done Make annotation type definition syntax more natural: Done Standardize VerbatimText behavior for C, C++, Java languages: Done Assorted typos, wording changes, organizational suggestions: Done Copyright © 2010 RTI - All rights Reserved 37
    38. 38. Summary The proposal: Satisfies all mandatory and optional requirements and supports identified real-world use cases  Without sacrificing backwards compatibility  Without sacrificing performance Is easily extensible w/ new type, data representations to handle additional use cases Only small changes since previous submission Incorporates all AB feedback We will be asking for a vote to recommend adoption. Copyright © 2010 RTI - All rights Reserved 38
    39. 39. Q & A Copyright © 2010 RTI - All rights Reserved 39

    ×