Extensible and Dynamic Topic Types For DDS

Extensible and Dynamic Topic Types for DDS MARS – San Antonio, TX – Sept., 2009 Rick Warren, RTI [email_address] document number: 2009-09-08

Introduction DDS Types: Assessment and Challenges

Good

Unmatched type safety => fewer defects

Unique type awareness => faster integration

Unique type awareness => higher performance

Bad

No standard API to define / access types at runtime

Hard to change types without rebuilding, redeploying

Hard to upgrade systems one subsystem at a time

Introduction Scope

Improve Portability

Clarify applicability of existing IDL type system

Codify existing DDS type system features, especially keys

Extend IDL to support new concepts

Improve Type System Expressiveness

Support maps ( i.e. associative collections)

Omit irrelevant fields , if desired

Make RTPS Discovery encapsulation first class, usable by any type/topic

Introduction Scope

Enable Agility

Support type evolution , subsystem by subsystem

Send and receive data using types unknown at compile time

Define and modify types programmatically

Discover remote type definitions

Get and set data member values by name

Simplify Integration

XML type definitions connect DDS to the XML-speaking world

Built-in types help users start using DDS quickly

Introduction Overview

Type System : abstract definition of what types can exist

Expressed as UML meta-model

Mostly familiar from IDL

IDL-compatible primitives

Strings and aliases (typedefs)

Arrays, sequences, and maps

Structures ( incl. aspects of valuetypes ) and unions

Enumerations and bit sets

Annotations

Type Representations : languages for describing types

IDL (with extensions)

XML and XSD

TypeCode

Introduction Overview

Data Representations : languages for describing samples

CDR (with extensions)

XML ( TBD )

Language Binding : programming APIs

“ Plain language”: extension of existing IDL-to- language bindings

Dynamic: reflective API for types and samples, defined in UML (PIM) and IDL (PSM)

Use by DDS : defines how DDS uses all of the above

Everything else is DDS-independent!

(so far, maybe only 90% independent)

IDL to Language Mapping:

Foo.h

Foo.c

FooTypeSupport.c

struct Foo {

char *name;

int ssn;

};

Foo f = {"hello", 2};

IDL to CDR: 00000006 68656C6C 6F000000 00000002 TypeSystem implicitly defined by the Type Representation

Introduction Technology Highlights

Dynamic Language Binding : dynamic types, dynamic data

XML Type Representations : XML and XSD

IDL Type Representation : …with extensions

CDR Data Representation : extensible binary data

Dynamic Types, Dynamic Data Dynamic Types: Overview

Summary : Define and/or modify type definitions at runtime

Informed by CORBA TypeCode

Interoperable with statically defined types

Small number of fundamental classes:

DynamicType : A programmatic type description

DynamicTypeFactory : Creates new types

DynamicTypeSupport : Registers types with DDS

Dynamic Types, Dynamic Data Dynamic Types: Example #1

Create the equivalent:

struct MyType {

long my_integer;

};

C++ code:

DynamicTypeFactory* factory =

DynamicTypeFactory::get_instance() ;

DynamicType* type = factory-> create_type (

TypeDescriptor (" MyType ", STRUCTURE_KIND ));

type-> add_member (

MemberDescriptor (

" my_integer ",

factory-> get_primitive_type (

INTEGER_KIND )));

Dynamic Types, Dynamic Data Dynamic Types: Example #2

Subscribe to a type you’ve never seen before

Java code:

DynamicType type =

myPubBuiltinTopicData.type ;

DynamicTypeSupport support =

DynamicDataTypeFactory.get_instance().

create_type_support ( type );

support. register_type (

myParticipant ,

myPubBuiltinTopicData.type_name );

// … init QoS based on built-in topic data

Topic topic = myParticipant.create_topic (

myPubBuiltinTopicData.name,

myPubBuiltinTopicData.type_name,

myTopicQos, myTopicListener);

…

Dynamic Types, Dynamic Data Dynamic Data: Overview

Summary : Introspect and modify samples at runtime

Informed by CORBA DynAny , JMS MapMessage , TIBCO Rendezvous self-describing messages

Small number of fundamental classes:

DynamicData : A sample of any (single) type

DynamicDataFactory : “Instantiates” a dynamic type

DynamicDataWriter : Writes samples of any (single) type, each represented by a DynamicData object

DynamicDataReader : Reads samples of any (single) type, each represented by a DynamicData object

Dynamic Types, Dynamic Data Dynamic Data: Names and IDs

DynamicData gets / sets member values in 2 ways:

By name : Obvious, human-readable approach

By ID number : Potentially faster, more efficient search

Example in C++:

DynamicData* sample = DynamicDataFactory::get_instance()-> create_data(my_type) ;

long my_value = 5;

sample-> set_int ( " my_integer ", // Specify name or ID; MEMBER_ID_ANY , // no need for both. my_value ); // <-- Value to set.

XML Type Representations XML Type Representations

#1: XSD

Based on IDL-to-WSDL mapping

Allows type sharing between DDS and web services

But…

Very verbose

Hard to read, hard to write, hard to parse

#2: XML

Straightforward mapping of IDL into XML

Suitable for embedding into other XML documents ( e.g. QoS profile files)

Easy to read, easy to write, easy to parse

XML Type Representations XML Type Representations: Examples

#1: XSD

<xsd:complexType name="T">

<xsd:sequence>

<xsd:element name="value" minOccurs="1" maxOccurs="1" type="char"/>

</xsd:sequence>

</xsd:complexType>

#2: XML

<struct name="T">

<member name="value" type="char"/>

</struct>

IDL Type Representation IDL Extension: Annotations

Summary : Bringing Java-style annotations to IDL

Don’t just solve today’s problem;

… also provide an extensible tool we can reuse tomorrow

Example :

@AnAnnotationAppliedToTheType

struct TheType

@AnAnnotationAppliedToTheMember

long the_member;

};

IDL Type Representation IDL Extension: Annotations

Define annotations:

@Annotation local interface MyAnnotation {

long value() default 32;

};

* Why “@Annotation” instead of “@interface”? Coming up…

Apply annotations:

struct MyStruct {

@MyAnnotation long my_field;

};

Code gen translates to Java annotations one-to-one

IDL Type Representation IDL Extension: Compatible Annotation Syntax

Assumption : Some vendors will implement this spec sooner than others

Assumption : Users share IDL files among products from multiple vendors

Problem : How to take advantage of new features w/o breaking old IDL compilers with new syntax?

Solution : Put annotations in single-line “comment,” e.g. :

You can do this: @Key long my_key_field;

… or equivalently: long my_key_field; //@Key

IDL Type Representation IDL Extension: Built-in Annotations

@Key

A member is part of its enclosing type’s key

@ID

Specify the ID of a particular type member

If omitted, ID is assigned automatically

@Optional

A type member may be omitted from data samples

@Extensible

A type might change in the future,

… so DDS will use a more-verbose, more-flexible encapsulation

CDR Data Representation Extensible Binary Data: Challenge

Problem : CDR encoded data looks like this:

10110010001011100001110101000111

… So how do I know if I’ve added / removed a field?

Possible Solution #1 : Include type definition with every sample

Uses too much bandwidth

Parsing is too slow

Possible Solution #2 : Have DataWriter publish its type definition; DataReader uses that to parse samples

Parsing is still too slow

CDR Data Representation Extensible Binary Data: Solution

Solution : Use member IDs w/ RTPS param CDR

+----+----+----+----+----+----+----+----+

| ID (16) | size (16) |

+----+----+----+----+----+----+----+----+

| data (size) ... |

+----+----+----+----+----+----+----+----+

Benefits :

Skip fields you don’t understand

Easily determine which fields are (not) present

Reorder fields

Brings RTPS simple discovery into type system mainstream

Cost : 4 bytes (+ alignment) per member

CDR Data Representation Extensible Binary Data: Details

Types marked as “extensible” encapsulated w/ parameterized CDR, all others w/ “plain” CDR

Extensible types can contain objects of non-extensible types and vice versa

100% compatible with existing types and serialization (…since built-in topic data are the only “extensible” types that exist today  )

Lots of members in your type? Data sizes > 64 KB?

+----+----+----+----+----+----+----+----+

| ID sentinel (16) | size = 8 (16) |

+----+----+----+----+----+----+----+----+

| extended ID (32) |

+----+----+----+----+----+----+----+----+

| extended size (32) |

+----+----+----+----+----+----+----+----+

| data (extended size) ... |

+----+----+----+----+----+----+----+----+

CDR Data Representation Extensible Binary Data: Type Versioning

Problem : How can types change over time – and at different times from subsystem to subsystem?

Possible Solution : Give types version number, keep track of past versions and who is using which version.

Too slow : must consult per-remote-writer definition to deserialize

Requires tracking lots of state – including every version of every type that’s ever existed ( e.g. for late joiners)

Requires network propagation of all type information: increased bandwidth use, possible security objections

Solution : Do nothing . Member IDs already provide compact per-member “version.”

Adding / changing a member? Define a new ID. Old applications can ignore it.

“ Removing” a member? Just stop sending it. Doesn’t invalidate existing persisted data.

Summary: Goals and Benefits

Portable, consistent type system

Standardized syntax for keys and other DDS concepts

General-purpose extensibility (annotations) reusable by future specifications, vendors, and users

Same features usable by user-defined types as built-in types

Dynamic access to type definitions

Introspect types for remote publications, subscriptions

Define and modify types at runtime

Publish and subscribe to dynamically defined types

Interoperate with statically defined types and across vendors

Simplified integration of DDS into other systems

XML and XSD type definitions to integrate w/ XML-centric systems

Built-in types for simplified programming, especially for new users

Fast : little or no performance overhead in steady state

Summary: Progress

Many more details than previous submission

Full specification of IDL extensions

Unification of CDR and parameterized CDR data representations

Concrete dynamic type, dynamic data APIs

Refinements to automatic type conversions

Some work remains

XML data representation

Refinements to XML type representation grammars

Refinements to endpoint matching rules

Targeted for completion by Dec. 2009 meeting

Extensible and Dynamic Topic Types For DDS

0 comments

Notes on slide 1

Favorites, Groups & Events