Beyond messaging

1,919 views

Published on

The OMG Data Distribution Service (DDS) has brought a disruptive wave of innovation into what is commonly referred to as “Messaging” technology. DDS raises the abstraction level from unstructured, or loosely structured, messages to sound data types that allow architects and application developers to deal with concepts close to their problem space as opposed to having to worry about “plumbing” -- as it happens as with many messaging technologies. As it is common for disruptive innovations to take some time to be digested, many people still see DDS as an ultra-fast real-time messaging -- yet this is only the tip of the iceberg.

This presentation will explain how Data Distribution is different from Messaging and as a result how DDS differs from messaging technologies such as JMS. The webcast will cover the architectural benefits brought by data centricity as well as how the features introduced by the recently adopted DDS specification on Dynamic and Extensible Topic Types will support the design of efficient, extensible and evolvable systems.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,919
On SlideShare
0
From Embeds
0
Number of Embeds
112
Actions
Shares
0
Downloads
79
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Beyond messaging

  1. 1. Angelo Corsaro, Ph.D. Chief Technology Officer PrismTech OMG DDS SIG Co-Chair angelo.corsaro@prismtech.com OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Beyond Messaging....
  2. 2. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ./messaging
  3. 3. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS What is Messaging about? ‣ Messaging is about distributing messages generated by a producer to its consumers ‣ Messaging technologies often supports P2P and Pub/Sub interactions ‣ The producer/consumer exchange are often mediated by brokers Topic Queue P2P Messaging Pub/Sub Messaging
  4. 4. ‣ A message is the mechanism provided to messaging applications for exchanging information ‣ A Message is usually composed by a header and an opaque body ‣ In other terms, a message is an untyped (or weakly typed) information holder OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS What is a Message? Header Body
  5. 5. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS An Example:AMQP Messages ‣ AMQP define a message as made by a header and a body ‣ The body is an opaque sequence of bytes ‣ Thus, data serialization is delegated at the application
  6. 6. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS An Example: JMS Messages ‣ JMS define a message as made by a header and a body ‣ The body can be of five different types ‣ Text ‣ Map ‣ Array of Bytes ‣ Stream of native types ‣ Serialized Object
  7. 7. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ISO/OSI view of Messaging ‣ Messaging operates at the Session layer but does deal with the presentation layer (e.g. data representation). ‣ Messaging applications have to deal directly with data presentation Messaging Messaging App
  8. 8. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Messaging in JMS ‣ The user data has to be represented in a format supported by the JMS message body ‣ Object serialization has some shortcomings: ‣ not interoperable with other programming languages ‣ Not the most time/space efficient way of encoding your data // Create a TextMessage TextMessage message = queueSession.createTextMessage(); // Create and Set the Message Body String msgText = "body"; message.setText(msgText); // Send the Message queueSender.send(message); Sending Message
  9. 9. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Messaging in JMS ‣ Application have to figure-out the actual type of the “Message” ‣ Data has to be extracted from the message ‣ Lot’s of potential for introducing errors that a sounder type-system could prevent // Receive the Message Message m = queueReceiver.receive(); // Figure-out what it is if (m instanceof TextMessage) { TextMessage message = (TextMessage) m; String body = message.getText(); System.out.println("Received: " + body); } else if (m instanceof MapMessage){ // do something else } else { // Handle Error } Receiving Message
  10. 10. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS In Summary ‣ Messaging provides a mechanism for shipping messages from producers to consumers ‣ The Message is a first class citizen, application data is not. ‣ The conversion of user data-types into the types supported by the messaging has to be addressed at the application level ‣ Byte sequences for AMQP ‣ Text/Map/Bytes/Stream/Object for JMS ‣ etc..
  11. 11. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ./data-distribution
  12. 12. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS What is Data Distribution? ‣ Data distribution is about making application defined data available where needed and when needed ‣ Data is a first class concept, it can be created, updated, read and eventually disposed ‣ The last value (or last N- values) of a Data is available VehicleSpeedVehicleSpeedVehicleSpeed plate “A123” “B456” “C789” dx dy 50 0 0 45 10 30 VehiclePositionVehiclePositionVehiclePosition plate “A123” “B456” “C789” x y 101 202 303 404 505 606 [1/2] TrakCo.VehiclePosition struct VehiclePosition { string plate; long x, y; } TrakCo.VehicleSpeed struct VehicleSpeed { string plate; long dx, dy; }
  13. 13. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS What is Data Distribution? ‣ The details of how application data is encapsulated and propagated are dealt by the data distribution technology VehicleSpeedVehicleSpeedVehicleSpeed plate “A123” “B456” “C789” dx dy 50 0 0 45 10 30 VehiclePositionVehiclePositionVehiclePosition plate “A123” “B456” “C789” x y 101 202 303 404 505 606 [2/2] TrakCo.VehiclePosition struct VehiclePosition { string plate; long x, y; } TrakCo.VehicleSpeed struct VehicleSpeed { string plate; long dx, dy; }
  14. 14. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data in DDS ‣ The definition of Data in DDS is tied to the definition of a Topic ‣ A Topic is the association between: ‣ Unique Name ‣ Type ‣ QoS ‣ A Topic can have associated a key in order to express data instances identities VehiclePosition TrackCo.VehiclePosition { Reliability Deadline, Priority Transient, ... } TopicType QoS Name [1/4] Topics
  15. 15. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data in DDS ‣ A Topic key can be made of an arbitrary number of Topic Type attributes ‣ Each unique key value identify a Topic Instance ‣ Topic Instance, can be Created, Read, Updated, and Disposed (CRUD) [2/4] VehiclePositionVehiclePositionVehiclePosition plate “A123” “B456” “C789” x y 101 202 303 202 101 606 TrakCo.VehiclePositionTopic TopicType Topic Instances Topic Key Topic Instances
  16. 16. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data in DDS ‣ An update for a Topic Instance is called Sample ‣ DDS can maintain a configurable history of samples per Topic Instance [3/4] VehiclePositionVehiclePositionVehiclePosition plate “A123” “B456” “C789” x y 101 202 303 202 101 606 TrakCo.VehiclePositionTopic TopicType Topic Instances Topic Key VehiclePositionVehiclePositionVehiclePosition plate “A123” “B456” “C789” x y 110 202 333 202 80 606 VehiclePositionVehiclePositionVehiclePosition plate “A123” “B456” “C789” x y 150 222 366 202 50 606 Samples Time Samples
  17. 17. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data in DDS Creation ‣ A Topic Instance is automatically created the first time it is written [4/4] Instances Lifecycle VehiclePositionVehiclePositionVehiclePosition plate “A123” x y 101 202 TrakCo.VehiclePositionTopic TopicType Topic Instances Topic Key Time NEW
  18. 18. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data in DDS Update ‣ Subsequent writes will simply update the value of the instance [4/4] Instances Lifecycle VehiclePositionVehiclePositionVehiclePosition plate “A123” x y 101 202 TrakCo.VehiclePositionTopic TopicType Topic Instances Topic Key Time NEW VehiclePositionVehiclePositionVehiclePosition plate “A123” x y 170 202
  19. 19. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data in DDS Dispose ‣ An instance can be explicitly disposed [4/4] Instances Lifecycle VehiclePositionVehiclePositionVehiclePosition plate “A123” x y 101 202 TrakCo.VehiclePositionTopic TopicType Topic Instances Topic Key Time NEW VehiclePositionVehiclePositionVehiclePosition plate “A123” x y 170 202 VehiclePositionVehiclePositionVehiclePosition plate “A123” x y 170 202 Disposed!
  20. 20. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ISO/OSIView of Data Distribution ‣ Data Distribution operates at the Session+Presentation layers ‣ Data Distribution Applications transparently read/ write application defined data Data Distribution Application
  21. 21. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data Distribution in DDS ‣ User defined types can be written with a simple write operation. Encapsulation and distribution are taken care by DDS [1/2] // Create data VehiclePosition pos(“A101”, 101, 202); // Write Data dataWriter.write(pos); Writing Data [Syntax compliant with the DDS-PSM-C++ upcoming standard as currently defined at http://code.google.com/p/dds-psm-cxx ]
  22. 22. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Data Distribution in DDS ‣ User defined types can be written with a simple write operation. Encapsulation and distribution are taken care by DDS [2/2] // Create data+info containers std::vector<VehiclePosition> vpos; std::vector<SampleInfo> info; // Read Data dataReader.read(vpos, info); Reading Data [Syntax compliant with the DDS-PSM-C++ upcoming standard as currently defined at http://code.google.com/p/dds-psm-cxx ]
  23. 23. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS In Summary ‣ Data Distribution provides a mechanism for sharing user-defined distributed data types and transparently reading and writing instances of these types ‣ Data types and data instances are first class citizens. This implies that data-life-cycle transitions are also dealt-with ‣ User defined Data-Types are preserved end-to-end. Their encoding is transparently dealt by the data distribution service
  24. 24. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ./vehicles-tracking
  25. 25. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS AVehicleTracking System Imagine to develop a s system that has to deal with the following: ‣ Identify when a car enters and leaves a city ‣ Track the position of each identified car and draw a path of the last N positions ‣ Track the instantaneous as well as the window average speed. Fine if the window average exceeds the speed-limit NEW BYE NEW
  26. 26. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS VehicleTracking with Messaging ‣ How are you going to manage the presence/absence of cars? ‣ This concern has to be implemented in your application logic. ‣ How are you going to easily distinguish between cars? ‣ Your application will have to deal with it. ‣ How are you going to maintain a window of vehicle speed so to enforce speed limits? ‣ Again, your application will have to explicitly maintain a last K-values cache of the speed for each care and make sure that this is updated each time you receive an update. Is there a better way to do this?
  27. 27. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS VehicleTracking with DDS ‣ How are you going to manage the presence/absence of cars? ‣ DDS provide that information through entity life-cycle management! ‣ How are you going to easily distinguish between cars? ‣ DDS allow to access samples on a per-instance basis. ‣ How are you going to maintain a window of vehicle speed so to enforce speed limits? ‣ DDS provide support for data history.
  28. 28. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS VehicleTracking with DDS ‣ Dynamic Discovery ‣ Cars will be discovered as soon as there is network connectivity ‣ QoS ‣ Local as well as End-to-End resource usage and temporal characteristic can be controlled via DDS QoSes ‣ Filtering ‣ Filter can be performed on the whole content and is not limited by the attributes that are attached to the header by data publishers ‣ DDS is wire-interoperable, language, and OS independent, thus it is easy to deploy it anywhere Much More with DDS
  29. 29. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Tracking SystemTopics ‣ Three different topics are used to represent the Vehicle, its position and the Speed ‣ The three topics define the same key so that information associated to the same car can be easily correlated TrakCo.VehiclePositionTopic TopicType Topic Key struct VehiclePosition { string plate; long x, y; } TrakCo.VehicleSpeedTopic TopicType Topic Key struct VehicleSpeed { string plate; long dx, dy; } TrakCo.VehicleTopic TopicType Topic Key struct Vehicle { string plate; string model; }
  30. 30. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Tracking SystemTopics QoS TrakCo.VehiclePosition struct VehiclePosition { string plate; long x, y; } TrakCo.VehicleSpeed struct VehicleSpeed { string plate; long dx, dy; } TrakCo.Vehicle struct Vehicle { string plate; string model; } Reliability History Durability Destination Order RELIABLE KEEP_LAST TRANSIENT SOURCE_TIMESTAMP Reliability History Durability Destination Order BEST_EFFORT KEEP_LAST(3) VOLATILE SOURCE_TIMESTAMP Reliability History Durability Destination Order BEST_EFFORT KEEP_LAST(5) VOLATILE SOURCE_TIMESTAMP
  31. 31. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ./extensible-topics
  32. 32. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Dealing withType Evolutions ‣ A DDS-based System might need to evolve its information model without inducing any global impact on required upgrades nor breaking interoperability between new and existing part for the system ‣ As an example let’s assume that we have now some cars that can provide more information on the vehicle, how do we capture this evolution?
  33. 33. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS DDS ExtensibleType System ‣ DDS is now (see Extensible and Dynamic Topic Types Specification) equipped with a structural type system ‣ This type system support type extension (evolution in general) relying on a structural sub-type operator and a set of annotations and QoS that allow to control and constraint the type system
  34. 34. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ./xtype-in-action
  35. 35. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS Type Extensibility Problem ‣ What if we need to extend the VehiclePosition type to add the error estimate on the x and y direction? ‣ How can we do this with no impact on running applications?
  36. 36. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS Type Extensibility Solution ‣ Simply extend the existing VehiclePosition and properly annotate new attributes ‣ Existing application will detect that the new type is an extension and will “project/extend” it to the known type ‣ Type “projection/extensions” can be controlled via QoS struct VehiclePosition { string plate; //@key long x, y; } struct VehiclePosition { string plate; //@key long x, y; float xe, ye;}
  37. 37. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS Type Extensibility Problem ‣ OK, that was good. But now I need to add an optional image to my Vehicle ‣ How can I add an optional field? Furthermore, I don’t want the figure to be copied into the topic, I’d like it to be a reference. Can I do this to limit the number of copies and reduce he memory utilization?
  38. 38. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS Type Extensibility Solution ‣ Simply extend the existing Vehicle and properly annotate new attributes struct Vehicle { string plate; //@key string model; } struct Vehicle { string plate; //@key string model; sequence<octet> photo; //@optional @shared }
  39. 39. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS ©2010PrismTech.AllRightsReserved. ./efficiency
  40. 40. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS Encoding Efficiency ‣ DDS serialization is several times more efficient than JSON, XML and YAML ‣ Differences can be quite extreme when dealing with complex types Size for Serialized CoFlight Flight Data Plan see Esposito et al. OMG RTWS 2008 DDS encoding is 10x more space efficient than XML!
  41. 41. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS Encoding Impact on RTT see Esposito et al. OMG RTWS 2008 DDS RTT with different Serialization Format ‣ Inefficient encoding can easily dominate RTT ‣ DDS native encoding is the most efficient among extensible encodings!
  42. 42. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS Size (bytes) Latency(usec) ‣ DDS implementations are capable of delivering very low and predictable latencies ‣ No other Pub/Sub technology can approach similar level of performance! DDS isVery Efficient!
  43. 43. ©2010PrismTech.AllRightsReserved. DDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDSOMGDDS In Summary ‣ DDS provides a strongly typed extensible type system ‣ Types can be evolved and extended without breaking backward compatibility nor loosing type-safety ‣ Type “Projection/Promotions” can be fully controlled via QoS ‣ This flexibility does not come at the cost of introducing the time +space overhead typical of XML-based approaches DDS provides extensibility without compromising efficiency!
  44. 44. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS ./summing-up
  45. 45. OpenSpliceDDSOpenSpliceDDOpenSpliceDDSOpenSpliceDDSOpenSpliceDDS Concluding Remarks ‣ Data Distribution give you far more than plain Messaging: ‣ higher level of abstraction and more built-in functionalities ‣ end-to-end type-safety ‣ extensibility ‣ performance ‣ Data Distribution is well beyond messaging, it is the most effective way of designing loosely coupled distributed applications.
  46. 46. http://www.opensplice.com/ http://www.opensplice.org/ emailto:opensplicedds@prismtech.com http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com http://bit.ly/1Sreg http://www.slideshare.net/angelo.corsaro http://twitter.com/acorsaro/ D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m OpenSplice DDS References

×