Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Communication Patterns Using Data-Centric Publish/Subscribe


Published on

Fundamental to any distributed system are communication patterns: point-to-point, request-reply, transactional queues, and publish-subscribe. Large distributed systems often employ two or more communication patterns. Using a single middleware that supports multiple communication patterns is a very cost-effective way of developing and maintaining large distributed systems. This talk will begin with an introduction of Data Distribution Service (DDS) – an OMG standard – that supports data-centric publish-subscribe communication for real-time distributed systems. DDS separates state management and distribution from application logic and supports discoverable data models. The talk will then describe how RTI Connext Messaging goes beyond vanilla DDS and implements various communication patterns including request-reply, command-response, and guaranteed delivery. You will also learn how these patterns can be combined to create interesting variations when the underlying substrate is as powerful as DDS. We’ll also discuss APIs for creating high-performance applications using the request-reply communication pattern.

Published in: Technology

Communication Patterns Using Data-Centric Publish/Subscribe

  1. 1. Sumant Tambe, Ph.D.Senior Software Research EngineerReal-Time Innovations,
  2. 2. Agenda• Communication Models• Data Distribution Service Standard• Data-Centricity 101 – Demo• Communication Patterns – Request/Reply – Guaranteed Delivery10/11/2012 2
  3. 3. Middleware Communication Models10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 3
  4. 4. RPC Vs. Pub-Sub/Messaging/Data-Distribution• RPC (WebServices, CORBA, DCOM) offer a remote method abstraction – Familiar OO programming model – Results in a tightly-coupled system • Forces synchronous invocations • Imposes global object model • Limited QoS(appearance of local method call) • Lack robustness: cascading points of failure – Typically built on top of TCP: • Impacts scalability and time-determinism – Best-suited to smaller, closely-coupled systems• Pub-Sub (Messaging Data-Distribution) offer a queue-based and/or replicated-data model – Subsystems are decoupled in time, space, and synchronization • Contracts established by verifying QoS compatibility – Supports a variety of transports including multicast UDP – Better suited for high-performance and real-time 10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 4
  5. 5. Queue Vs. Pub/Sub• Queue – Message sent to Queue – Multiple readers can read from the queue – Each message is delivered to ONLY one reader • Readers “affect each other” – Apps: • Job Scheduling, Load Balancing, Collaboration• Pub/Sub – Message sent to Topic – Multiple readers can subscribe to Topic with or without filters – Each message delivered to ALL subscribers that pass filter • Readers are decoupled – Apps: • Notifications, Information Distribution 10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 5
  6. 6. Pub/Sub Vs. Data-Distribution• Pub-Sub – Only messages no concept of data – Each message is interpreted without context – Messages must be delivered FIFO or according to some “priority” attribute – No Caching of data – Simple QoS: filters, durability, lifespan• Data-Distribution – Messages represent update to data-objects – Data-Objects identify by a key Data Bus – Middleware maintains state of each object – Objects are cached. Applications can read at leisure – Smart QoS • Ownership, History (per key), Deadline – Subsumes Pub-Sub 10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 6
  7. 7. DDS: Standards-based Integration Infrastructure for Critical Applications Streaming Sensors Events Data Real-Time Enterprise Actuators Applications Applications© 2009 Real-Time Innovations, Inc.
  8. 8. Systems that interact with the Real World • Must adapt to changing environment • Cannot stop processing the information • Live within world-imposed timing Beyond traditional interpretation of real-time© 2010 Real-Time Innovations, Inc.
  9. 9. Real-Time Data Distribution10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 9
  10. 10. Family of Specifications2008 2009 2010 2010 2012 2013UML Profile DDS for DDS DDS-STD-C++ DDS Security RPC over DDSfor DDS Lw CCM X-Types DDS-JAVA5 App 2004 App App DDS Spec DDS 2006 DDS DDSImplementation DDS Implementation Implementation Interoperablity Network / TCP / UDP / IP © 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 10
  11. 11. Broad Adoption• Vendor independent Cross-vendor portability – API for portability – Wire protocol for interoperability• Multiple implementations – 10 of API DDS API – 8 support RTPS• Heterogeneous Middleware – C, C++, Java, .NET (C#, C++/CLI) – Linux, Windows, VxWorks, other DDS Real-Time embedded & realtime Publish-Subscribe Wire Protocol (RTPS)• Loosely coupled Cross-vendor interoperability© 2009 Real-Time Innovations, Inc. 11
  12. 12. DDS adopted by key programs• European Air Traffic Control – DDS used to interoperate ATC centers• UK Generic Vehicle Architecture – Mandates DDS for vehicle comm. – Mandates DDS-RTPS for interop.• DISR (DoD Information Technology Standards Registry) – Mandates DDS for Pub-Sub API and Interop.• US Navy Open Architecture – Mandates DDS for Pub-Sub• SPAWAR NESI – Mandates DDS for Pub-Sub SOA © 2010 Real-Time Innovations, Inc.
  13. 13. Evolution of DataBusData-centricity basics
  14. 14. Everyday Example: Schedule Meeting via Emails Alternative Process #1 (message-centric): 1. Email: “Meeting Monday at 10:00.” 2. Email: “Here’s dial-in info for meeting…” 3. Email: “Meeting moved to Tuesday” 4. You: “Where do I have to be? When?” 5. You: (sifting through email messages…) 14
  15. 15. Everyday Example: Schedule Meeting Using a Calendar Alternative Process #2: 1. Calendar: (add meeting Monday at 10:00) 2. Calendar: (add dial-in info) 3. Calendar: (move meeting to Tuesday) 4. You: “Where do I have to be? When?” 5. You: (check calendar. Contains consolidated-state)The difference is state!The infrastructure consolidates changes and maintains it 15
  16. 16. What’s the Difference? State.• Objects have identity and “State” (“data”) is a attributes – The meeting will run 1:00– snapshot of those 2:00 in the conference room. attributes and – My friend’s phone number is characteristics. 555-1234 his email is… – The car is blue and is traveling north from Sunnyvale at 65 mph. If the infrastructure• …whether they exist in maintains the state, the real world, in the the application does computer, or both not need to re-• …whether or not we construct it… observe or acknowledge them
  17. 17. Why is it better to have the (data-centric) middleware manage the state?• Reconstructing the state of an object is hard – Must infer based on all previous messages – Maintaining all these messages is expensive – Each app makes these inferences  duplicate effort• Reconstructing state is not robust – Many copies of state  may be different  bugs vs. Uniform operations on state  fewer bugs – Any missing change compromises integrity• State awareness results in better performance – Middleware can be smart about what to send and when• Data-type awareness simplifies programming – Middleware supports direct definition and instantiation of the data- model
  18. 18. DDS Communication Model10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 18
  19. 19. Demo: Publish-Subscribe© 2009 Real-Time Innovations, Inc.
  21. 21. Communication PatternsBeyond DDS10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 21
  22. 22. Why Patterns• From app programming POV – Patterns solve specific problems nicely• From platform or systems POV – Handling each pattern in independent ad hoc ways results in brittle, poorly performing systems• Solution: Build platform in terms of fundamental patterns – Derive high-level patterns from basic patterns; expose those to apps © 2012 RTI 22
  23. 23. Communication PatternRequest/Reply
  24. 24. Introduction to Request/Reply• Publish/Subscribe (Pub/Sub) – Fundamental pattern in RTI Connext – Pub/Sub shines where there is • Push model • Unidirectional stream of data • One-to-many (multicast)• Request/Reply – Client-server model built on top of pub/sub – Request/Reply shines where there is • Pull model • Bidirectional communication • A need for remote code execution – E.g., Get/set configuration, send commands10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 24
  25. 25. Request-Reply Request Request TopicRequestor (Foo) Replier Reply Topic Reply (Bar) © 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
  26. 26. Correlation Correlation Message ID 1 3 2 1 Requests Replier 1 1 2 3Requestor Replies Correlation ID © 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
  27. 27. How Does Correlation Work Requester App Replier App sample_id1 Data VGUID1 VSN1 Foo Data Foo Write Writer Reader (VGUID1) Part of Foo ProcessingSampleInfo VSN1 VGUID1 Bar Read Data Data Bar Reader Writer VGUID2 VSN2 VGUID1 VSN1 Bar (VGUID2) sample_id-id2 related_sample_id1 Index Condition “@related_sample_id.VSN == VSN1” Content-based Filter “@related_sample_id.VGUID == VGUID1” Filtering Implementation details10/11/2012 © 2012 RTI RTI • COMPANY CONFIDENTIAL © 2012 • COMPANY CONFIDENTIAL 27 27
  28. 28. Single-Request Multiple-Reply RequestRequestor Replier 1 2 3 Replies Sequence ID © 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
  29. 29. Multiple Repliers Reply Replier ARequester Replier B Request Replier C © 2011 Real-Time Innovations, Inc.
  30. 30. Replier “Hello, World!” (Server-side)using namespace connext;DDS::DomainParticipantFactory factory = DDS::DomainParticipantFactory::get_instance();DDS::DomainParticipant * participant = factory->create_participant(domain_id, DDS::PARTICIPANT_QOS_DEFAULT, NULL, DDS::STATUS_MASK_NONE);std::auto_otr<Replier<Foo, Bar> > replier (new Replier<Foo, Bar>(participant, "TestService"));Sample<Foo> request;// Receive one requestbool received = replier->receive_request(request, MAX_WAIT);if (!received) { std::cout << "Requests not received" << std::endl; return false;}// A WriteSample automatically initializes and finalizes// the data using the TypeSupport (in this case BarTypeSupport)WriteSample<Bar> reply;if ( { sprintf(, "Reply for %s",; // Send a reply for that request replier->send_reply(reply, request.identity()); // Note: a replier can send more than one reply for the same request}10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 30
  31. 31. Requester “Hello, World!” (Client-side) using namespace connext; // Create a DomainParticipant DDS::DomainParticipant * participant = ...; // Create a Requester std::auto_ptr<Requester<Foo, Bar> > requester (new Requester<Foo, Bar>(participant, "TestService")); // Send request WriteSample<Foo> request; strcpy(, "A Request"); requester->send_request(request); if (requester->wait_for_replies(2, MAX_WAIT, request.identity())) { LoanedSamples<Bar> replies = requester->take_replies(request.identity()); // If there are no replies, the container is empty for(LoanedSamples<Bar>::const_iterator it = replies.begin(); it != replies.end(); ++it) { if (it->info().valid_data) { std::cout << "Received reply: " << it->data().message << std::endl; } } }10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 31
  32. 32. Implementing Request/Reply Pattern Steps With Vanilla DDS API With BIGPINE API Define request and reply types Manual Manual Create Participant Manual Manual Pollute application data model with a Manual Not necessary correlation-id Create publisher/subscriber Manual Automatic Create datareader/datawriter Manual Automatic Register request/reply types Manual Automatic Create request/reply topics Manual Automatic Setup CFT to filter unwanted replies Manual Automatic Correlation of request/reply samples Manual Semi-Automatic Memory management of samples Manual Automatic Lifecycle management of entities Manual Automatic Language friendly (idiomatic API) Partially Yes Scalable Depends on impl Yes10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 32
  33. 33. Salient Features of Connext Request/Reply• New API, built on top of DDS• Data-centric• Scalable• Simple to use, abstracts many DDS details• Idiomatic, language-friendly API (C++, Java, and C#)• It’s not RPC, though can be the foundation for an RPC solution 10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 33
  34. 34. Scalable Request-Reply Requester 1 Reply 1 Replier Requester 2 Replies Reply 2 …Thousands of clients! Requester N Reply N © 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
  35. 35. Combining Patterns Subscriber Wire Tap Request Request TopicRequestor Replier Reply Topic Reply Subscriber © 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
  36. 36. Communication PatternGuaranteed Delivery
  37. 37. The Guaranteed Delivery Problem• Delivery of samples from a Producer to the Intended Consumers such that the delivery is robust to multiple kind of failures• Guaranteed delivery does not imply complete certainty  Continuum of Delivery Guarantees © 2012 RTI • COMPANY CONFIDENTIAL
  38. 38. The Guaranteed Delivery Scenario • Producer ‘P’ publish a Sample ‘S’ that should be received by Consumer ‘C’ Consumer process failure Producer process failure (the process crashes after the sample is(the producer process crashes after the received by the middleware and before it is sample has been written) processed by the application) Sample ‘S’ Producer ‘P’ Consumer ‘C’ (DataWriter) (DataReader) Network and transport failures (packets are lost due to temporary network failures or Late joiner consumer transport/middleware buffer overflows) (the consumer is not running when the sample is written) © 2012 RTI • COMPANY CONFIDENTIAL
  39. 39. Guaranteed Delivery App-level ackPublisher Subscriber Message Message Durable Subscriber Message Message Disk © 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL
  40. 40. DDS Durability QoS PrimerDDS Durability controls data availability with respect tolate joining datareaders• VOLATILE – No need to keep data samples for late joiners• TRANSIENT_LOCAL – Data instances available for late joiners – Data does not survive beyond datawriter• TRANSIENT – Data is not tied to the lifecycle of the datawriter. – A separate service (persistence service) keep data in memory• PERSISTENT – Data is kept on permanent storage, so that they can outlive a system session 10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 40
  41. 41. Beyond DDS: Required Subscriptions• DataWriter can be configured with: – a list of required subscriptions • Required subscriptions are named subscriptions identified by a role name• DataWriter must store a sample until both: – Acknowledged by all active reliable readers – Acknowledged by all required subscriptions• DataReader identifies itself: – As serving a required subscription – Uniquely via a virtual GUID
  42. 42. Beyond DDS: Durable Subscriptions• Concept – It is a required subscription with durability >= TRANSIENT – Features that allows to keep data until received by durable subscriptions for which readers may or may not exist at any given time – Data is not lost even if the DataWriter crashes• Benefits and Use Cases • In combination with other features durable subscriptions provides guaranteed messaging• Requires persistence service – To persist data – To persist existence of durable subscriptions and their stats
  43. 43. Beyond DDS: Collaborative DataWriters• Concept: – Multiple DataWriters can relay samples from a common stream (data source). The DataReaders reconstruct the original stream and deliver to the application the complete set of samples• Benefits & Use cases: – High availability and fault tolerance • Multiple DataWriters publishing the same samples – Load balancing and data partition • Multiple DataWriters publishing different sets of samples from the same source – Scalability • Partition across different machines and cores
  44. 44. Delivery Models: Best-Effort to Guaranteed!Use Case Features in RTI Connext Messaging 5.0.0Periodic data from live producers Best effort reliabilityEvent data from live producers Reliable reliabilityLast value cache from live producers (entire Transient local durability +state of the datawriter cache) Reliable reliabilityDurable last value cache (datawriter may Persistent durability +come and go) Reliable reliability + Persistence ServiceGuaranteed messaging from live producers Transient local durability + Reliable reliability + Required subscriptions + Application level ACKDurable Guaranteed messaging Persistent durability + Reliable reliability + Collaborative DataWriters + Persistence Service + Durable subscriptions + Application level ACK © 2012 RTI • COMPANY CONFIDENTIAL
  45. 45. References• Real-Time Innovations, Inc. –• RTI Connext DDS –• RTI Connext Request-Reply API – Just Google!• OMG DDS Specifications – 10/11/2012 © 2012 RTI • COMPANY CONFIDENTIAL 45
  46. 46. Thank you © 2012 RTI • COMPANY CONFIDENTIAL