0
Data-DistributionService (DDS)The Ideal Bus for CEP     dds/2008-03-04     Gerardo Pardo-Castellote, Ph.D.     Co-Chair Da...
Agenda     Introduction     Middleware and CEP     The OMG DDS standard as a message bus for CEP     Conclusion2          ...
History:              DDS the Standard     Data Distribution Service for Real-Time Systems      – Adopted in June 2003    ...
DDS Adoption – Aerospace & Defense              Boeing              AWAKS          Northtrop                              ...
DDS Adoption – Transportation, Industrial               WiTronix               Train and vehicle         Schneider Electri...
Many others6                 © 2007 Real-Time Innovations, Inc.
DDS Mandates for Net-Centric Systems     DISR (formerly JTA)      – DoD Information Technology         Standards Registry ...
Middleware and CEP: The hose that feeds the engine        Inputs                                             Outputs      ...
Middleware and CEP: Scalability & Load Balancing                     CEP Engine                      CEP Engine      CEP E...
Middleware and CEP: Integration &     Interoperability                    Vendor 2                      Vendor 3        Ve...
Implementing the Global Event Bus/Data     Streams                                EventType1                              ...
Standardizing the Adaptor/Transport     NEEDED:       Not so much a standard adaptor…       …but:        – A standard even...
Requirements on a CEP Message Bus     Fit to the CEP information Model      – CEP engines need access to data regardless o...
DDS: The best “message bus” for CEP?     Fit to the CEP information Model      – DDS is Data-Centric, not message centric ...
#1 DDS Data-Centric Model     “Global Data Space” generalizes Subject-Based Addressing       – Data objects addressed by D...
#1 DDS Data-Centric Model     “Global Data Space” generalizes Subject-Based Addressing       – Data objects addressed by D...
#1 DDS Data-Centric Model     “Global Data Space” generalizes Subject-Based Addressing       – Data objects addressed by D...
Subscriptions: By Topic, Subject, Content      Topic: “Market Data”                              Key Fields               ...
DDS communications model                         Data         Domain                                            Data      ...
Demo: Concepts                                                         Start demo      Display Area:      Shows state of o...
QoS: Quality of Service     Standardized Middleware QoS semantics       QoS Policy                                QoS Poli...
Demo: Quality of Service (QoS)                                                                                        Star...
DDS: The best “message bus” for CEP?     #1 Fit to the CEP information Model      – DDS is Data-Centric, not message centr...
Data-Distribution and Real-Time      Messaging Technologies and Standards                                                 ...
Latency – (Linear Scale)     IBM HS20 blades     Dual 2.8 GHz Xeon, 1 GB RAM     Gigabit Ethernet                  DDS/JMS...
Jitter – (Linear Scale)                                                                                            DDS/COR...
DDS : Low Latency and Jitter                                         Latency and Jitter on Unloaded Network               ...
DDS: Low Overhead Enables High                          Throughput                          70,000                        ...
DDS: Scalable Performance (Confirmed                                          Reliability)                                ...
DDS operates without brokers             DDS                                                 Broker-based                 ...
DDS: The best “message bus” for CEP?     #1 Fit to the CEP information Model      – DDS is Data-Centric, not message centr...
#3 QoS and Powerful Services       – Redundancy &                 QoS Policy              QoS Policy         Failover     ...
Ownership and High Availability     Producer / Writer                               I1 Prim       strength=10             ...
Data Persistence     A standalone service that persists data outside of     the context of a DataWriter                   ...
Conclusion       To get the best performance from your CEP you       need the best message bus       DDS is a natural fit ...
Upcoming SlideShare
Loading in...5
×

Dds the ideal_bus_for_event_processing_engines

790

Published on

Use of the Data-Distribution Service (DDS) --a publish-subscribe middleware standard from OMG -- as a communication infrastructure for Event Processing Engines.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
790
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Dds the ideal_bus_for_event_processing_engines"

  1. 1. Data-DistributionService (DDS)The Ideal Bus for CEP dds/2008-03-04 Gerardo Pardo-Castellote, Ph.D. Co-Chair Data-Distribution SIG CTO, Real-Time Innovations, Inc. 13 March 2008
  2. 2. Agenda Introduction Middleware and CEP The OMG DDS standard as a message bus for CEP Conclusion2 © 2007 Real-Time Innovations, Inc.
  3. 3. History: DDS the Standard Data Distribution Service for Real-Time Systems – Adopted in June 2003 – Finalized in June 2004 – Revised June 2005, June 2006 – Joint submission (RTI, THALES, OIS) – Specification of API for Data-Centric Publish-Subscribe in real- time distributed systems. Interoperability wire protocol – Adopted in July 2006 – Revised in July 2007 Multiple Implementations – 4 commercial – 3 open source – Several more in-house3 © 2007 Real-Time Innovations, Inc.
  4. 4. DDS Adoption – Aerospace & Defense Boeing AWAKS Northtrop E2C Hawkeye Boeing Future Combat Systems Raytheon SSDS Lockheed AEGIS Insitu Unmanned Air Vehicles4
  5. 5. DDS Adoption – Transportation, Industrial WiTronix Train and vehicle Schneider Electric Tracking Industrial Automation Tokyo Japan Traffic Control Kuka Robotics EU and US Air Traffic Management Varian Medical Instruments5
  6. 6. Many others6 © 2007 Real-Time Innovations, Inc.
  7. 7. DDS Mandates for Net-Centric Systems DISR (formerly JTA) – DoD Information Technology Standards Registry US Navy Open Architecture FCS SOSCOE – Future Combat System – System of System Common Operating Environment NESI – Net-centric Enterprise Solutions for Interoperability UK MOD & NATO – Advocating Open Systems7 © 2007 Real-Time Innovations, Inc.
  8. 8. Middleware and CEP: The hose that feeds the engine Inputs Outputs CEP Engine CEP Engine8 © 2007 Real-Time Innovations, Inc.
  9. 9. Middleware and CEP: Scalability & Load Balancing CEP Engine CEP Engine CEP Engine CEP Engine9 © 2007 Real-Time Innovations, Inc.
  10. 10. Middleware and CEP: Integration & Interoperability Vendor 2 Vendor 3 Vendor 1 Vendor 410 © 2007 Real-Time Innovations, Inc.
  11. 11. Implementing the Global Event Bus/Data Streams EventType1 EventType2 Global Event Bus EventType2 ADAPTOR != MIDDLEWARE Multiple adaptor technologies will be needed… but not all are created equal It is highly desirable to use a middleware where the event model and use- case requirements are handled as first-class citizens Event semantic model can be pushed into the middleware information11 model © 2007 Real-Time Innovations, Inc.
  12. 12. Standardizing the Adaptor/Transport NEEDED: Not so much a standard adaptor… …but: – A standard event meta-model Relevant OMG standards: UML, MOF, OCL, … – A ‘standard’ way to adapt to the existing Transports/Message Busses & information model Using the same Adaptor technology will not allow CEP engines to “interoperate” Relevant OMG standards: DDS, CORBA Notification Service Communicate != Interoperate12 © 2007 Real-Time Innovations, Inc.
  13. 13. Requirements on a CEP Message Bus Fit to the CEP information Model – CEP engines need access to data regardless of its source – CEP operates on structured data, not opaque messages Performance & Scalability – CEP systems are event-driven – CEP operates on streaming data and has low-latency & high- throughput requirements Configurability, QoS, Filtering & Services – CEP is deployed in critical systems that be robust to failures – CEP weaves streams with different QoS requirements (updates rates, priority) – CEP often needs access to only a subset of the information13 © 2007 Real-Time Innovations, Inc.
  14. 14. DDS: The best “message bus” for CEP? Fit to the CEP information Model – DDS is Data-Centric, not message centric – Beyond messaging DDS also does “state management” – DDS Supports Topic-Based and Content-Based Subscriptions – DDS built-in Window/Time selectors delivers *only* relevant data Performance & Scalability – DDS offers notification-based APIs – DDS has the best performance (latency, throughput, jitter, scalability) of any standard middleware technology Configurability, QoS, Filtering & Services – DDS Publish-Subscribe model delivers wherever needed – DDS has rich QoS to support real-time event-driven systems (latency budget, priority, etc.) – DDS Built-in Fault-Tolerance Mechanism allow for redundant sources of data – DDS built-in services Persistence and Historical Data Services ensures relevant data is never lost14 © 2007 Real-Time Innovations, Inc.
  15. 15. #1 DDS Data-Centric Model “Global Data Space” generalizes Subject-Based Addressing – Data objects addressed by DomainId, Topic and Key – Domains provide a level of isolation – Topic groups homogeneous subjects (same data-type & meaning) – Key is a generalization of subject Key can be any set of fields, not limited to a “x.y.z …” formatted string – Data Structure is known by the middleware Data Reader Data Writer Topic Data Reader Data Writer Data Reader Data Writer15 © 2007 Real-Time Innovations, Inc.
  16. 16. #1 DDS Data-Centric Model “Global Data Space” generalizes Subject-Based Addressing – Data objects addressed by DomainId, Topic and Key – Domains provide a level of isolation – Topic groups homogeneous subjects (same data-type & meaning) – Key is a generalization of subject Key can be any set of fields, not limited to a “x.y.z …” formatted string – Data Structure is known by the middleware Data Reader Data Writer Key (subject) Data Reader Data Writer Data Reader Data Writer16 © 2007 Real-Time Innovations, Inc.
  17. 17. #1 DDS Data-Centric Model “Global Data Space” generalizes Subject-Based Addressing – Data objects addressed by DomainId, Topic and Key – Domains provide a level of isolation – Topic groups homogeneous subjects (same data-type & meaning) – Key is a generalization of subject Key can be any set of fields, not limited to a “x.y.z …” formatted string – Data Structure is known by the middleware Data Reader Data Writer Data Object Data Reader Data Writer Data Reader Data Writer17 © 2007 Real-Time Innovations, Inc.
  18. 18. Subscriptions: By Topic, Subject, Content Topic: “Market Data” Key Fields Other Fields Payload Field Source Symbol Type Exchange Volume Bid Ask … Value * * * * * Topic: “Order Entry” Key Fields Other Fields Field Symbol Type Exchange OrderNumber OrderKind Stop Limit … Value * * NYSE * Subject Filter (for a Reader) Topic: “Market Data” Key Fields Other Fields Field Source Symbol Type Exchange Payload Value REUTERS * EQ NYSE Volume > x, Ask < y18 © 2007 Payload Filter (for a Reader) Subject Filter (for a Reader) Real-Time Innovations, Inc.
  19. 19. DDS communications model Data Domain Data Domain New Writer Participant Reader Participant “Alarm” Got new “Alarm” subscriber! data Offered Offered Listener QoS Listener QoS Participants scope the global data space (domain) Topics define the data-objects (collections of subjects) Writers publish data on Topics Readers subscribe to data on Topics QoS Policies are used configure the system Listeners are used to notify the application of events19 © 2007 Real-Time Innovations, Inc.
  20. 20. Demo: Concepts Start demo Display Area: Shows state of objects Topics – Square, Circle, Triangle – Attributes Data types (schemas) – Shape (color, x, y, size) Color is instance Key – Attributes Shape & color used for key QoS – Deadline, Liveliness – Reliability, Durability – History, Partition Control Area: – Ownership20 Allows selection of objects and Real-Time Innovations, Inc. © 2007 QoS
  21. 21. QoS: Quality of Service Standardized Middleware QoS semantics QoS Policy QoS Policy DURABILITY USER DATA HISTORY (per subject) TOPIC DATA READER DATA LIFECYCLE GROUP DATA WRITER DATA LIFECYCLE PARTITION LIFESPAN PRESENTATION ENTITY FACTORY DESTINATION ORDER RESOURCE LIMITS OWNERSHIP RELIABILITY OWNERSHIP STRENGTH TIME BASED FILTER LIVELINESS DEADLINE LATENCY BUDGET CONTENT FILTERS TRANSPORT PRIORITY21 © 2007 Real-Time Innovations, Inc.
  22. 22. Demo: Quality of Service (QoS) Start demo Writers and readers state Their needs Topics – Square, Circle, Triangle – Attributes Data types (schemas) – Shape (color, x, y, size) Color is instance Key – Attributes Shape & color used for key QoS – Deadline, Liveliness – Reliability, Durability – History, Partition – Ownership RTI DDS delivers22 © 2007 Real-Time Innovations, Inc.
  23. 23. DDS: The best “message bus” for CEP? #1 Fit to the CEP information Model – DDS is Data-Centric, not message centric – DDS Supports Topic-Based and Content-Based Subscriptions – DDS built-in Window/Time selectors delivers *only* relevant data #2 Performance & Scalability – DDS offers notification-based APIs – DDS has the best performance (latency, throughput, jitter, scalability) of any standard middleware technology #3 Configurability, QoS, Filtering & Services – DDS Publish-Subscribe model delivers wherever needed – DDS has rich QoS to support real-time event-driven systems (latency budget, priority, etc.) – DDS Fault-Tolerance Mechanism allows for redundant sources of data – DDS built-in services Persistence and Historical Data Services ensures relevant data is never lost23 © 2007 Real-Time Innovations, Inc.
  24. 24. Data-Distribution and Real-Time Messaging Technologies and Standards Web Services Java RTSJ (soft RT) RTSJ (hard RT) Java/RMI Java/JMS CORBA RT CORBA Data Distribution Service / DDS MPI Non-real-time Soft real-time Hard real-time Extreme real-time24 © 2007 Real-Time Innovations, Inc. Adapted from NSWC-DD OA Documentation
  25. 25. Latency – (Linear Scale) IBM HS20 blades Dual 2.8 GHz Xeon, 1 GB RAM Gigabit Ethernet DDS/JMS/Notification Service Comparison - Latency 2500 2000 DDS JMS Notification Service 1500 1000 500 0 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 Message Size (bytes) Adapted from Vanderbilt presentation at July 2006 OMG Workshop on RT Systems25 © 2007 Real-Time Innovations, Inc.
  26. 26. Jitter – (Linear Scale) DDS/CORBA Notification Service Comparison - Jitter 100 DDS/JMS/CORBA Notification Service Comparison - Jitter Standard Deviation (usecs) 80 2000 1800 DDS JMS Notification service 60 Standard Deviation (usecs) 1600 1400 40 DDS JMS Notification service 1200 1000 20 800 600 0 400 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 200 Message Size (bytes) 0 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 65536 Message Size (bytes) Source: Vanderbilt presentation at July 2006 OMG Workshop on RT Systems26 © 2007 Real-Time Innovations, Inc.
  27. 27. DDS : Low Latency and Jitter Latency and Jitter on Unloaded Network 400 Latency (microseconds) 350 300 Maximum 250 99.99% 200 99% 150 Median Minimum 100 50 0 32 64 128 256 512 1024 2048 4096 8192 Message/Data Size (bytes) Reliable, ordered delivery over Gigabit Ethernet between 2.0 GHz Opteron processors running 32-bit Red Hat Enterprise Linux 4.027 © 2007 Real-Time Innovations, Inc.
  28. 28. DDS: Low Overhead Enables High Throughput 70,000 1000 900 60,000 800 50,000 700 Megabits per Second Updates per Second 600 40,000 500 30,000 400 20,000 300 200 10,000 100 0 0 16 32 64 128 256 512 1024 2048 4096 8192 16384 32768 Message/Data Size Reliable, ordered delivery over Gigabit Ethernet between 2.0 GHz Opteron processors running 32-bit Red Hat Enterprise Linux 4.028 © 2007 Real-Time Innovations, Inc.
  29. 29. DDS: Scalable Performance (Confirmed Reliability) 60,000 50,000 Point-to-Point Updates per Second 40,000 1-1 1-10 1-24 30,000 20,000 10,000 0 16 32 64 128 256 512 1024 Message/Data Size (bytes - without batching) Reliable, ordered delivery over Gigabit Ethernet between 2.0 GHz Opteron processors29 running 32-bit Red Hat Enterprise Linux 4.0 © 2007 Real-Time Innovations, Inc.
  30. 30. DDS operates without brokers DDS Broker-based Middleware No Brokers => Better Performance & Determinism + Increased Robustness30 © 2007 Real-Time Innovations, Inc.
  31. 31. DDS: The best “message bus” for CEP? #1 Fit to the CEP information Model – DDS is Data-Centric, not message centric – DDS Supports Topic-Based and Content-Based Subscriptions – DDS built-in Window/Time selectors delivers *only* relevant data #2 Performance & Scalability – DDS offers notification-based APIs – DDS has the best performance (latency, throughput, jitter, scalability) of any standard middleware technology #3 Configurability, QoS, Filtering & Services – DDS Publish-Subscribe model delivers wherever needed – DDS has rich QoS to support real-time event-driven systems (latency budget, priority, etc.) – DDS Fault-Tolerance Mechanism allows for redundant sources of data – DDS built-in services Persistence and Historical Data Services ensures relevant data is never lost31 © 2007 Real-Time Innovations, Inc.
  32. 32. #3 QoS and Powerful Services – Redundancy & QoS Policy QoS Policy Failover DURABILITY USER DATA – Persistent Data HISTORY (per subject) TOPIC DATA – Last value cache READER DATA GROUP DATA LIFECYCLE – Historical cache WRITER DATA PARTITION LIFECYCLE LIFESPAN PRESENTATION ENTITY FACTORY DESTINATION ORDER RESOURCE LIMITS OWNERSHIP RELIABILITY OWNERSHIP STRENGTH TIME BASED FILTER LIVELINESS DEADLINE LATENCY BUDGET CONTENT FILTERS TRANSPORT PRIORITY32 © 2007 Real-Time Innovations, Inc.
  33. 33. Ownership and High Availability Producer / Writer I1 Prim strength=10 ary Topic T1 Producer / Writer I1 Backup strength=5 I2 Primary I1 I2 Producer / Writer I2 Backup strength=1 Owner determined per subject Only extant writer with highest strength can publish a subject (or topic for non-keyed topics) Automatic failover when highest strength writer: – Loses liveliness – Misses a deadline – Stops writing the subject Shared Ownership allows any writer to update the subject33 © 2007 Real-Time Innovations, Inc.
  34. 34. Data Persistence A standalone service that persists data outside of the context of a DataWriter Data Data Writer Data Writer Reader Data Global Reader Data Space Persistence Persistence Service Service Permanent Permanent Storage Storage34 © 2007 Real-Time Innovations, Inc.
  35. 35. Conclusion To get the best performance from your CEP you need the best message bus DDS is a natural fit for CEP: – The Data-Centric Pub-Sub information model is an excellent match to the CEP structured event model – DDS supports many of the CEP use-cases and QoS as first-class citizens – DDS offers the best performance and latency of any middleware standard35 © 2007 Real-Time Innovations, Inc.
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×