Your SlideShare is downloading. ×
  • Like
Large-Scale System Integration with DDS for SCADA, C2, and Finance
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Large-Scale System Integration with DDS for SCADA, C2, and Finance

  • 673 views
Published

Presentation to the OMG Real-Time Workshop in May 2010 on system integration patterns, especially (but not exclusively) with respect to OMG Data Distribution Service (DDS) technology.

Presentation to the OMG Real-Time Workshop in May 2010 on system integration patterns, especially (but not exclusively) with respect to OMG Data Distribution Service (DDS) technology.

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
673
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
13
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • When I say “large” in this presentation, I’m primarily talking about complexity. I’m not talking primarily about a unified simple design that happens to have lots of participants in it.

    Same principles apply to defense systems, financial systems, power systems, industrial automation, etc.
  • Care about lots of the same things as any system designers — functionality, performance, … — but care about certain things much more.

    Isolation relates to governance: making sure integration doesn’t violate SLAs
  • This will eventually be a talk about DDS, but we’ll get to that later
  • Integrating two subsystems with different data spaces is not the same as joining them into one data space.

    You will be tempted to just mush things together. (Just connect the network cable; it’s easy, right?) Beware that temptation.
    Are the different subsystems using the same structural and behavioral data model?
    Are the network environments the same?
    Do they evolve together?
    Suppose the answers are both Yes.
    Will they continue to be the same over time as the composed system evolves?
    Will the system behave the same when all the data is going to twice as many consumers?
    Can one team of people understand the design and operation of a now-much-more-complex single subsystem?
  • Same principles apply in every industry
  • Same principles apply in every industry
  • Same principles apply in every industry
  • Same principles apply in every industry
  • 2 problem areas brought up before that we haven’t discussed yet.
    These will be bridge to technology-specific discussion in 2nd half.
  • Zoom in…
  • Test results from lab of hundreds of multi-core Linux machines connected by gigabit Ethernet
  • Brokered application data should be self-explanatory at this point.

Transcript

  • 1. The Real-Time Middleware Experts Large-Scale System Integration with DDS for SCADA, C2, and Finance Rick Warren, Principal Engineer rick.warren@rti.com
  • 2. What do I mean by “large system”?  Systems of systems – Modular, hierarchical design – Legacy components, subsystems – Multiple technologies – Global scale  Decoupled subsystem lifecycles – Independent development – Independent deployment and use – Independent management – Independent revision and retirement  Multiple communities of interest – Different data interest, entitlements – Non-uniform levels of trust © 2009 Real-Time Innovations, Inc. 2 LAN DDS #1 (Initech) JMS (Dunder Mifflin) DDS #2 (Acme) Legacy (COBOL ‘R’ US) Satellite Links
  • 3. What matters to integrators of large systems?  Governance (including over security) – What information will be exchanged? – Under what conditions will it be exchanged? – Who is allowed to exchange this information? – If these SLAs are violated, can the exchange be prevented? Can I be notified? – (In the past, what has occurred wrt these SLAs?)  Isolation – When I connect A and B, ensure they don’t break (each other) – When I disconnect A, ensure B doesn’t break – When I connect C, don’t change A or B  Scalability (like in any system,  Fault Tolerance but stakes are higher) © 2009 Real-Time Innovations, Inc. 3
  • 4. Part 1: Architecture (we’ll get to technology later) © 2009 Real-Time Innovations, Inc. 4
  • 5. System Component Component Component Class A Modest Proposition © 2009 Real-Time Innovations, Inc. 5 State Behavior Class  Fundamental design principles scale – Abstraction: Provide interface based on relevant concepts – Encapsulation: Hide internal implementation, communication – Composition: Combine existing capabilities into new ones
  • 6. Subsystem 2 P P PP P Subsystem 1 P P PP P Schematic of a Composed System  Subsystems may have different network environments  Integration may have different network environment than subsystems themselves  Data may need to be transformed / cleansed as it moves among subsystems  Routing / gateway services will adapt data types / formats / protocols LAN LAN WAN Router/G ateway Router/G ateway Isolation Additional Governance
  • 7. Schematic of a Composed System  Wash, rinse, repeat Subsystem 1 P P PP P Subsystem 2 P P PP P Router/G ateway Router/G ateway
  • 8. Same data model? Same network env.? Same lifecycle? Behavior unaffected? Understandable? Apples and Oranges P P PP P Subsystem 1 + Subsystem 2 P P PP P Subsystem 2 P P PP P Subsystem 1 P P PP P Router/ Gateway Router/ Gateway
  • 9. Nothing New Under the Sun © 2009 Real-Time Innovations, Inc. 9 Subsystem Data Space Subsystem Data Space Subsystem Data Space Integration Data Space Router/ Gateway Router/ Gateway Router/ Gateway
  • 10. Nothing New Under the Sun © 2009 Real-Time Innovations, Inc. 10 U.S. Combat Data Space Allied Combat Data Space U.S. C4I Data Space Integration Data Space Router/ Gateway Router/ Gateway Router/ Gateway Defense Industry
  • 11. Nothing New Under the Sun © 2009 Real-Time Innovations, Inc. 11 NYC Trading Data Space London Trading Data Space Tokyo Trading Data Space Integration Data Space Router/ Gateway Router/ Gateway Router/ Gateway Financial Services Industry
  • 12. Nothing New Under the Sun © 2009 Real-Time Innovations, Inc. 12 Generation Data Space Substation #1 Data Space Substation #2 Data Space Integration Data Space Router/ Gateway Router/ Gateway Router/ Gateway Power Industry
  • 13. Scalability and Fault Tolerance  Fault tolerance: Router/gateway can’t be single point of failure. 3 answers: – Persistent data survives system failures – Redundancy allows continued service during failure – Segmented/load-balanced configuration limits scope of failures  Scalability: How big can a subsystem be? 1. How big a subsystem can you understand, test, and debug? 2. How well does infrastructure scale? © 2009 Real-Time Innovations, Inc. 13
  • 14. Part 2: DDS © 2009 Real-Time Innovations, Inc. 14
  • 15. How Does DDS Stack Up?  Governance – Structural SLA: data type – Behavioral SLA: delivery QoS – Security – Problem detection/prevention/notification – System monitoring and recording  Isolation – Prevent data “leaks” and side effects – Allow dynamic (dis-|re-)connection – Support independent integration and evolution  Fault tolerance – Data persistence – Segmented/load-balanced routing – Redundant routing  Scalability – WAN connectivity – Peer-to-peer communication – Brokered/managed communication – Built in – Built in – Products available; future standards – Built in – Products available – Built in: domains, partitions – Built in – Built in, esp. w/ DDS-XTypes – Built in – Built in if router uses DDS – Built in if router uses DDS – Products available; future standards – Highly scalable – Protocol support if necessary © 2009 Real-Time Innovations, Inc. 15
  • 16. DDS-RTPS Supports Redundant Data Routing © 2009 Real-Time Innovations, Inc. 16 Write MyData Read MyData' Transform and Forward Router 1 Router 2 Multiple readers: no problem Multiple writers: prevent duplicates! Fortunately, built into DDS-RTPS
  • 17. DDS-RTPS Supports Redundant Data Routing © 2009 Real-Time Innovations, Inc. 17 Read MyData' DDS-RTPS Message Inline QoS Data Inline QoS Orig. Writer Info Other Metadata Original Writer Info Orig. Writer GUID Orig. Writer Seq. # Identity of original writer Identity of original sample Reader discards duplicates based on these
  • 18. DDS-RTPS Supports Redundant Data Routing © 2009 Real-Time Innovations, Inc. 18 Read MyData' Transform and Forward Router 1 Router 2 Writer 5AC2 Data Seq. # 42 Writer 5AC2 Data Seq. # 42 Writer 5AC2 Data Seq. # 42 Writer 5AC2 Data Seq. # 42 Writer 5AC2 Writer 5AC2 Seq. # 42 Seq. # 42 If your impl. doesn’t support, fall back to ownership. Write MyData
  • 19. DDS Scalability: Two Aspects 1. Application data – How does performance fall off as # participants, writers, readers increase? – Any single writer/reader pair can saturate gig-E: achieving high aggregate throughput is not main issue – Interesting issue: Fan-out – number of readers per writer 2. Discovery – How many DDS applications can discover one another? – Interesting issue: How many applications need to discover one another?  Hard limits, if any, very dependent on DDS implementation, machine configuration, network, etc. © 2009 Real-Time Innovations, Inc. 19
  • 20. 1. DDS Scalability: Application Data  DDS-RTPS reliable multicast scales at least: – …to hundreds of readers per writer – …with very little degradation in throughput. – Larger testing facilities welcomed.  © 2009 Real-Time Innovations, Inc. 20 < 15% decrease 1~900 readers RTI Data Distribution Service 4.3 Red Hat EL 5.0, 32-bit 2.4 GHz processors Gigabit Ethernet UDP/IPv4 Reliable ordered delivery
  • 21. 2. DDS Scalability: P2P Discovery Scenarios  Single domain, symmetric discovery – Everyone discovers everyone else – Easiest to configure; most challenging wrt scalability – RTI test results: 1,800 participants; 3.2M endpoint matches  Single domain, asymmetric discovery – Each participant only knows of certain others in its domain – Good for partitioning domains in which not everyone talks  Multiple domains – Greatest separation for subsystems that rarely exchange data • DDS-RTPS maps to different IP ports © 2009 Real-Time Innovations, Inc. 21 1 … n
  • 22. Summary: Large-Scale P2P Scalability  Experimental results: DDS-RTPS allows participant to talk to ≥ 1-2K others in single symmetric domain  Implication: Compose arbitrarily large systems out of subsystems this size or smaller  Large (sub)systems often have requirements for: – Increased governance, e.g. over security boundaries – Data transformation – Protocol mediation – These typically require data brokering anyway © 2009 Real-Time Innovations, Inc. 22
  • 23. DDS Scalability: Stronger Measures Not big enough? 1. Brokered application data – Covered this 2. Brokered discovery © 2009 Real-Time Innovations, Inc. 23 … … m … n n < m m n
  • 24. DDS Scalability: Brokered Discovery  Can be built with DDS-RTPS-standard building blocks © 2009 Real-Time Innovations, Inc. 24 Application Topic Built-In Topic RTPS Metadata Data RTPS Metadata Data Orig. Metadata RTPS Metadata Data RTPS InfoSource submessage provides data forwarding capability: “Treat the following as if it came from that guy instead of from me.”
  • 25. Future Direction: Enhanced Built-in Security  System of systems often have non-uniform trust – Multiple communities of interest w/ different entitlements/authorization – Infrastructure components may (not) be trusted (e.g. brokers, daemons, persistence, recording/playback, etc.) – There is no “system high”  Desirable new specifications: – Fine-grained access control: topic, instance, type member levels – Sample-level tagging and labeling – Fine-grained signing and/or encryption: topic, member level © 2009 Real-Time Innovations, Inc. 25
  • 26. Future Direction: Standardized WAN Support  Site-to-site comms need DDS across WAN(s) – Including untrusted networks – Including firewall, NAT traversal – Challenge: UDP may not be routable, especially if multicast  DDS-RTPS protocol designed for layering atop diverse lower-level protocols – Proven: UDP, TCP, serial, switched fabric, Infiniband, VME, shared memory… – Blessed for interop by OMG: UDP  Additional OMG-recognized protocol layerings can improve site-to-site interop – TCP for WAN routing – (D)TLS for transport-level security (a clarification, not new) – Could be standardized quickly © 2009 Real-Time Innovations, Inc. 26
  • 27. Recap  Large systems usually composed of smaller systems – Developed independently – …With different network environments – …And different data interest/entitlements  Not an accident: hierarchical design best practice at every scale – Subsystems guarded by DDS router/gateway • Restrict access to internal topics/flows • Mediate internal/external protocols • Transform/cleanse internal/external data types • Impose governance: enforce policy, attach monitoring – Inter-router integration space is itself a “subsystem” for further hierarchical composition © 2009 Real-Time Innovations, Inc. 27
  • 28. Recap  DDS uniquely qualified to meet these needs – Strong SLAs for governance – Multiple topologies for fault-tolerant subsystem isolation – High-performance, scalable, deterministic interoperability – What other standard technology offers this combination? (Not JMS. Not CORBA. Not AMQP. Not web services. …)  Promising future directions: – DDS-RTPS/TCP protocol layering for WAN traversal – Enhanced security model • Fine-grained access control • Sample-level tagging, signing, and encrypting © 2009 Real-Time Innovations, Inc. 28
  • 29. Thank You © 2009 Real-Time Innovations, Inc. 29