Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Distribution Service (DDS) in Action Part 2

526 views

Published on

Part 2 of the DDS in Action Presentation. Introduced in 2004, the Data Distribution Service (DDS) has been steadily growing in popularity and adoption. Today, DDS is at the heart of a large number of mission and business critical systems, such as, Air Traffic Control and Management, Train Control Systems, Energy Production Systems, Medical Devices, Autonomous Vehicles, and Smart Cities. Considering the technological trends towards data-centricity and the rate of adoption, tomorrow, DDS will be at the heart of an incredible number of Industrial IoT (IIoT) systems. To help you become an expert in DDS and exploit your skills in the growing DDS market, we have designed the DDS in Action presentation series. This series is a learning journey through which you will (1) discover the essence of DDS, (2) understand how to effectively exploit DDS to architect and program distributed applications that perform and scale, (3) learn the key DDS programming idioms and architectural patterns, (4) understand how to characterize DDS performance and configure for optimal latency/throughput, (5) grow your system to Internet scale, and (6) secure you DDS system.

Published in: Software
  • Be the first to comment

Data Distribution Service (DDS) in Action Part 2

  1. 1. Angelo Corsaro, PhD Chief Technology Officer ADLINK Technologies Inc. angelo.corsaro@adlinktech.com DDS in Action Part-II
  2. 2. CopyrightPrismTech,2015 Recap
  3. 3. CopyrightPrismTech,2015 What is it?
  4. 4. CopyrightPrismTech,2015 DDS is a standard technology for ubiquitous, interoperable, secure, platform independent, and real- time data sharing across network connected devices DDS in 131 characters
  5. 5. CopyrightPrismTech,2015 Who is using it?
  6. 6. CopyrightPrismTech,2015
  7. 7. CopyrightPrismTech,2015 Part II
  8. 8. CopyrightPrismTech,2015 Dynamic Discovery
  9. 9. CopyrightPrismTech,2015 Applications can autonomously and asynchronously read and write data enjoying spatial and temporal decoupling Virtualised Data Space DDS Global Data Space ... Data Writer Data Writer Data Writer Data Reader Data Reader Data Reader Data Reader Data Writer TopicA QoS TopicB QoS TopicC QoS TopicD QoS
  10. 10. CopyrightPrismTech,2015 DDS Dynamic discoveries is responsible for (1) uncovering DDS applications, (2) matching interests and (3) setting up data path between matching readers and writers Dynamic Discovery DDS Global Data Space ... Data Writer Data Writer Data Writer Data Reader Data Reader Data Reader Data Reader Data Writer TopicA QoS TopicB QoS TopicC QoS TopicD QoS
  11. 11. CopyrightPrismTech,2015 Built-in dynamic discovery isolates applications from network topology and connectivity details Dynamic Discovery DDS Global Data Space ... Data Writer Data Writer Data Writer Data Reader Data Reader Data Reader Data Reader Data Writer TopicA QoS TopicB QoS TopicC QoS TopicD QoS
  12. 12. CopyrightPrismTech,2015 Lab 1: Wireshark
  13. 13. CopyrightPrismTech,2015 Decomposing DDS
  14. 14. Information Organisation
  15. 15. Domain • DDS data lives within a domain • A domain is identified with a non negative integer, such as 1, 3, 31 • The number 0 identifies the default domain • A domain represent an impassable communication plane DDS Domain
  16. 16. Partitions • Partitions are the mechanism provided by DDS to organise information within a domain • Access to partitions is controlled through QoS Policies • Partitions are defined as strings: • “system:telemetry” • “system:log” • “data:row-2:col-3” • Partitions addressed by name or regular expressions: • ”system:telemetry” • “data:row-2:col-*” Partition s
  17. 17. Information Definition
  18. 18. Topic • A Topic defines a domain-wide information’s class • A Topic is defined by means of a (name, type, qos) tuple, where • name: identifies the topic within the domain • type: is the programming language type associated with the topic. Types are extensible and evolvable • qos: is a collection of policies that express the non-functional properties of this topic, e.g. reliability, persistence, etc. Topic Type Name QoS
  19. 19. Topic and Instances • As explained in the previous slide a topic defines a class/type of information • Topics can be defined as Singleton or can have multiple Instances • Topic Instances are identified by means of the topic key • A Topic Key is identified by a tuple of attributes -- like in databases • Remarks: • A Singleton topic has a single domain-wide instance • A “regular” Topic can have as many instances as the number of different key values, e.g., if the key is an 8-bit character then the topic can have 256 different instances
  20. 20. Example
  21. 21. Active Floor • Assume we are building an active floor • This active floor is made by a matrix of pressure sensors used to detects position, and indirectly movement • This information is leveraged by the application that uses the active floor for positioning or entertainment Cell: (i,j)
  22. 22. Active Floor • The generic active cell can be modelled with a topic that has an instance for each value of (i,j). The topic type can be defined as: • Each cell is now distinguishable and associated with a topic instance Cell: (i,j) struct TCell { short row; short column; float pressure; // in kPa }; #pragma keylist TCell row column
  23. 23. Active Floor • How can we know when something is on the cell? • The detection can be based on the difference between the atmospheric pressure, say P0, and the pressure sensed by the cell • We can model this as a Singleton Topic ReferencePressure defined by the type: Cell: (i,j) struct TReferencePressure { float pressure; // in kPa float precision; }; #pragma keylist TReferencePressure
  24. 24. Active Floor • Each sensor has associated a topic instance identified by the (row,column) coordinate -- the instance key • Each instance produces a stream of pressure values that in DDS terms are called samples 0 1 2 3 0 1 2 3 4 Pressur e time Pressur e time Pressur e time struct TCell { short row; short column; float pressure; // in kPa }; #pragma keylist Cell row column
  25. 25. Exercice • What if we want to extend our model to deal with floors on different levels? • How would you extend the data model? • How would you use partitions? 1 2 3 Pressur e time Pressur e time Pressur e time
  26. 26. CopyrightPrismTech,2015 Producing Information
  27. 27. Data Writer • A DataWriter (DW) is a strongly typed entity used to produce samples for one or more instances of a Topic, with a given QoS • Conceptually, the DataWriter QoS should be the same as the Topic QoS or more stringent • However, DDS does enforce a specific relationship between the Topic and DataWriter QoS Topic Type Name QoS
  28. 28. Data Writer The DataWriter controls the life-cycle of Topic Instances and allows to: • Define a new topic instance • Write samples for a topic instance • Dispose the topic instance Topic Type Name QoS
  29. 29. The DataWriter controls the life-cycle of Topic Instances and allows to: • Define a new topic instance • Write samples for a topic instance • Dispose the topic instance Topic Type Name QoS Data Writer
  30. 30. Writer Cache Each DDS DataWriter has an associated cache • Writes are always local to the cache • This cache provides two degrees of temporal decoupling between writers and readers. One w.r.t. processing speed the other w.r.t. temporal coupling • The writer cache along with DDS QoS Policies provides built-in support for the Circuit-Breaker pattern DataWriter Cache DataWriter ... Samples Instances Cache
  31. 31. CopyrightPrismTech,2015 Consuming Information
  32. 32. Data Reader • A DataReader (DR) is a strongly typed entity used to access and/or consume samples for a Topic, with a given QoS • Conceptually, the DataReader QoS should be the same as the Topic QoS or less stringent • However, DDS does enforce a specific relationship between the Topic and DataReader QoS Topic Type Name QoS
  33. 33. Reader Cache The Reader Cache stores the last n∊𝜨∞ samples for each relevant instance Where: 𝜨∞ =𝜨 ∪ {∞} DataReader Cache DataReader ... Samples Instances Cache
  34. 34. Data Reader Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created 0 1 2 3 0 1 2 3 4 Pressure time Pressure time Pressure time
  35. 35. 0 1 2 3 0 1 2 3 4 Pressure time Pressure time Pressure time Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created n=3 3
  36. 36. 0 1 2 3 0 1 2 3 4 Pressure time Pressure time Pressure time Data Reader Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created n=3
  37. 37. Data Reader Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created n= 3 0 1 2 3 0 1 2 3 4 Pressure time Pressure time Pressure time
  38. 38. Data Reader Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created n=3 0 1 2 3 0 1 2 3 4 Pressure time Pressure time Pressure time
  39. 39. Data Reader Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created 0 1 2 3 0 1 2 3 4 n=3 Pressure time Pressure time Pressure time
  40. 40. Data Reader Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created 0 1 2 3 0 1 2 3 4 n=3 Pressure time Pressure time Pressure time
  41. 41. Data Reader Depending on its QoS a DataReader may provide access to: • last sample • last n samples • all samples produced since the DataReader was created 0 1 2 3 0 1 2 3 4 n=3 Pressure time Pressure time Pressure time
  42. 42. Data Reader • Samples are stored in the DataReader Cache • Samples can be read or taken from the cache • Samples taken are evicted from the cache • Samples read remain in the cache and are simply market as read • The cache content can be selected based on content or state. More on this later... 0 1 2 3 0 1 2 3 4 Pressure time Pressure time Pressure time
  43. 43. Data Selection
  44. 44. Reading Data • The action of reading samples for a Reader Cache is non-destructive. • Samples are not removed from the cache DataReader Cache DataReader ... DataReader Cache DataReader ...read
  45. 45. Taking Data • The action of taking samples for a Reader Cache is destructive. • Samples are removed from the cache DataReader Cache DataReader ... DataReader Cache DataReader ...take
  46. 46. Samples Selector • Samples can be selected using composable content and status predicates DataReader Cache DataReader ...
  47. 47. Content Filtering • Filters allow to control what gets into a DataReader cache • Filters are expressed as SQL where clauses or as Java/C/JavaScript predicates DataReader Cache DataReader ... Filter Application Network
  48. 48. Content Filters Content Filters can be used to project on the local cache only the Topic data satisfying a given predicate struct CarDynamics { @key string cid; long x; long y; float dx; long dy; } cid x y dx dy GR 33N GO 167 240 45 0 LO 00V IN 65 26 65 0 AN 637 OS 32 853 0 50 AB 123 CD 325 235 80 0 “dx > 50 OR dy > 50” Type CarDynamics cid x y dx dy LO 00V IN 65 26 65 0 AB 123 CD 325 235 80 0 Reader Cache
  49. 49. Content-Based Selection • Queries allow to control what gets out of a DataReader Cache • Queries are expressed as SQL where clauses or as Java/C/JavaScript predicates DataReader Cache DataReader ... Query DataReader Cache DataReader ... Application Network
  50. 50. Queries Queries can be used to select out of the local cache the data matching a given predicate Reader Cache struct CarDynamics { @key string cid; long x; long y; float dx; long dy; } cid x y dx dy GR 33N GO 167 240 45 0 LO 00V IN 65 26 65 0 AN 637 OS 32 853 0 50 AB 123 CD 325 235 80 0 “dx > 50 OR dy > 50” Type CarDynamics cid x y dx dy GR 33N GO 167 240 45 0 LO 00V IN 65 26 65 0 AN 637 OS 32 853 0 50 AB 123 CD 325 235 80 0 cid x y dx dy LO 00V IN 65 26 65 0 AB 123 CD 325 235 80 0 query
  51. 51. Copyright 2013, PrismTech – All Rights Reserved. Sample, Instance, and View State • The samples included in the DataReader cache have associated some meta- information which, among other things, describes the status of the sample and its associated stream/instance • The Sample State (READ, NOT_READ) allows to distinguish between new samples and samples that have already been read • The View State (NEW, NOT_NEW) allows to distinguish a new instance from an existing one • The Instance State (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of the instance to which a sample belongs
  52. 52. State-Based Selection • State based selection allows to control what gets out of a DataReader Cache • State base selectors predicate on samples meta-information DataReader Cache DataReader ... State Selector DataReader Cache DataReader ... Application Network
  53. 53. CopyrightPrismTech,2015 Interaction Models
  54. 54. Copyright 2013, PrismTech – All Rights Reserved. Interaction Models Polling •The application proactively polls for data availability as well as special events, such as a deadline being missed, etc. Notice that all DDS API calls, exclusion made for wait operations, are non-blocking Synchronous Notification •The application synchronously waits for some conditions to be verified, e.g., data availability, instance lifecycle change, etc. Asynchronous Notification •The application registers the interest to be asynchronously notified when specific condition are satisfied, e.g. data available, a publication matched, etc.
  55. 55. Copyright 2013, PrismTech – All Rights Reserved. Synchronous Notifications • DDS provides a mechanism known as WaitSet to synchronously wait for a condition • Condition can predicate on: • communication statuses • data availability • data availability with specific content • user-triggered conditions
  56. 56. Copyright 2013, PrismTech – All Rights Reserved. Asynchronous Notifications • DDS provides a mechanism known as Listeners for asynchronous notification of a given condition • Listener interest can predicate on: • communication statuses • data availability
  57. 57. CopyrightPrismTech,2015 Putting it all together
  58. 58. CopyrightPrismTech,2015 Anatomy of a DDS Application
  59. 59. CopyrightPrismTech,2015 Writing Data in C++ #include <dds.hpp> int main(int, char**) { DomainParticipant dp(0); Topic<Meter> topic(“SmartMeter”); Publisher pub(dp); DataWriter<Meter> dw(pub, topic); while (!done) { auto value = readMeter() dw.write(value); std::this_thread::sleep_for(SAMPLING_PERIOD); } return 0; } enum UtilityKind { ELECTRICITY, GAS, WATER }; struct Meter { string sn; UtilityKind utility; float reading; float error; }; #pragma keylist Meter sn
  60. 60. CopyrightPrismTech,2015 Reading Data in C++ #include <dds.hpp> int main(int, char**) { DomainParticipant dp(0); Topic<Meter> topic(”SmartMeter”); Subscriber sub(dp); DataReader<Meter> dr(dp, topic); LambdaDataReaderListener<DataReader<Meter>> lst; lst.data_available = [](DataReader<Meter>& dr) { auto samples = data.read(); std::for_each(samples.begin(), samples.end(), [](Sample<Meter>& sample) { std::cout << sample.data() << std::endl; } } dr.listener(lst); // Print incoming data up to when the user does a Ctrl-C std::this_thread::join(); return 0; } enum UtilityKind { ELECTRICITY, GAS, WATER }; struct Meter { string sn; UtilityKind utility; float reading; float error; }; #pragma keylist Meter sn
  61. 61. CopyrightPrismTech,2015 Writing Data in Scala import dds._
 import dds.prelude._
 import dds.config.DefaultEntities._
 object SmartMeter {
 
 def main(args: Array[String]): Unit = {
 val topic = Topic[Meter](“SmartMeter”)
 val dw = DataWriter[Meter](topic) while (!done) { val meter = readMeter()
 dw.write(meter) Thread.sleep(SAMPLING_PERIOD) }
 }
 } enum UtilityKind { ELECTRICITY, GAS, WATER }; struct Meter { string sn; UtilityKind utility; float reading; float error; }; #pragma keylist Meter sn
  62. 62. CopyrightPrismTech,2015 Reading Data in Scala import dds._
 import dds.prelude._
 import dds.config.DefaultEntities._
 object SmartMeterLog {
 def main(args: Array[String]): Unit = {
 val topic = Topic[Meter](“SmartMeter”)
 val dr = DataReader[Meter](topic)
 dr listen {
 case DataAvailable(_) => dr.read.foreach(println)
 }
 }
 } enum UtilityKind { ELECTRICITY, GAS, WATER }; struct Meter { string sn; UtilityKind utility; float reading; float error; }; #pragma keylist Meter sn
  63. 63. CopyrightPrismTech,2015 Writing Data in Python import dds import time
 
 if __name__ == '__main__':
 topic = dds.Topic("SmartMeter", "Meter")
 dw = dds.Writer(topic)
 
 while True:
 m = readMeter()
 dw.write(m)
 time.sleep(0.1) enum UtilityKind { ELECTRICITY, GAS, WATER }; struct Meter { string sn; UtilityKind utility; float reading; float error; }; #pragma keylist Meter sn
  64. 64. CopyrightPrismTech,2015 Reading Data in Python import dds
 import sys
 
 def readData(dr): 
 samples = dds.range(dr.read())
 for s in samples:
 sys.stdout.write(str(s.getData()))
 
 if __name__ == '__main__':
 t = dds.Topic("SmartMeter", "Meter")
 dr = dds.Reader(t)
 dr.onDataAvailable = readData enum UtilityKind { ELECTRICITY, GAS, WATER }; struct Meter { string sn; UtilityKind utility; float reading; float error; }; #pragma keylist Meter sn
  65. 65. CopyrightPrismTech,2015 Lab2: Code Walkthrough
  66. 66. CopyrightPrismTech,2015 Performance
  67. 67. Latency Vortex DDS latency can be as low as ~30 usec This is the lowest among DDS implementations several times better than what can be obtained with MQTT, AMQP, OPC-UA
  68. 68. Throughput Vortex DDS can easily saturate a 10Gbps network Vortex throughput is 2-3x better than competing technologies
  69. 69. CopyrightPrismTech,2015 Lab3: Performance Eval.
  70. 70. CopyrightPrismTech,2015 Concluding Remarks
  71. 71. CopyrightPrismTech,2015 DDS provides a very high level abstractions to architect and implement distributed systems DDS has built-in support for several patterns that are essential to keep systems working at scale, such as circuit-breakers and temporal decoupling DDS can address the most challenging environment w.r.t. volumes, latencies and scale Wrapping Up

×