Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Vortex Tutorial -- Part I

2,041 views

Published on

This presentation introduced Vortex by means of a running example. Throughout the presentation we will show how Vortex makes it easy to build a micro-blogging platform a la Twitter.

Published in: Technology

Vortex Tutorial -- Part I

  1. 1. Angelo  Corsaro,  PhD   Chief  Technology  Officer   angelo.corsaro@prismtech.com Vortex Tutorial Part I
  2. 2. CopyrightPrismTech,2014 Vortex enables seamless, ubiquitous, efficient and timely data sharing across mobile, embedded, desktop, cloud and web applications Vortex is based on the OMG DDS standard The Vortex Platform Vortex Device Tools Integration Vortex Cloud MaaS
  3. 3. CopyrightPrismTech,2014 Vortex Architecture
  4. 4. Security Model
  5. 5. CopyrightPrismTech,2014 Support for transport level security via TLS and DTLS Support for fine grained access control Access control plug-in compliant with DDS Security Vortex Security Model Arthur Dent Arthur Dent Ford Prerfect Zaphod Beeblebrox Marvin Trillian A(r,w), B(r) A(r,w), B(r,w), X(r) *(r,w) *(r) A(r,w), B(r,w), C(r,w) Ford Prerfect Zaphod Beeblebrox Trillian Marvin A B A,B X * * A,B,C Identity Access Rights Sessions are authenticated and communication is encrypted Only the Topic included as part of the access rights are visible and accessible
  6. 6. Use Cases
  7. 7. CopyrightPrismTech,2014 Smart Grids Analytics Big  Data  Store
  8. 8. CopyrightPrismTech,2014 E-Health Hospital A Private Cloud Analytics Big  Data  Store Hospital B
  9. 9. CopyrightPrismTech,2014 Smart City http://bit.ly/connected-boulevard
  10. 10. CopyrightPrismTech,2014 SESAR PENS PENS: Pan European Network Service FDP: Flight Data Processor CWP: Controller Working Position IPv6 network SSM support FDP FDPFDP CWP Tower Device FDP FDPFDP Tower Device CWP FDP FDPFDP Tower Device CWP FDP FDPFDP CWP Tower Device EU Nation 4 EU Nation 1 EU Nation 2 EU Nation 3
  11. 11. Building ChirpIt
  12. 12. CopyrightPrismTech,2014 To explore the various features provided by the Vortex platform we will be designing and implementing a micro-blogging platform called ChirpIt. Specifically, we want to support the following features: ChirpIt users should be able to “chirp”, “re-chirp”, “like” and “dislike” trills as well as get detailed statistics The ChirpIt platform should provide information on trending topics — identified by hashtags — as well as trending users Third party services should be able to flexibly access slices of produced trills to perform their own trend analysis ChirpIt should scale to millions of users ChirpIt should be based on a Lambda Architecture ChirpIt Requirements
  13. 13. CopyrightPrismTech,2014 ChirpIt Architecture analytics data centre chirp emotions stats batch     layer chirpit  apps Cloud Messaging serving     layer chirp emotions stats master   dataset view view view chirp emotions stats 3rd  party  svcs Speed     layer
  14. 14. The Data Distribution Service
  15. 15. CopyrightPrismTech,2014 DDS provides a Global Data Space abstraction that allows applications to autonomously, anonymously, securely and efficiently share data DDS’ Global Data Space is fully distributed (decentralised), highly efficient and scalable Data Distribution Service (DDS) DDS Global Data Space ... Data Writer Data Writer Data Writer Data Reader Data Reader Data Reader Data Reader Data Writer TopicA QoS TopicB QoS TopicC QoS TopicD QoS
  16. 16. CopyrightPrismTech,2014 DataWriters and DataReaders are automatically and dynamically matched by the DDS Discovery A rich set of QoS allows to control existential, temporal, and spatial properties of data Data Distribution Service (DDS) DDS Global Data Space ... Data Writer Data Writer Data Writer Data Reader Data Reader Data Reader Data Reader Data Writer TopicA QoS TopicB QoS TopicC QoS TopicD QoS
  17. 17. CopyrightPrismTech,2014 Elegant and High Level Data Sharing Abstraction Polyglot and platform independent • Java, Scala, C, C++, C#, JavaScript, CoffeeScript etc. • Android, Windows, Linux, VxWorks, etc. Peer-to-Peer by nature, Brokered when useful Key Highlights
  18. 18. CopyrightPrismTech,2014 Content and Temporal Filtering (both sender and receiver filtering supported) Queries 20+ QoS to control existential, temporal, and spatial properties of data Key Highlights
  19. 19. CopyrightPrismTech,2014 High Performance and Scalable • ~30 usec latency* • 7M msgs/sec node-to-node throughput* Key Highlights (*) Performance measured on Linux Host with i7 processor on a 1Gbps Ethernet network
  20. 20. Decomposing DDS
  21. 21. Information Organisation
  22. 22. CopyrightPrismTech,2014 DDS data lives within a domain A domain is identified with a non negative integer, such as 1, 3, 31 The number 0 identifies the default domain A domain represent an impassable communication plane Domain DDS Domain
  23. 23. CopyrightPrismTech,2014 Partitions are the mechanism provided by DDS to organise information within a domain Access to partitions is controlled through QoS Policies Partitions are defined as strings: - “system:telemetry” - “system:log” - “data:row-2:col-3” Partitions addressed by name or regular expressions: - ”system:telemetry” - “data:row-2:col-*” Partitions Partitions
  24. 24. Information Definition
  25. 25. CopyrightPrismTech,2014 A Topic defines a domain-wide information’s class A Topic is defined by means of a (name, type, qos) tuple, where • name: identifies the topic within the domain • type: is the programming language type associated with the topic. Types are extensible and evolvable • qos: is a collection of policies that express the non-functional properties of this topic, e.g. reliability, persistence, etc. Topic Topic Type Name QoS ... TopicA QoS TopicB QoS TopicC QoS TopicD QoS
  26. 26. CopyrightPrismTech,2014 As explained in the previous slide a topic defines a class/type of information Topics can be defined as Singleton or can have multiple Instances Topic Instances are identified by means of the topic key A Topic Key is identified by a tuple of attributes -- like in databases Remarks: - A Singleton topic has a single domain-wide instance - A “regular” Topic can have as many instances as the number of different key values, e.g., if the key is an 8-bit character then the topic can have 256 different instances Topic and Instances
  27. 27. ChirpIt Data Model
  28. 28. CopyrightPrismTech,2014 Chirp Actions      union  ChirpBody  switch  (ChirpActionKind)  {          case  CHIRP_KIND:              string<128>  chirp;          case  RECHIRP_KIND:              string<128>  chirp;              string  user;          case  LIKE_KIND:              string  luser;          case  DISLIKE_KIND:              string  duser;          };   !                  struct  ChirpAction  {              ChirpHeader  header;              ChirpBody      body;          };          #pragma  keylist  ChirpAction  header.id.cid          struct  ChirpId  {              string  uid;              string  cid;          };   !        enum  ChirpActionKind  {              CHIRP_KIND,  RECHIRP_KIND,                LIKE_KIND,  DISLIKE_KIND          };                    struct  ChirpHeader  {              ChirpId  id;              Location  location;              unsigned  long  long  timestamp;              ChirpActionKind  kind;          };            
  29. 29. CopyrightPrismTech,2014 ChirpIt Statistics        struct  ChirpStats  {              ChirpId  id;              unsigned  long  rechirps;              unsigned  long  likes;          };          #pragma  keylist  ChirpStats  id.cid          struct  UserStats  {              string  userId;              unsigned  long  followers;              unsigned  long  chirps;              unsigned  long  followed;          };          #pragma  keylist  UserStats  userId  
  30. 30. CopyrightPrismTech,2014 User Actions        struct  enum  UserActionKind  {              FOLLOW,  UNFOLLOW          };                    struct  UserAction  {              UserActionKind  kind;              string  fromUser;              string  toUser;          };  
  31. 31. CopyrightPrismTech,2014 App Config        typedef  sequence<string>  Partitions;   !        struct  AppConfig  {              string  userId;              Partitions  ps;              unsigned  short  history;          };              #pragma  keylist  UserConfig  userId  
  32. 32. ChirpIt Information Scoping
  33. 33. CopyrightPrismTech,2014 To make the system scale properly we need to ensure that the chirps, rechirps, etc., are distributed only to the interested parties As we have a single topic representing chirp actions, how can we ensure that information is received only by followers? DDS Provides two mechanisms: - Partitions: Allow to create information scopes - Content Filters: Allow to receive the data that satisfy certains predicates Scoping Information
  34. 34. CopyrightPrismTech,2014 ChirpIt Architecture ChirpAction @wolverine @drx chirp:wolverine ChirpAction @drx  follows  @wolverine
  35. 35. CopyrightPrismTech,2014 @drx ChirpAction ChirpIt Architecture data centre ChirpAction @wolverine chirp:wolverine ChirpAction @magneto chirp:magneto @drx  follows  @wolverine  and  @magneto
  36. 36. CopyrightPrismTech,2014 In summary partitions are used to scope information Each user will “join” a partition for each followed party Example: - If @drx follows @magneto, @wolverine, @cyclops and @mistique then he will receive ChirpActions from the partitions: • chirp:@magneto, chirp:@wolverine, chirp:@cyclops and chirp:@mistique - On the other hand, regardless of the people that follows @wolverine he will always and only write ChirpActions in the partitions • chirp:@wolverine Using Partitions to Scope Chirps
  37. 37. Producing Information
  38. 38. CopyrightPrismTech,2014 A DataWriter (DW) is a strongly typed entity used to produce samples for one or more instances of a Topic, with a given QoS Conceptually, the DataWriter QoS should be the same as the Topic QoS or more stringent However, DDS does enforce a specific relationship between the Topic and DataWriter QoS DataWriter DW Type Topic QoS
  39. 39. CopyrightPrismTech,2014 The DataWriter controls the life-cycle of Topic Instances and allows to: Define a new topic instance Write samples for a topic instance Dispose the topic instance DataWriter DW Type Topic QoS
  40. 40. Consuming Information
  41. 41. CopyrightPrismTech,2014 A DataReader (DR) is a strongly typed entity used to access and/or consume samples for a Topic, with a given QoS Conceptually, the DataReader QoS should be the same as the Topic QoS or less stringent However, DDS does enforce a specific relationship between the Topic and DataReader QoS DataReader DR Type Topic QoS
  42. 42. Putting all Together
  43. 43. CopyrightPrismTech,2014 DomainParticipant: Provides access to a data cloud -- called a domain in DDS Topic: Domain-wide definition of a kind of Information Publisher/Subscriber: Provide scope to data sharing through the concept of partitions DataReader/DataWriter: Allow to read/write data for a given topic in the partitions their Subscriber/Publisher are associated with. DDS Entities Domain (e.g. Domain 123) Domain Participant Topic Publisher DataWrter Subscriber DataReader Partition (e.g. “Telemetry”, “Shapes”, ) Topic Instances/Samples Ta Tb Tc Tx Ty T1 T1 T3
  44. 44. ChirpIt Implementation — Chirping —
  45. 45. CopyrightPrismTech,2014 ChirpOut App            //  Domain  Participant  defined  by  dependency  injection                            //  Define  the  topic              val  topic  =  Topic[ChirpAction](Config.ChirpActionTopic)              //  Define  the  publisher  and  link-­‐it  up  to  the  “chirp:@user”  partition              val  pubQos  =  PublisherQos()                        .withPolicy(Partition(Config.ChirpRootPartition  +  user))              implicit  val  pub  =  Publisher(pubQos)              //  Define  the  data  writer  and  make  it  reliable  and  for  the  time  being              //  Transient  Local…  Will  refined  this  in  the  Part  II              val  dwQos  =  DataWriterQos().withPolicies(                  Reliability.Reliable,                  Durability.TransientLocal)              //  Create  the  DataWriter              val  dw  =  DataWriter[ChirpAction](pub,  topic,  dwQos)              //  Chirp!!!              val  chirp  =  …              dw.write(chirp)
  46. 46. CopyrightPrismTech,2014 ChirpIn            //  Domain  Participant  defined  by  dependency  injection                            //  Define  the  topic              val  topic  =  Topic[ChirpAction](Config.ChirpActionTopic)              //  Define  the  subscribe  and  link-­‐it  up  to  the  partition  associated  to              //  the  user  followed              val  subQos  =  PublisherQos()                        .withPolicy(Partition(followerList.map(Config.ChirpRootPartition  +  _))              implicit  val  sub  =  Subscriber(subQos)              //  Define  the  data  reader  and  make  it  reliable  and  for  the  time  being              //  Transient  Local…  Will  refined  this  in  the  Part  II              val  drQos  =  DataReaderQos().withPolicies(                  Reliability.Reliable,                  Durability.TransientLocal)              //  Create  the  DataWriter              val  dr  =  DataReader[ChirpAction](sub,  topic,  drQos)  
  47. 47. CopyrightPrismTech,2014 ChirpIn //  React  to  new  Chirp   dr  listen  {        case  DataAvailable(_)  =>  {                      //  take  vs.  read  will  become  clear  in  Part  II                      val  samples  =  dr.take()                    //  For  the  time  being  simply  print-­‐out  the  new  chirps                      samples         .filter(s  =>  (s.getData  !=  null))                        .foreach(s  =>  displayChirp)                  }     }
  48. 48. CopyrightPrismTech,2014 ChirpOut App // Define the topic — assuming the topic is registered with the server…! topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)! // Define the data writer Qos! dwQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(userPartition))! ! // Use an Option monad to maintain the writer! chirpWriter = z_.None! ! // Create the writer when the runtime is connected! runtime.onconnect(() ->! dw = new dds.DataWriter(runtime, topic, dwQos)! chirpWriter = z_.Some(dw)! )! ! // …! // Write chirps! chirp = …! chirpWriter.map((dw) -> dw.write(chirp))!
  49. 49. CopyrightPrismTech,2014 ChirpIn App // Define the topic — assuming the topic is registered with the server…! topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)! // Define the data reader Qos! drQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(followedList))! ! // Use an Option monad to maintain the reader! chirpReader = z_.None! ! // Create the writer when the runtime is connected! runtime.onconnect(() ->! dr = new dds.DataReader(runtime, topic, dwQos)! // Attach a listener! dr.addListener((chirp) -> diplayChirp(chirp))! chirpReader = z_.Some(dr)! )
  50. 50. CopyrightPrismTech,2014 ChirpIt Thus Far… data centre chirp emotions stats chirp emotions stats chirp emotions stats chirp emotions stats
  51. 51. Live Demo
  52. 52. CopyrightPrismTech,2014 $ ./chirpOut @wolverin chirp>> @magneto will win!$ ./chirpOut @magneto @magneto>> @wolverine chirping while asleep… again! Download the package zip or tar.gz package available at: - https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.zip - https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.tar.gz Extract the package somewhere in your file-system and open a terminal at that location and run the app as shown below ChirpIt Demo $ ./chirpIn @magneto @wolverinse @mystique @wolverin>> @magneto will win! @magneto>> @wolverine chirping while asleep… again!
  53. 53. CopyrightPrismTech,2014 In Part II we extend our implementation to manage application configuration storage and analytics To do so, we will need to introduce additional features provided by the DDS and the Vortex Platform, such as Durability and Integration with Big-Data Stores What’s Next
  54. 54. CopyrightPrismTech,2014 In this presentation we have performed the first step toward implementing a micro-blogging platform The combination of Vortex Device and Vortex Cloud made it very easy to create and deploy our Internet-Scale, multi-device micro-blogging platform Some of the DDS QoS, such as Durability have already shown how historical data, i.e. chirps made while we were offline — can be provided by the platform The Part II will illustrate how the complete application fits together and what is the role of Vortex in the Lambda Architecture Summary
  55. 55. CopyrightPrismTech,2014 Online Resources

×