• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Got Big Data? Get OpenSplice!
 

Got Big Data? Get OpenSplice!

on

  • 1,338 views

The past few years have witnessed a tremendous increase in the amount of real-time data that applications in domains, such as, web analytics, social media, automated trading, smart cities, smart ...

The past few years have witnessed a tremendous increase in the amount of real-time data that applications in domains, such as, web analytics, social media, automated trading, smart cities, smart grids, etc., have to deal with. The challenges faced by these applications, commonly called Big Data Applications, are manifold as the staggering growth in volumes is complicating the collection, storage, analysis and distribution of data.
In this Webcast we focus our attention on the challenges tied to the collection and distribution of the large volumes of data characteristic of Big Data applications – the first and last stage on the pipeline shown in the Figure below – and partly on its storage.

Big data applications have to be capable of collecting as well as distributing massive amounts of data, much of which needs to be timely processed. As a result these applications need to optimally exploit networking as well as computing resources. Some of all of this data also need to be saved on Big Data stores for further analysis, thus an effective liaison has to be created between the data distribution technology and the storage technology.

This webcast explain the problems that OpenSplice DDS can solve for Big Data applications, it will introduce some specific extensions designed for dealing with throughputs of several millions of messages per second, and finally will show how OpenSplice DDS can be integrated with Big Data Stores, such as Cassandra and HBase.

Statistics

Views

Total Views
1,338
Views on SlideShare
1,310
Embed Views
28

Actions

Likes
4
Downloads
0
Comments
0

4 Embeds 28

https://twitter.com 14
http://icorsaro.net 7
http://www.icorsaro.net 4
http://www.linkedin.com 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Got Big Data? Get OpenSplice! Got Big Data? Get OpenSplice! Presentation Transcript

    • Got Big Data? Get OpenSplice DDS!OpenSplice DDS Angelo CORSARO, Ph.D. Chief Technology Officer OMG DDS Sig Co-Chair PrismTech angelo.corsaro@prismtech.com
    • Defining Big DataOpenSplice DDS
    • Big Data 3Vs (pull, k/v, small) (push, k/v, small) Copyright  2011,  PrismTech  –    All  Rights  Reserved. (pull, k/v, big) (push, k/v, big) Big Data is best understood by looking at the 3Vs Cube varietyOpenSplice DDS velocity e ol um (pull, pk/fk, small) (push, pk/fk, small) v (pull, pk/fk, big) (push, pk/fk, big)
    • OpenSplice DDS Big Data Applications Collect | Store | Organize | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved.
    • Big Data Applications Collect | Store | Organize | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Collect the data from the cyber- physical worldOpenSplice DDS ☐ Depending on applications this could be: ☐ Sensor data ☐ Market Data ☐ Web page statistics ☐ ...
    • Big Data Applications Collect | Store | Organize | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Store the data for on-line and off-line processingOpenSplice DDS
    • Big Data Applications Collect | Store | Organize | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Validate, Correct and Enrich DataOpenSplice DDS
    • Big Data Applications Collect | Store | Organize | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Make sense of the data ☐ Detect short term / long term trendsOpenSplice DDS ☐ ...
    • Big Data Applications Collect | Store | Organize | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Distribute Analytics -- or any other kind of clues about the data -- to applications thatOpenSplice DDS are supposed to act, display, publish, store, etc.
    • Big Data AnalyticsOpenSplice DDS
    • What is Big Data Analytics? Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Big Data Analytics is the process of examining large amounts of data to uncover hidden patterns, unknown correlations and other useful information.OpenSplice DDS ☐ Such information can provide competitive advantages over rival organizations and result in business benefits, such as more effective marketing and increased revenue. [quoted from; Big Data Analytics]
    • Traditional Big Data Analytics Collect | Store | ... | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved.OpenSplice DDS Messaging / Big Data Store Map Reduce Messaging / Web Svcs (e.g. HBase) (e.g. Hadoop) Web Svcs
    • In-Memory Analytics ☐ RAM declining costs and widespread availability of 64-bit processors has created Copyright  2011,  PrismTech  –    All  Rights  Reserved. the opportunity for maintaining large sets of data in main memory. [Today 64-bit machines with up to 1TB of RAM can be easily found] ☐ In-Memory Analytics exploits the availability of RAM to maintain some -- or all -- the Big Data in memory so to reduce query response times and allow fasterOpenSplice DDS Business Intelligence (BI) and analytics decisions. ☐ In-Memory Analytics can reduce or eliminate the need for data indexing and allows more effective implementation of BI and analytic applications. Analysts anticipate that as BI and analytic applications will embrace in-memory analytics, traditional data stores may eventually be used only for data that is not queried frequently.
    • In-Memory Big Data Analytics Collect | Store | ... | Analyze | Share Copyright  2011,  PrismTech  –    All  Rights  Reserved.OpenSplice DDS Data Window Real-time Analytics Big Data In-Memory Messaging / In-Memory Store Analitics Messaging / Web Svcs Full Data Set Mid/Long Term Predictions Web Svcs Big Data Store Map Reduce (e.g. HBase) (e.g. Hadoop)
    • In-Memory Big Data Analytics Data Window Real-time Analytics Copyright  2011,  PrismTech  –    All  Rights  Reserved. Big Data In-Memory Messaging / In-Memory Store Analitics Web SvcsOpenSplice DDS Messaging / Messaging / Web Svcs Web Svcs Model Parameters Full Data Set Mid/Long Term Predictions etc. Big Data Store Map Reduce (e.g. HBase) (e.g. Hadoop)
    • How about the Cube? (pull, k/v, small) (push, k/v, small) Copyright  2011,  PrismTech  –    All  Rights  Reserved. (pull, k/v, big) (push, k/v, big) varietyOpenSplice DDS velocity e ol um (pull, pk/fk, small) (push, pk/fk, small) v (pull, pk/fk, big) (push, pk/fk, big)
    • In-Memory Big Data Analytics Data Window (_, _ , big) Real-time Analytics (push, _ , big) Copyright  2011,  PrismTech  –    All  Rights  Reserved. Big Data In-Memory Messaging / (push, _ , big) In-Memory Store Analitics Web Svcs (push, _ , big)OpenSplice DDS Messaging / Messaging / Web Svcs Web Svcs Model Parameters Full Data Set Mid/Long Term Predictions (push, _ , big) etc. Big Data Store Map Reduce (e.g. HBase) (e.g. Hadoop) (pull, k/v, big)
    • In-Memory Big Data Analytics with OpenSplice DDSOpenSplice DDS
    • OpenSplice & Big Data Analytics Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ OpenSplice DDS can be used to address several Big Data -- specifically In-Memory Big Data -- Analytics applications ☐ OpenSplice’s key features of interest are:OpenSplice DDS ☐ Ultra-High Performance In-Memory Store ☐ Ultra- High Performance Data Distribution ☐ Ultra- High Performance Persistent Store ☐ High Availability ☐ Connectivity (e.g. HBase, CouchDB, REST, WebSockets, etc.)
    • OpenSplice & Big Data Analytics Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ OpenSplice DDS can be used to address several Big Data, and more specifically In-Memory Big Data, Analytics applications ☐ The key features OpenSplice can provide are:OpenSplice DDS ☐ Ultra-High Performance In-Memory Store ☐ Ultra-High Performance Data Distribution ☐ Ultra- High Performance Persistent Store ☐ High Availability ☐ Connectivity (e.g. HBase, CouchDB, REST, WebSockets, etc.)
    • OpenSplice DDS D D BaS ics
    • Data Distribution Service For Real-Time Systems DDS provides a Topic-Based Publish/ Copyright  2011,  PrismTech  –    All  Rights  Reserved. Subscribe abstraction based on: Data Reader Data Writer ☐ Topics: data distribution subject’sOpenSplice DDS Data Reader Data TopicD Writer DataWriters: data producers TopicA ☐ Data TopicB Reader Data Writer ☐ DataReaders: data consumers TopicC ... Data Data Writer Reader DDS Global Data Space
    • Data Distribution Service For Real-Time Systems Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ DataWriters and DataReaders are automatically and Data Reader dynamically matched by the Data Writer DDS Dynamic DiscoveryOpenSplice DDS Data Reader Data TopicD Writer TopicA ☐ A rich set of QoS allows to Data Reader TopicB control existential, temporal, Data Writer TopicC ... and spatial properties of data Data Data Writer Reader DDS Global Data Space
    • OpenSplice DDS DDS Entities
    • Domain Participant Domain (e.g. Domain 123) Domain Participant Copyright  2011,  PrismTech  –    All  Rights  Reserved. Topic Partition (e.g. “Telemetry”, “Shapes”, ) T1 Publisher T1 T3 Subscriber Topic Instances/Samples TxOpenSplice DDS Ty Tb Ta DataWrter DataReader Tc ☐ A DomainParticipant provides access to a DDS Domain ☐ DDS Domains are identified by natural numbers, such as 0, 1, 2, ... ☐ Each DDS Domain has associated some unique communication and discovery endpoints, automatically deduced by the Domain ID
    • OpenSplice DDS Topics
    • DDS Topics “Circle”, “Square”, “Triangle”, ... ☐ A Topic defines a class of streams, Copyright  2011,  PrismTech  –    All  Rights  Reserved. also called topic instances Name ☐ A Topic has associated a unique Topic name, a user defined extensible Typ SOpenSplice DDS type and a set of QoS policies DURABILITY, Qo e DEADLINE, ShapeType ☐ QoS Policies capture the Topic PRIORITY, … non-functional invariants struct ShapeType { ☐ Topics can be discovered or @Key string color; locally defined long x; long y; long shapesize; };
    • Topic Instances ☐ Each unique key value Copyright  2011,  PrismTech  –    All  Rights  Reserved. Instances Instances identifies a unique stream of data color =”Green” DDS not only struct ShapeType { ☐ color =”red” @Key string color; TopicOpenSplice DDS long x; long y; demultiplexes “streams” color = “Blue” long shapesize;}; but provides also lifecycle information ☐ A DDS DataWriter can write multiple instances
    • Anatomy of a DDS ApplicationOpenSplice DDS
    • Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Publisher Topic SubscriberOpenSplice DDS DataWriter DataReader
    • Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Session // Create a Topic val topic = Topic[ShapeType](dp, “Circle”) Publisher Topic SubscriberOpenSplice DDS // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) DataWriter DataReader
    • Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Session // Create a Topic val topic = Topic[ShapeType](dp, “Circle”) Publisher Topic SubscriberOpenSplice DDS // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types DataWriter DataReader // Create a DataWriter/DataWriter val writer = DataWriter[ShapeType](pub, topic) Reader/Writer for val reader = DataReader[ShapeType](sub, topic) application defined Topic Types
    • Anatomy of a DDS Application [Scala API] Domain Domain Copyright  2011,  PrismTech  –    All  Rights  Reserved. val dp = DomainParticipant(domainId) Participant Session // Create a Topic val topic = Topic[ShapeType](dp, “Circle”) Publisher Topic SubscriberOpenSplice DDS // Create a Publisher / Subscriber val pub = Publisher(dp) val sub = Subscriber(dp) Reader/Writers for User Defined for Types DataWriter DataReader // Write data val data = new ShapeType(“RED”, 131, 107, 75) writer write data Reader/Writer for // But you can also write like this... application defined writer ! data Topic Types // Read new data and print it on the screen (reader read) foreach (prinln)
    • Data Reader Caches Your In-Memory Big Data StoreOpenSplice DDS
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches write( ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Topic Instance Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Topic Instance Topic Instance Network
    • Reader/Writer Caches write( ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Writer History Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Reader History Reader History Network
    • Reader/Writer Caches write( ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache QoS Policies ‣ History ‣ Destination Order Network ‣ Presentation ‣ Partition QoS Policies ‣ Time Based Filter ‣ Reliability ‣ Durability ‣ Ownership ‣ History ‣ Transport Priority ‣ Latency Budget ‣ Time Based Filter
    • Reader Cache Properties ☐ An entry for each topic instance Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Each cache entry might contain one or more samples, the number of samples is controlled by the History QoS Policy DataReader ☐ History.KeepLast(n) keeps the last n samplesOpenSplice DDS per instance ... ☐ History.KeepAll maintains all the samples, modulo resource constraints ☐ For a given instance: DataReader Cache ☐ Samples from the same writer are inserted in write-order ☐ Samples from multiple-writers are ordered using time-stamps (source or reception depending on configurable a QoS Policy)
    • Understanding HistoryOpenSplice DDS
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches write( 1 ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache 1 Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches write( 2 ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 1 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 ...OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache 2 Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 2 ... 2OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches write( 3 ) Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 2 1 1 2 ... 2OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 3 1 1 2 ... 2OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 3 1 1 2 ... 2OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache 3 Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Reader/Writer Caches Application Application Application Copyright  2011,  PrismTech  –    All  Rights  Reserved. DataWriter DataReader DataReader 3 2 2 3 ... 3OpenSplice DDS DataWriter Cache DataReader Cache DataReader Cache Network Assumptions: DataReader configured with history depth of 2 and DataWriter with default history depth of 1
    • Interacting with the DataReader CacheOpenSplice DDS
    • Reading Data Samples ☐ Samples can be read from the Data Reader History Cache Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The action of reading a sample is non-destructive. Samples are not removed from the cacheOpenSplice DDS DataReader DataReader ... read ... DataReader Cache DataReader Cache
    • Taking Data Samples ☐ Samples can be taken from the Data Reader History Cache Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The action of taking a sample is destructive. Samples are removed from the cacheOpenSplice DDS DataReader DataReader ... take ... DataReader Cache DataReader Cache
    • Read vs. Take Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The read operation should always be access the latest know value for topics that represent distributed stateOpenSplice DDS ☐ The take operation should be used to get the last notification from a topic that represent an event
    • Data SelectorsOpenSplice DDS
    • Cherry Picking in DDS Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ DDS provides some very flexible mechanisms for selecting the data to be read:OpenSplice DDS ☐ Data Content ☐ Data Status ☐ These mechanisms are composable
    • Content-Based Data SelectionOpenSplice DDS
    • Filters and Queries Application ☐ DDS Filters allow to control what gets into a DataReader cache Copyright  2011,  PrismTech  –    All  Rights  Reserved. Query ☐ DDS Queries allow to control what gets out of a DataReader cache DataReader Filters are defined by means ofOpenSplice DDS ☐ ... ContentFilteredTopics ... ... ... ☐ Queries operate in conjunction with DataReader Cache read operations Filter ☐ Filters and Queries are expressed as SQL where clauses
    • State-Based SelectionOpenSplice DDS
    • Sample, Instance, and View State ☐ The samples included in the DataReader cache have associated Copyright  2011,  PrismTech  –    All  Rights  Reserved. some meta-information which, among other things, describes the status of the sample and its associated stream/instance ☐ The Sample State (READ, NOT_READ) allows to distinguish betweenOpenSplice DDS new samples and samples that have already been read ☐ The View State (NEW, NOT_NEW) allows to distinguish a new instance from an existing one ☐ The Intance State (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of the instance to which a sample belongs
    • List ComprehensionOpenSplice DDS
    • List Comprehension in DDS Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ When using the Scala API, the DDS readers expose data as LinearSeqOptimized thus making possible list comprehension to address quite a few taskOpenSplice DDS ☐ Example ☐ dr read foreach (println) ☐ dr read map (transformer) ☐ ...
    • Examples Compute the average shape and write it as a Square sample: Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ val data = dr history val s0 = new ShapeType(“GRAY”, 0, 0 ,0) dw ! scaleShape((data : s0) (sumShape), data.size)OpenSplice DDS ☐ Filter samples: (dr read) filter (_.x <= 100 && _.y <= 200)
    • High Performance Persistent Store with OpenSpliceOpenSplice DDS
    • OpenSplice Durability ☐ OpenSplice DDS provides a highly available high-performance Copyright  2011,  PrismTech  –    All  Rights  Reserved. persistent store optimized for storing DDS data ☐ This store can be queries to retrieve data using, content as well as temporal filtersOpenSplice DDS ☐ Ex: val topic = Topic[ShapeType](dp, “Circle”) val dr = DataReader[ShapeType](topic) // ... val query = Query(“x < %0 AND y < %1”, List(“200”, “300”)) val t0 = new Time(...) val t1 = new Time(...) // Retrieve all data for the Circle Topic that matches // the query and has timestamp within [t0, t1] dr waitForHistoricalData(query, t0, t1)
    • OpenSplice DDS Connectivity
    • OpenSplice Gateway Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ The OpenSplice Gateway supports Apache Camel Connectors for Big Data Stores thus making it very simple to stream data to/from HBase, CouchDB, etc.OpenSplice DDS ☐ Example: val circleURI = "ddsi:Circle:0/org.opensplice.demo.ShapeType” val hbaseURI = "hbase:ishapes?mappingStrategyName=body&operation=CamelHBasePut" // ... // Put incoming circles into an HBase Table circleURI unmarshal(cdrData) process { e2d (_, "circle") } to(hbaseURI)
    • Putting it All TogetherOpenSplice DDS
    • iShapes In-Memory Analytics Real-time Analytics Display Data Window (_, _ , big) Real-time Analytics Copyright  2011,  PrismTech  –    All  Rights  Reserved. OpenSplice In-Memory Reader Cache Analitics (push, _ , big)OpenSplice DDS OpenSplice JMS Full Data Set XM ST PP RE OpenSplice Gateway P Cu AMQ sto m JMS DDSI-RTPS XM ST PP RE OpenSplice Gateway P Cu AMQ sto m DDSI-RTPS
    • Shapes to HBase Tables struct ShapeType { Copyright  2011,  PrismTech  –    All  Rights  Reserved. @Key string color; long x; create ishapes, color long y; long shapesize; };OpenSplice DDS ☐ Shapes are mapped into the HBase ‘ishapes’ ☐ The row key is the topic name, e.g ‘circle’, ‘square’, etc. ☐ The ‘color” column family is used to distinguish about different instances of a given shape ☐ Different samples for a given instance are distinguished using HBase versions
    • OpenSplice DDS Shapes to HBase Copyright  2011,  PrismTech  –    All  Rights  Reserved.
    • OpenSplice DDS Versions in HBase Copyright  2011,  PrismTech  –    All  Rights  Reserved.
    • Scale Out Techniques Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Scale out in OpenSplice DDS is achieved through data partitioning ☐ OpenSplice DDS provides two orthogonal mechanisms for dataOpenSplice DDS partitioning ☐ Data Partitions ☐ Content Filters
    • OpenSplice DDS Live Demo
    • Concluding Remarks ☐ OpenSplice DDS provide Big Data Applications with ☐ Ultra-High Performance In-Memory Big Data Store Copyright  2011,  PrismTech  –    All  Rights  Reserved. ☐ Ultra-High Performance Data Distribution ☐ High Performance Persistent Store ☐ Connectivity to Big Data StoresOpenSplice DDS ☐ OpenSplice DDS is an ideal fit for In-Memory Big Data Analytics Applications OpenSplice DDS Collect Store | Organize | Analyze Share OpenSplice DDS
    • OpenSplice DDS
    • References Copyright  2011,  PrismTech  –    All  Rights  Reserved. OpenSplice | DDS Escalier ¥ #1 OMG DDS Implementation ¥ Fastest growing JVM Language ¥ Scala API for OpenSplice DDS ¥ Open Source ¥ Open Source ¥ Open Source ¥ www.opensplice.org ¥ www.scala-lang.org ¥ github.com/kydos/escalierOpenSplice DDS ¥ Simple C++ API for DDS ¥ DDS-PSM-Java for OpenSplice DDS ¥ DDS-based Advanced Distributed ¥ Open Source ¥ Open Source Algorithms Toolkit ¥ github.com/kydos/simd-cxx ¥ github.com/kydos/simd-java ¥ Open Source ¥ github.com/kydos/dada
    • :: Connect with Us :: ¥opensplice.com ¥forums.opensplice.org ¥@acorsaro ¥opensplice.org ¥opensplicedds@prismtech.com ¥@prismtechOpenSplice DDS ¥ crc@prismtech.com ¥sales@prismtech.com ¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro