Got Big Data? Get OpenSplice!

Got Big Data?
Get OpenSplice DDS!
OpenSplice DDS

Angelo CORSARO, Ph.D.
Chief Technology Oﬃcer
OMG DDS Sig Co-Chair
PrismTech
angelo.corsaro@prismtech.com

Deﬁning Big Data
OpenSplice DDS

Big Data 3Vs
(pull, k/v, small) (push, k/v, small)

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
(pull, k/v, big) (push, k/v, big)

Big Data is best understood
by looking at the 3Vs Cube

variety
OpenSplice DDS

velocity
e
ol um (pull, pk/fk, small) (push, pk/fk, small)
v

(pull, pk/fk, big) (push, pk/fk, big)

OpenSplice DDS

Big Data Applications
Collect | Store | Organize | Analyze | Share

Copyright
2011,
PrismTech
–

All
Rights
Reserved.


Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Collect the data from the cyber-
physical world
OpenSplice DDS

☐ Depending on applications this
could be:
☐ Sensor data
☐ Market Data
☐ Web page statistics
☐ ...


Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Store the data for on-line and off-line
processing
OpenSplice DDS


Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Validate, Correct and Enrich Data
OpenSplice DDS


Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Make sense of the data

☐ Detect short term / long term trends
OpenSplice DDS

☐ ...


Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Distribute Analytics -- or any
other kind of clues about the
data -- to applications that
OpenSplice DDS

are supposed to act, display,
publish, store, etc.

Big Data Analytics
OpenSplice DDS

What is Big Data Analytics?

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Big Data Analytics is the process of examining large amounts of
data to uncover hidden patterns, unknown correlations and other
useful information.
OpenSplice DDS

☐ Such information can provide competitive advantages over rival
organizations and result in business benefits, such as more effective
marketing and increased revenue.

[quoted from; Big Data Analytics]

Traditional Big Data Analytics
Collect | Store | ... | Analyze | Share

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
OpenSplice DDS

Messaging / Big Data Store Map Reduce Messaging /
Web Svcs (e.g. HBase) (e.g. Hadoop) Web Svcs

In-Memory Analytics
☐ RAM declining costs and widespread availability of 64-bit processors has created

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
the opportunity for maintaining large sets of data in main memory. [Today 64-bit
machines with up to 1TB of RAM can be easily found]

☐ In-Memory Analytics exploits the availability of RAM to maintain some -- or all --
the Big Data in memory so to reduce query response times and allow faster
OpenSplice DDS

Business Intelligence (BI) and analytics decisions.

☐ In-Memory Analytics can reduce or eliminate the need for data indexing and
allows more effective implementation of BI and analytic applications. Analysts
anticipate that as BI and analytic applications will embrace in-memory analytics,
traditional data stores may eventually be used only for data that is not queried
frequently.

In-Memory Big Data Analytics
Collect | Store | ... | Analyze | Share

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
OpenSplice DDS

Data Window Real-time Analytics
Big Data In-Memory
Messaging / In-Memory Store Analitics Messaging /
Web Svcs Full Data Set Mid/Long Term Predictions Web Svcs
Big Data Store Map Reduce
(e.g. HBase) (e.g. Hadoop)

Data Window Real-time Analytics

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Big Data In-Memory Messaging /
In-Memory Store Analitics Web Svcs
OpenSplice DDS

Messaging /
Messaging / Web Svcs
Web Svcs Model Parameters
Full Data Set Mid/Long Term Predictions
etc.

How about the Cube?
(pull, k/v, small) (push, k/v, small)

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
(pull, k/v, big) (push, k/v, big)

variety
OpenSplice DDS

velocity
e
ol um (pull, pk/fk, small) (push, pk/fk, small)
v

(pull, pk/fk, big) (push, pk/fk, big)

Data Window (_, _ , big) Real-time Analytics (push, _ , big)

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Big Data In-Memory Messaging /
(push, _ , big) In-Memory Store Analitics Web Svcs
(push, _ , big)
OpenSplice DDS

Messaging /
Messaging / Web Svcs
Web Svcs Model Parameters
Full Data Set Mid/Long Term Predictions
(push, _ , big) etc.
(pull, k/v, big)

with OpenSplice DDS
OpenSplice DDS

OpenSplice & Big Data Analytics

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ OpenSplice DDS can be used to address several Big Data --
specifically In-Memory Big Data -- Analytics applications

☐ OpenSplice’s key features of interest are:
OpenSplice DDS

☐ Ultra-High Performance In-Memory Store
☐ Ultra- High Performance Data Distribution
☐ Ultra- High Performance Persistent Store
☐ High Availability
☐ Connectivity (e.g. HBase, CouchDB, REST, WebSockets, etc.)

OpenSplice & Big Data Analytics

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ OpenSplice DDS can be used to address several Big Data, and
more specifically In-Memory Big Data, Analytics applications

☐ The key features OpenSplice can provide are:
OpenSplice DDS

☐ Ultra-High Performance In-Memory Store
☐ Ultra-High Performance Data Distribution
☐ Ultra- High Performance Persistent Store
☐ High Availability
☐ Connectivity (e.g. HBase, CouchDB, REST, WebSockets, etc.)

OpenSplice DDS

D
D
BaS ics

Data Distribution Service
For Real-Time Systems

DDS provides a Topic-Based Publish/

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Subscribe abstraction based on: Data
Reader
Data
Writer
☐ Topics: data distribution subject’s
OpenSplice DDS

Data
Reader
Data TopicD
Writer
DataWriters: data producers
TopicA
☐
Data
TopicB Reader
Data
Writer
☐ DataReaders: data consumers TopicC
...

Data Data
Writer Reader

DDS Global Data Space

Data Distribution Service
For Real-Time Systems

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ DataWriters and DataReaders
are automatically and Data
Reader
dynamically matched by the Data
Writer

DDS Dynamic Discovery
OpenSplice DDS

Data
Reader
Data TopicD
Writer
TopicA
☐ A rich set of QoS allows to Data
Reader
TopicB
control existential, temporal,
Data
Writer
TopicC
...
and spatial properties of data Data Data
Writer Reader

DDS Global Data Space

OpenSplice DDS

DDS Entities

Domain Participant
Domain (e.g. Domain 123)

Domain
Participant

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Topic
Partition (e.g. “Telemetry”, “Shapes”, )

T1 Publisher
T1 T3 Subscriber
Topic Instances/Samples Tx
OpenSplice DDS

Ty
Tb
Ta DataWrter DataReader
Tc

☐ A DomainParticipant provides access to a DDS Domain
☐ DDS Domains are identified by natural numbers, such as 0, 1, 2, ...
☐ Each DDS Domain has associated some unique communication and
discovery endpoints, automatically deduced by the Domain ID

OpenSplice DDS

Topics

DDS Topics
“Circle”, “Square”, “Triangle”, ...
☐ A Topic defines a class of streams,

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
also called topic instances Name
☐ A Topic has associated a unique Topic
name, a user defined extensible

Typ

S
OpenSplice DDS

type and a set of QoS policies DURABILITY,

Qo
e
DEADLINE,
ShapeType
☐ QoS Policies capture the Topic PRIORITY,
…
non-functional invariants
struct ShapeType {
☐ Topics can be discovered or @Key
string color;
locally defined long x;
long y;
long shapesize;
};

Topic Instances
☐ Each unique key value

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Instances Instances
identifies a unique stream
of data color =”Green”

DDS not only
struct ShapeType {
☐ color =”red” @Key string color;
Topic
OpenSplice DDS

long x; long y;

demultiplexes “streams” color = “Blue”
long shapesize;};

but provides also lifecycle
information
☐ A DDS DataWriter can
write multiple instances

Anatomy of a DDS Application
OpenSplice DDS

[Scala API]

Domain
Domain

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
val dp = DomainParticipant(domainId)
Participant

Publisher
Topic
Subscriber
OpenSplice DDS

DataWriter
DataReader

[Scala API]

Domain
Domain

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Participant

Session
// Create a Topic
val topic = Topic[ShapeType](dp, “Circle”) Publisher
Topic
Subscriber
OpenSplice DDS

// Create a Publisher / Subscriber
val pub = Publisher(dp)
val sub = Subscriber(dp)

DataWriter
DataReader

[Scala API]

Domain
Domain

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Participant

Session
// Create a Topic
Topic
Subscriber
OpenSplice DDS


Reader/Writers for User Deﬁned for Types DataWriter
DataReader
// Create a DataWriter/DataWriter
val writer = DataWriter[ShapeType](pub, topic) Reader/Writer for
val reader = DataReader[ShapeType](sub, topic) application deﬁned
Topic Types

[Scala API]

Domain
Domain

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Participant

Session
// Create a Topic
Topic
Subscriber
OpenSplice DDS


Reader/Writers for User Deﬁned for Types DataWriter
DataReader
// Write data
val data = new ShapeType(“RED”, 131, 107, 75)
writer write data
Reader/Writer for
// But you can also write like this... application deﬁned
writer ! data Topic Types
// Read new data and print it on the screen
(reader read) foreach (prinln)

Data Reader Caches
Your In-Memory Big Data Store
OpenSplice DDS

Reader/Writer Caches
Application Application Application

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
DataWriter DataReader DataReader

...
OpenSplice DDS

DataWriter Cache DataReader Cache DataReader Cache

Network

write( ) Application Application

Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


Network


Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


Topic Instance
Network


Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


Topic Instance Topic Instance

Network


Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


Writer History

Network


Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


Reader History Reader History

Network


Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


QoS Policies
‣ History
‣ Destination Order Network
‣ Presentation
‣ Partition QoS Policies
‣ Time Based Filter ‣ Reliability ‣ Durability
‣ Ownership ‣ History ‣ Transport Priority
‣ Latency Budget ‣ Time Based Filter

Reader Cache Properties
☐ An entry for each topic instance

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Each cache entry might contain one or
more samples, the number of samples is
controlled by the History QoS Policy DataReader
☐ History.KeepLast(n) keeps the last n samples
OpenSplice DDS

per instance
...
☐ History.KeepAll maintains all the samples,
modulo resource constraints

☐ For a given instance: DataReader Cache
☐ Samples from the same writer are inserted in
write-order
☐ Samples from multiple-writers are ordered using
time-stamps (source or reception depending
on configurable a QoS Policy)

Understanding History
OpenSplice DDS


Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


Network

Assumptions: DataReader configured with history depth of 2 and
DataWriter with default history depth of 1

write( 1 ) Application Application

Copyright
2011,
PrismTech
–

All
Rights
Reserved.

...
OpenSplice DDS


Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

1
...
OpenSplice DDS


Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

1
...
OpenSplice DDS


1 Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

1 1 1
...
OpenSplice DDS


Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

2 1 1
...
OpenSplice DDS


Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

2 1 1
...
OpenSplice DDS


2 Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

2 1 1
2 ... 2
OpenSplice DDS


Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

3 1 1
2 ... 2
OpenSplice DDS


Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

3 1 1
2 ... 2
OpenSplice DDS


3 Network



Copyright
2011,
PrismTech
–

All
Rights
Reserved.

3 2 2
3 ... 3
OpenSplice DDS


Network


Interacting with the
DataReader Cache
OpenSplice DDS

Reading Data Samples
☐ Samples can be read from the Data Reader History Cache

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ The action of reading a sample is non-destructive. Samples are
not removed from the cache
OpenSplice DDS

DataReader DataReader

... read ...

DataReader Cache DataReader Cache

Taking Data Samples
☐ Samples can be taken from the Data Reader History Cache

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ The action of taking a sample is destructive. Samples are
removed from the cache
OpenSplice DDS

DataReader DataReader

... take ...

DataReader Cache DataReader Cache

Read vs. Take

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ The read operation should always be access the latest know value
for topics that represent distributed state
OpenSplice DDS

☐ The take operation should be used to get the last notification from a
topic that represent an event

Cherry Picking in DDS

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ DDS provides some very flexible mechanisms for selecting the data
to be read:
OpenSplice DDS

☐ Data Content
☐ Data Status

☐ These mechanisms are composable

Content-Based Data Selection
OpenSplice DDS

Filters and Queries Application

☐ DDS Filters allow to control what gets
into a DataReader cache

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
Query
☐ DDS Queries allow to control what gets
out of a DataReader cache DataReader

Filters are defined by means of
OpenSplice DDS

☐
...
ContentFilteredTopics

...
...

...
☐ Queries operate in conjunction with DataReader Cache
read operations
Filter
☐ Filters and Queries are expressed as SQL
where clauses

State-Based Selection
OpenSplice DDS

Sample, Instance, and View State
☐ The samples included in the DataReader cache have associated

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
some meta-information which, among other things, describes the
status of the sample and its associated stream/instance
☐ The Sample State (READ, NOT_READ) allows to distinguish between
OpenSplice DDS

new samples and samples that have already been read
☐ The View State (NEW, NOT_NEW) allows to distinguish a new
instance from an existing one
☐ The Intance State (ALIVE, NOT_ALIVE_DISPOSED,
NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of
the instance to which a sample belongs

List Comprehension
OpenSplice DDS

List Comprehension in DDS

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ When using the Scala API, the DDS readers expose data as
LinearSeqOptimized thus making possible list comprehension to
address quite a few task
OpenSplice DDS

☐ Example
☐ dr read foreach (println)
☐ dr read map (transformer)
☐ ...

Examples
Compute the average shape and write it as a Square sample:

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐

val data = dr history
val s0 = new ShapeType(“GRAY”, 0, 0 ,0)

dw ! scaleShape((data : s0) (sumShape), data.size)
OpenSplice DDS

☐ Filter samples:

(dr read) filter (_.x <= 100 && _.y <= 200)

High Performance
Persistent Store
with OpenSplice
OpenSplice DDS

OpenSplice Durability
☐ OpenSplice DDS provides a highly available high-performance

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
persistent store optimized for storing DDS data

☐ This store can be queries to retrieve data using, content as well as
temporal filters
OpenSplice DDS

☐ Ex: val topic = Topic[ShapeType](dp, “Circle”)
val dr = DataReader[ShapeType](topic)
// ...
val query = Query(“x < %0 AND y < %1”, List(“200”, “300”))
val t0 = new Time(...)
val t1 = new Time(...)

// Retrieve all data for the Circle Topic that matches
// the query and has timestamp within [t0, t1]
dr waitForHistoricalData(query, t0, t1)

OpenSplice DDS

Connectivity

OpenSplice Gateway

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ The OpenSplice Gateway supports Apache Camel Connectors for
Big Data Stores thus making it very simple to stream data to/from
HBase, CouchDB, etc.
OpenSplice DDS

☐ Example:
val circleURI = "ddsi:Circle:0/org.opensplice.demo.ShapeType”
val hbaseURI = "hbase:ishapes?mappingStrategyName=body&operation=CamelHBasePut"

// ...

// Put incoming circles into an HBase Table
circleURI unmarshal(cdrData) process { e2d (_, "circle") } to(hbaseURI)

Putting it All Together
OpenSplice DDS

iShapes
In-Memory Analytics Real-time Analytics Display

Data Window (_, _ , big) Real-time Analytics

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
OpenSplice In-Memory
Reader Cache Analitics

(push, _ , big)
OpenSplice DDS

OpenSplice
JMS

Full Data Set
XM
ST

PP
RE

OpenSplice

Gateway
P
Cu

AMQ
sto
m

JMS
DDSI-RTPS

XM
ST

PP
RE

OpenSplice

Gateway
P
Cu

AMQ
sto
m

DDSI-RTPS

Shapes to HBase Tables
struct ShapeType {

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
@Key
string color;
long x;
create 'ishapes', 'color'
long y;
long shapesize;
};
OpenSplice DDS

☐ Shapes are mapped into the HBase ‘ishapes’
☐ The row key is the topic name, e.g ‘circle’, ‘square’, etc.
☐ The ‘color” column family is used to distinguish about different instances of a given shape
☐ Different samples for a given instance are distinguished using HBase versions

Scale Out Techniques

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Scale out in OpenSplice DDS is achieved through data partitioning

☐ OpenSplice DDS provides two orthogonal mechanisms for data
OpenSplice DDS

partitioning
☐ Data Partitions
☐ Content Filters

OpenSplice DDS

Live Demo

Concluding Remarks
☐ OpenSplice DDS provide Big Data Applications with
☐ Ultra-High Performance In-Memory Big Data Store

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
☐ Ultra-High Performance Data Distribution
☐ High Performance Persistent Store
☐ Connectivity to Big Data Stores
OpenSplice DDS

☐ OpenSplice DDS is an ideal fit for In-Memory Big Data Analytics
Applications

OpenSplice DDS
Collect Store | Organize | Analyze Share
OpenSplice DDS

References

Copyright
2011,
PrismTech
–

All
Rights
Reserved.
OpenSplice | DDS Escalier
¥ #1 OMG DDS Implementation ¥ Fastest growing JVM Language ¥ Scala API for OpenSplice DDS
¥ Open Source ¥ Open Source ¥ Open Source
¥ www.opensplice.org ¥ www.scala-lang.org ¥ github.com/kydos/escalier
OpenSplice DDS

¥ Simple C++ API for DDS ¥ DDS-PSM-Java for OpenSplice DDS ¥ DDS-based Advanced Distributed
¥ Open Source ¥ Open Source Algorithms Toolkit
¥ github.com/kydos/simd-cxx ¥ github.com/kydos/simd-java ¥ Open Source
¥ github.com/kydos/dada

:: Connect with Us ::

¥opensplice.com ¥forums.opensplice.org
¥@acorsaro
¥opensplice.org ¥opensplicedds@prismtech.com ¥@prismtech
OpenSplice DDS

¥ crc@prismtech.com
¥sales@prismtech.com
¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro

Got Big Data? Get OpenSplice!

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Viewers also liked

Viewers also liked (20)

Similar to Got Big Data? Get OpenSplice!

Similar to Got Big Data? Get OpenSplice! (20)

More from Angelo Corsaro

More from Angelo Corsaro (20)

Got Big Data? Get OpenSplice!