2. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
What I’ll Cover
☐ Understand DDS and its Data Sharing abstraction
☐ Learn about the benefits that Data Sharing provide over
plain Messaging
☐ Learn how much simpler is to build distributed systems
with DDS than with messaging technologies
3. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Distributed System Definition
A Distributed System is a collection autonomous software
components distributed across a network that coordinate their
activities and share system resources to run as a “single system”
N.B. It is useful to remark how this definition does not mention message passing or
shared memory, those are implementation details
4. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Distributed Systems Challenges
A number of challenges are inherent to the nature of distributed
systems:
☐ Transparency
☐ Scalability
☐ Dependability
☐ Performance
☐ Flexibility
5. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Building Distributed Systems
Several abstractions have been proposed to build Distributed
Systems:
☐ Distributed Shared Memory
☐ Message Passing/Queueing
☐ Distributed Objects
☐ Tuple Spaces
6. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Building Distributed Systems
Several abstractions have been proposed to build Distributed
Systems:
☐ Distributed Shared Memory
☐ Message Passing/Queueing
☐ Distributed Objects
☐ Tuple Spaces
7. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Message Queueing
☐ Message Queueing, as
suggested by its name, provide a
coordination abstraction based
on the exchange of messages
between distributed processes
☐ Message delivery is often
mediated by brokers
☐ A Message is composed by a
header and a body, where the
body is opaque (often binary)
data
Broker
BodyHeader
Message
8. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Tuple Space
☐ Tuple Space provide the
abstraction of a Global Tuple
Space that can be used by
autonomous applications to
coordinate by inserting (out),
reading (rd) and removing
(in) tuples
☐ The Tuple Space model was
initially introduced by
Gelernter and Carrero’s Linda
and extended over the years
<a,b,c>
<x,y, <j, k>>
<x,y, <j, k>>
out(<a,b,c>) rd(<a,_,_>)
in(<15,_>)eval(...)
9. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Tuple Spaces vs Messaging
☐ Tuple Spaces provides spatial
and temporal decoupling along
with content awareness
☐ Tuples remain in the tuple space
up to when they are not
removed
☐ Tuple Spaces allow application
to share information
☐ Overall Tuple Spaces provide a
high-level abstraction for building
distributed systems
☐ Messaging doesn’t provide
content awareness and usually
provides limited temporal
decoupling
☐ Messages usually disappear once
they are delivered to consumers
☐ Messaging allows application to
exchange messages. Information
has to be reconstructed from
messages
☐ Messaging provides a relatively
low-level abstraction for building
distributed systems
10. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Why is Messaging so Popular?
At this point you may ask yourself:
If Tuple Spaces are so cool, why are messaging technologies so
popular?
The answer is actually pretty simple:
Tuple Spaces where originally quite inefficient and this lead people to
adopt different approaches, such as messaging and distributed objects.
Nowadays, however, there are plenty of technologies deriving from the
tuple space concept that have very high performance (often superior
than messaging).
11. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Why is Messaging so Popular?
At this point you may ask:
OK, that makes sense. Yet there are still plenty of people using
messaging technologies. Why?
The answer is again pretty simple:
The adoption of Tuple Space inspired technologies is growing steadily,
but many people are still using messaging since this is what they are
familiar with. But enlightenment will eventually hit them ;-)
14. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Data Distribution Service
☐ Topics: Unit of data sharing
☐ DataWriters: data producers
☐ DataReaders: data consumers
DDS provides a relaxed Tuple Space
Abstraction abstraction based on:
DDS Global Data Space
...
TopicA
TopicB
TopicC
TopicD
Data
Writer
Data
Writer
Data
Writer
Data
Writer
Data
Reader
Data
Reader
Data
Reader
Data
Reader
For Real-Time Systems
15. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Data Distribution Service
☐ DataWriters and DataReaders
are automatically and
dynamically matched by the
DDS Dynamic Discovery
☐ A rich set of QoS allows to
control existential, temporal,
and spatial properties of data
DDS Global Data Space
...
TopicA
TopicB
TopicC
TopicD
Data
Writer
Data
Writer
Data
Writer
Data
Writer
Data
Reader
Data
Reader
Data
Reader
Data
Reader
For Real-Time Systems
16. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Fully Distributed Tuple Space
☐ DDS Implements a fully
distributed Tuple Space
with relaxed consistency.
☐ DDS provides eventual
consistency as opposed
to strong consistency
☐ In addition, DDS provides
a local “take” and a
global “dispose” in place
of the “in” operation
<a,b,c>
<x,y, <j, k>>
<x,y, <j, k>>
out(<a,b,c>) rd(<a,_,_>)
in(<15,_>)eval(...)
Traditional Tuple Space
DDS Data Space
Logical Global Data Space
Physical
Local Data Space
The
local
Data
Space
is
a
projection
of
the
Logical
Global
Data
Space.
This
projection
reflects
the
interest
of
the
given
application.
The
Global
Data
Space
is
an
abstraction
in
DDS
built
by
the
composition
of
Local
Data
Spaces.
17. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
What DDS Provides
☐ Location Transparency => Dynamic Discovery
☐ Anonymity and Temporal Decoupling => Tuple Space
☐ Data Centricity => Tuple Space
☐ Dependability => Fault-Tolerant and Secure
☐ Scalability => Fully distributed architecture easy to scale-out
☐ Portability => OS and Programming Language Independence
☐ Interoperability => Standardized Wire Protocol
18. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
IDL
DDS Topics
☐ A Topic defines a class of
streams
☐ A Topic has associated a user
defined extensible type and a
set of QoS policies
☐ The Topic name, type and QoS
defines the key functional and
non-functional invariants
☐ Topics can be discovered or
locally defined
DURABILITY,
DEADLINE,
PRIORITY,
…
“TVehicleDynamics”
Topic
Type
Name
QoS
struct VehicleDynamics {
long vid;
long x; long y;
long dx; long dy;};
#pragma keylist VehicleDynamics vid
VehicleDynamics
19. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Instances
☐ Each unique key value identifies a
unique topic instance
☐ DDS not only demultiplexes
“streams” but provides also lifecycle
information
☐ A DDS DataWriter can write multiple
instances struct Person {
long ssn;
String name; String surname;
};
#pragma keylist Person ssn
<101,Leslie, Lamport >
<010,Edsger, Dijkstra>
<110,Haskell, Curry>
21. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Running App
☐ During this webcast we’ll be looking at a few DDS aspects through
the iShapes applications
☐ The advantage of targeting the iShapes application is that we can
actually “see” what our application will do
24. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
iShapes Information Model
“Circle”
Topic
Type
Name
QoS
ShapeType
IDL
struct ShapeType {
string color;
long x; long y;
long shapesize;};
#pragma keylist ShapeType color
“Square”
Topic
Type
Name
QoS
ShapeType
“Triangle”
Topic
Type
Name
QoS
ShapeType
25. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Anatomy of a DDS Application
Domain Participant
[ISO C++ API]
auto dp = DomainParticipant(domainId);
26. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Anatomy of a DDS Application
Domain Participant
Topics
auto dp = DomainParticipant(domainId);
// Create a Topic
auto topic = Topic<ShapeType>(dp, “Circle”);
[ISO C++ API]
27. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Anatomy of a DDS Application
Domain Participant
Topics
auto dp = DomainParticipant(domainId);
// Create a Publisher
auto pub = Publisher(dp);
// Create a Topic
auto topic = Topic<ShapeType>(dp, “Circle”);
Publisher
// Create a Subscriber
auto sub = Subscriber(dp);
Subscriber
[ISO C++ API]
28. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Anatomy of a DDS Application
Domain Participant
Data Writer
Topics
// Create a DataWriter
auto writer = DataWriter<ShapeType>(pub, topic);
auto dp = DomainParticipant(domainId);
// Create a Publisher
auto pub = Publisher(dp);
// Create a Topic
auto topic = Topic<ShapeType>(dp, “Circle”);
Publisher
// Create a Subscriber
auto sub = Subscriber(dp);
Subscriber
Data Reader
// Create a DataWriter
auto reader = DataReader<ShapeType>(sub, topic);
[ISO C++ API]
29. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Putting it all together
1 int main(int argc, char* argv[]) {
2 try {
3 DomainParticipant dp(0);
4 Topic<ShapeType> topic(dp, "Circle");
5 Publisher pub(dp);
6 DataWriter<ShapeType> dw(pub, topic);
7
10 for (int i = 0; i < N; ++i) {
11 ShapeType bc = {"RED", i, i, 60};
12 ShapeType rc = {"BLUE", N-i, N-i, 60};
13 dw.write(bc);
14 // You can also write with streaming operators!
15 dw << rc;
16 std::cout << "." << std::flush;
17 usleep(sleep_time);
18 }
19
20 } catch (const dds::core::Exception& e) {
21 std::cout << e.what() << std::endl;
22 }
23 return 0;
24 }
30. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Anatomy of a DDS Application
Domain Participant
Data Writer
Topics
// Create a DataWriter
DataWriter<ShapeType> writer =
pub.createDataWriter<ShapeType>(topic);
DomainParticipant dp = dpf.createParticipant(domainId);
// Create a Publisher
Publisher pub = dp.createPublisher();
// Create a Topic
Topic<ShapeType> topic = dp.createTopic(“Circle”,ShapeType.class);
Publisher
// Create a Subscriber
Subscriber sub = dp.createSubscriber();
Subscriber
Data Reader
[Java 5 API]
Bootstrap
ServiceEnvironment env = ServiceEnvironment.createInstance(this.getClass.getClassLoader);
DomainParticipantFactory dpf = DomainParticipantFactory.getInstance(env)
// Create a DataWriter
DataReader<ShapeType> reader =
sub.createDataReader<ShapeType>(topic);
31. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Anatomy of a DDS Application
Dependency Injection through Implicits
Data Writer
Topics
// Create a DataWriter
val writer = DataWriter[ShapeType](topic)
[Scala API]
import dds.config.DefaultEntities._
// Create a Topic
val topic = Topic[ShapeType](“Circle”)
Data Reader
// Create a DataWriter
val reader = DataReader[ShapeType](topic)
33. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Writing Data in DDS
writer.write(SpapeType(“RED”, 10, 20, 30));
// -- or --
writer << SpapeType(“RED”, 10, 20, 30);
[ISO C++ API]
34. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Writing Data in DDS
[Java 5 API]
writer.write(new SpapeType(“RED”, 10, 20, 30));
35. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Writing Data in DDS
[Scala API]
writer.write(new ShapeType(“RED”, 10, 20, 30))
// -- Or equivalently:
writer write (new ShapeType(“RED”, 10, 20, 30))
37. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Reading Data
// Read Data
auto data = reader.read();
// -- you can also read with streaming operators
auto data = reader << read();
// Print the received data on the console
std::for_each(samples.data().begin(), samples.data.end(), demo::printShape)
[ISO C++ API]
38. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Reading Data
// You can read data on your favorite container using iterators
auto samples = std::vector<Samples<ShapeType>>(MAX_SHAPES);
// Read data into the provided container
reader.read(samples.begin(), MAX_SHAPES);
// Print the received data on the console
std::for_each(samples.data().begin(), samples.data.end(), demo::printShape)
[ISO C++ API]
39. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Reading Data
//
Read
Data
Sample.Iterator<ShapeType>
iterator
=
reader.read();
while
(iterator.hasNext())
{
//
Assuming
ShapeType
ovrrides
toString
out.println(iterator.next().getData());
}
[Java API]
40. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Reading Data
//
Read
Data
reader.read
foreach
{
s
=>
println(s.data())
}
[Scala API]
43. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Cherry Picking in DDS
☐ DDS provides some very flexible mechanisms for selecting the data
to be read:
☐ Data Content
☐ Data Status
☐ These mechanisms are composable
45. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Filters and Queries
☐ DDS Filters allow to control what gets
into a DataReader cache
☐ DDS Queries allow to control what gets
out of a DataReader cache
☐ Filters are defined by means of
ContentFilteredTopics
☐ Queries operate in conjunction with
read operations
☐ Filters and Queries are expressed as SQL
where clauses
DataReader Cache
DataReader
...
...
...
...
Filter
Query
Application
46. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Query struct VehicleDynamics {
long vid;
long x; long y;
long dx; long dy;};
#pragma keylist VehicleStatus vid
// Define the query and the parameters
std::vector<std::string> p;
p.push_back("100");
p.push_back("100");
dds::core::Query q("x < %0 AND y < %1", p.begin(), p.end());
auto data = reader
.select()
.content(q)
.read();
[DDS C++ API 2010]
47. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Instances
☐ DDS provides a very efficient way of reading data belonging to a
specific Topic Instance
☐ Obviously, one could use queries to match the key’s value, but this
is not as efficient as the special purpose instance selector
// C++
auto data = reader
.select()
.instance(handle)
.read();
// Scala
val data = reader read(handle)
49. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Sample, Instance, and View State
☐ The samples included in the DataReader cache have associated
some meta-information which, among other things, describes the
status of the sample and its associated stream/instance
☐ The Sample State (READ, NOT_READ) allows to distinguish between
new samples and samples that have already been read
☐ The View State (NEW, NOT_NEW) allows to distinguish a new
instance from an existing one
☐ The Intance State (ALIVE, NOT_ALIVE_DISPOSED,
NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of
the instance to which a sample belongs
50. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
State Selector in Action
// Read only new samples
auto data = reader
.select()
.state(status::DataState::new_data())
.read()
// Read any samples from live instances
auto data = reader
.select()
.state(status::DataState::any_data())
.read()
C++
// Read only new samples
val data = reader read
// Read any samples from live instances
val data = reader read(SampleSelector.AnyData)
Scala
51. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Putting all Together
☐ Selectors can be composed in a flexible and expressive manner
C++
auto data = reader
! .select()
.instance(handle)
! ! .state(status::DataState::new_data())
! ! .content(q)
.read();
53. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
QoS Model
☐ QoS-Policies control local and
end-to-end properties of DDS
entities
☐ Local properties controlled by
QoS are related resource usage
☐ End-to-end properties
controlled by QoS are related
to temporal and spatial aspects
of data distribution
☐ Some QoS-Policies are matched
based on a Request vs. Offered
Model thus QoS-enforcement
Publisher
DataWriter
Topic
Type
QoS
Name
writes
QoS
DataWriter
Topic
Typewrites
Subscriber
DataReaderreads
DataReaderreads
...
QoS
Name
QoS
QoS QoS
QoS matching
......
QoS QoS
Type Matching
DomainParticipant DomainParticipant
QoS QoS
54. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
QoS Policies
QoS Policy Applicability RxO Modifiable
USER_DATA
TOPIC_DATA
GROUP_DATA
DURABILITY
DURABILITY
SERVICE
HISTORY
PRESENTATION
RELIABILITY
PARTITION
DESTINATION
ORDER
LIFESPAN
DP, DR, DW N Y
ConfigurationT N Y Configuration
P, S N Y
Configuration
T, DR, DW Y N
Data Availability
T, DW N N
Data Availability
T, DR, DW N N
Data Availability
P, S Y N
Data Delivery
T, DR, DW Y N
Data Delivery
P, S N Y
Data Delivery
T, DR, DW Y N
Data Delivery
T, DW N Y
Data Delivery
[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]
55. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
QoS Policies
QoS Policy Applicability RxO Modifiable
DEADLINE
LATENCY
BUDGET
TRANSPORT
PRIORITY
TIME BASED
FILTER
OWNERSHIP
OWNERSHIP
STRENGTH
LIVELINESS
T, DR, DW Y Y
Temporal/
Importance
Characteristics
T, DR, DW Y Y
Temporal/
Importance
Characteristics
T, DW N Y
Temporal/
Importance
Characteristics
DR N Y
Temporal/
Importance
Characteristics
T, DR, DW Y N
ReplicationDW N Y Replication
T, DR, DW Y N Fault-Detection
[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]
56. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Setting QoS Policies
C++
// Setting Partition QoS-Policy on Publisher
qos::PublisherQos pubQos;
pubQos << policy::Partition("Partition");
Publisher pub(dp, pubQoS);
// Setting various QoS-Policy on a Topic
qos::TopicQos tqos;
tqos << policy::Reliability::Reliable()
<< policy::Durability::Transient()
<< policy::History::KeepLast(5);
Topic<VehicleDynamics> topic(dp,"Partition", tqos);
58. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
The “Ipse Dixit” Dejavu
☐ As an historical reference“Ipse
Dixit” was used by the Holy
Inquisition to push back any
argument that would contrast
with established knowledge
☐ A famous example was the
argument around the Geo-
Centric model of the Solar System
☐ What has this to do with
Middleware... Let me show you!
60. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
What’s Wrong with this Paper?
☐ When looking at DDS the
Authors did not refer to the
state of the art
☐ DDS is discarded as being too
complicated (really?!?!) -- they
did not check out the new
APIs available since 2010
☐ Beyond that, I think the authors
where comparing a bit apples
and oranges since DDS
provides far more than
ZeroMQ!
61. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
DDS vs. 0MQ -- What’s Simpler?
import
dds.config.DefaultEntities._
//
Create
a
Topic
val
topic
=
Topic[ShapeType](“Circle”)
//
Create
a
DataWriter
val
writer
=
DataWriter[ShapeType](topic)
//
Create
a
DataWriter
writer
write
(new
ShapeType(“RED”,
1,2,3))
import zmq
import time
//
Create
a
Context
context = zmq.Context()
//
Create
a
Socket
publisher = context.socket (zmq.PUB)
//
Bind
the
socket
to
a
name
publisher.bind ("ipc://nasdaq-feed")
//
Send
Data
publisher.send (message)
Sending Data
62. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
DDS vs. 0MQ -- What’s Simpler?
import
zmq
import
time
//
Create
a
Context
context
=
zmq.Context()
//
Create
a
Socket
subscriber
=
context.socket
(zmq.SUB)
//
!!!
MANUALLY
CONNECT
IT
!!!
What
if
your
//
topology
changes?!?
subscriber.connect
("tcp://192.168.55.112:5556")
subscriber.connect
("tcp://192.168.55.201:7721")
//
Define
subscrption
subscriber.setsockopt
(zmq.SUBSCRIBE,
"NASDAQ")
//
Receive
message
message
=
subscriber.recv()
//
...
Now
decode
the
msg
and
do
something
//
with
it
Receiving Data
import
dds.config.DefaultEntities._
//
Create
a
Topic
val
topic
=
Topic[ShapeType](“Circle”)
//
Create
a
DataReader
val
reader
=
DataReader[ShapeType](topic)
//
Create
a
DataWriter
reader.read()
foreach(s
=>
println(s.data))
63. Copyright
2013,
PrismTech
–
All
Rights
Reserved.
OpenSpliceDDS
Concluding Remarks
☐ DDS provides a powerful and feature-rich abstraction for building
distributed systems
☐ This technology is widely used in mission and business critical
systems and it being swiftly adopted in data-centric/big-data
systems