1. Message & Stream
Oriented Communication
CS4262 Distributed Systems
Dilum Bandara
Dilum.Bandara@uom.lk
Some slides extracted from Dr. Srinath Perera & Dr. Rajkumar Buyya’s
Presentation Deck

2. Outline
 Message oriented communication
 Event queues
 Pub/sub networks
 MPI
 Stream-based communication
 Multicast communication
2

3. Definitions – Persistent
vs. Transient Communication
 Persistent communication
 Message submitted for transmission is stored by communication
system for as long as it takes to deliver it to receiver
 e.g., e-mail, SMS
 Not necessary for sender to continue execution after submitting
a message
 Not necessary for receiver to be executing at the time message
submission
 Transient communication
 Message is stored by communication system only as long as
sending & receiving applications are executing
 e.g., transport-level communication services (store-and-forward
router)
 Receiver needs to be there when a message is received
3

4. Definitions – Asynchronous vs.
Synchronous Communication
 Asynchronous communication
 Sender continues immediately after it has submitted
its message for transmission
 Message may be stored, in a local buffer at sending
host or at an intermediate communication server
 Synchronous communication
 Sender is blocked until its message is stored in a
local buffer at receiving host or actually delivered to
receiver
 Strongest form – Sender blocked until receiver has
processed message
4

5. Types of Communication
5
a) Persistent asynchronous communication (e.g., e-mail)
b) Persistent synchronous communication
Source: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms

6. Types of Communication (Cont.)
6
c) Transient asynchronous communication (e.g., UDP)
d) Receipt-based transient synchronous communication
Source: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms

7. Types of Communication (Cont.)
7
e) Delivery-based transient synchronous communication
f) Response-based transient synchronous communication
Source: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms

8. Types of Communication (Cont.)
1. Persistent, asynchronous communication
 Messages are persistently stored in local host buffer,
or at an intermediate communication server
 e.g., e-mail
2. Persistent, synchronous communication
 Messages can be persistently stored only at receiving
host
 Weaker form of synchronous communication
 It isn’t necessary for receiving application to be executing
8

9. Types of Communication (Cont.)
3. Transient, asynchronous communication
 Messages is temporarily stored in a local buffer &
sender immediately continues
 e.g., UDP, RPC fire & forget
4. Transient, synchronous communication
I. Weakest form
 Receipt-based, transient, synchronous communication
 Sender is blocked until message is stored in a local buffer of
receiving host
 e.g., Asynchronous RPC delivery part (send with Ack)
9

10. Types of Communication (Cont.)
II. Weaker form
 Delivery-based, transient, synchronous communication
 e.g., Asynchronous RPC delivery part (Send with Ack)
III. Strongest form
 Response-based, transient, synchronous communication
 e.g., RPC & RMI
10

11. Message-Oriented Communication
 Message-oriented transient communication
 Transport-level sockets
 Message-Passing Interface (MPI)
 Message transfer latency  milliseconds to seconds
 Message-oriented persistent communication
 Message-queuing systems or Message-Oriented
Middleware (MOM)
 Provide intermediate-term storage capacity for
messages
 Doesn’t requiring either sender or receiver to be active during
message transmission
 Message transfer latency  seconds to minutes 11

12. Message-Queuing Model
 Applications communicate by inserting messages into a
series of queues
 Loosely-coupled communication
 Sender is given guarantee that its message will eventually be
inserted in recipient’s queue
 No guarantee on timing, or message will actually be read
12
Source: http://msdn.microsoft.com/en-
us/library/windows/desktop/ms699870%
28v=vs.85%29.aspx

13. Message-Queuing Model
Combinations
13
Loosely-coupled communications using Queues.
Sender & receiver can execute completely independent of each other.
Source: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms

14. Message-Queuing Interface
 Basic interface to a queue in a message-
queuing system
14
Primitive Meaning
Put Append a message to a specified queue
Get Block until the specified queue is nonempty, & remove first
message
Poll Check a specified queue for messages, & remove first
message. Never block
Notify Install a handler to be called when a message is put into
specified queue

15. Architecture of a Typical Message-
Queuing System With Routers
15
Source: http://csis.pace.edu/~marchese/SE765/L7/L7.htm

16. Message Queue Applications
 Amazon Simple Queue Service (Amazon SQS)
 Decouple components of a cloud application
 Can transmit any volume of data, at any level of
throughput, without losing messages or requiring
other services to be always available
16
Source: http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-using-sqs-queue.html

17. Message Queue Applications (Cont.)
 Java Message Service (JMS) queues
 Based on Java Enterprise Edition (JEE)
 Loosely coupled, reliable, & asynchronous
 Other applications
 E-mail
 Workflow & Groupware
 Batch processing
 Job queues
 Stream/complex-event processing
17

18. Pub/Sub Networks
 Publishers publish messages
 Usually to a topic
 Subscribers may express
interest for a subset of
messages
 Pub/Sub system makes sure
interested parties get
corresponding messages
 80-90% implementations are
topic based
 Content based is hard
18
Source: http://msdn.microsoft.com/en-
us/library/ff649664.aspx

19. 19

20. Eventing in Pub/Sub
 Decouple
 Time – both parties need not be online same time
 Space – don’t know each other’s addresses
 Synchronization – don’t have to wait for each other
 Models
 Event producer  Event consumer
 Event producer  Broker  Event consumer
 Actually it’s Event producer  Event Bus  Notifier
 Event bus can be 1+ nodes
 Decoupling makes it is hard to debug
20

21. Pub/Sub Brokers
 RSS feeds
 Apache ActiveMQ, Qpid
 OGCE WS-Messenger
 For web services
 WSO2 Event Server
 Microsoft BizTalk Server
 Distributed brokers
 Narada Broker (www.naradabrokering.org)
 Whihdum (http://code.google.com/p/wihidum/)
21

22. Message Brokers
 Applications need to understand messages they
receive
 Options
 Standard message formats
 Not suitable as message-queuing applications typically
operate at a higher level of abstraction
 Convert messages using a Message Broker
 Convert incoming messages to a format that can be
understood by destination application
22

23. Message Brokers
23
Source: http://www.fi.muni.cz/~xkolar2/dp/html/index.html

24. Message Broker Architecture
24
Source: A.S. Tanenbaum & M.V. Steen, Distributed Systems: Principles and Paradigms

25. Message Bus
 A.k.a. Enterprise Service Bus (ESB)
 Tasks
 Monitor & control routing of message exchange between services
 Resolve contention between communicating service components
 Control deployment & versioning of services
 Marshal use of redundant services
25
Source: http://msdn.microsoft.com/en-us/library/ff647328.aspx

26. Complex Event Processing System
26

27. WSO2 Siddhi CEP Architecture
27
Source: S. Suhothayan et al., “Siddhi: A Second Look at Complex Event Processing Architectures”, Nov. 2011

28. Siddhi Pipeline Architecture
28
Source: S. Suhothayan et al., “Siddhi: A Second Look at Complex Event Processing Architectures”, Nov. 2011

29. Message-Passing Interface (MPI)
 Designed for communication among parallel
applications
 Primarily used in HPC systems with high-speed
interconnection networks
 Provides an interface with advanced features
such as different forms of buffering &
synchronization
 Provides hardware independence
 Supports many types/forms of communication
 Algorithm/application specific performance
optimization 29

30. MPI Operations
30
Source: http://www.broadinstitute.org/gatk/about/#high-performance
Source: http://mpitutorial.com/mpi-reduce-and-allreduce/

31. MPI_Allreduce
31
Global sum followed
by distribution of result
Source: Peter Pacheco, "An Introduction
to Parallel Programming"

32. Butterfly-Structured Global Sum
32
Source: Peter Pacheco, "An Introduction to Parallel Programming"

33. MPI Primitives
33
Primitive Meaning
MPI_bsend Append outgoing message to local send buffer.
MPI_send Send a message & wait until copied to local or remote
buffer.
MPI_ssend Send a message & wait until receipt starts.
MPI_sendrecv Send a message & wait for reply.
MPI_isend Pass reference to outgoing message, and continue.
MPI_issend Pass reference to outgoing message, & wait until receipt
starts.
MPI_recv Receive a message; block if there is none.
MPI_irecv Check if there is an incoming message, but do not block.

34. Stream Oriented Communication
 Continuous streams of data
 e.g., real media stream
 Modes
 Asynchronous – no time limit
 Synchronous – max time limit
 Isochronous – both max & lower limit
 Simple stream – One type of streams
 Complex stream – Many streams
 e.g., movie with video, 2 audio, & subtitles
 QoS – bit rate, delay, jitter, etc.
 Enforcing QoS is a main challenge 34

35. Streams (Cont.)
 Enforcing QOS
 Mark packets as high priority
 Use buffers to reduce jitter (play from buffer)
35Source: T. Banka et al., “An architecture and a programming interface for application-aware data
dissemination using overlay networks,” COMSWARE 2007

36. Streams (Cont.)
 Stream synchronization
 Read alternatively
 Control interface to control rates
 Distribution – merge at sender
36

37. Multicast Communication
 Network level – IP multicast
 Very efficient within LAN
 No global routing support
 Application level
 Main challenge is to setup a path
 Options
 Tree based
 Mesh based
 Can recover from failures
 Often used in parallel computing clusters
 Group communication
 Ordered reliable multicast 37

38. Tree-Push & Mesh-Pull
38
Source:
J. Liu et al., "Opportunities and challenges of peer-to-peer
internet video broadcast,” 2008.
X. Zhang et al., "CoolStreaming/DONet: a data-driven overlay
network for efficient live media streaming," INFOCOM 2005.