Communication in
Distributed Systems
CS4262 Distributed Systems
Dilum Bandara
Dilum.Bandara@uom.lk
Some slides extracted from Dr. Srinath Perera & Dr. Rajkumar Buyya’s
Presentation Deck

Outline
 Network Programming
 Remote Procedure Calls
 Remote Method Invocation
 Message Oriented Communication
2

Network Programming
 Communication is typically over IP
 HPC Clusters may use Infiniband or Myrinet
 Unicast, multicast, anycast, broadcast
 Only unicast is globally routable
 2 main approaches
 TCP – connection oriented
 UDP – connection less
 Lot more popular than you think!
3

End-to-End Communication
4Source: Communication Networks by
Alberto Leon-Garcia (Author), Indra Widjaja, 2000

TCP Socket Calls
5
Source: Communication Networks by Alberto Leon-
Garcia (Author), Indra Widjaja, 2000

UDP Socket Calls
6
• No “handshake”
• No simultaneous close
• No fork() for concurrent servers
Source: Communication
Networks by Alberto Leon-
Garcia (Author), Indra Widjaja,
2000

Messages
 Used to convey data among nodes
 Has a limited size
 Typically has a header
 Multiple messages may be send through same
connection
 Identify message boundaries based on size, unique
symbol patterns, etc.
 Messages can be clear text, binary, XML, JSON,
etc.
7

Communication in Distributed Systems
– History
 Started with RPC
 Went to distributed objects
 Come back to RPC + Message Oriented
Communication
8

Remote Procedure Calls (RPCs)
 Extend local procedure calls to work across computers
 Call a procedure on a remote machine “just” as you would on
local machine
 Send the call as a message & get response/fault as a
message
9
Source: http://pubs.opengroup.org/onlinepubs/9629399/chap6.htm

RPC in Action…
10Source: Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007

RPC – Steps
1. Client procedure calls client stub in normal way
2. Client stub builds message, calls Middleware
3. Middleware sends message to remote Middleware
4. Remote Middleware gives message to server Skeleton
5. Server stub unpacks parameters, calls local
implementation
6. Local implementation does work, returns result to
Skeleton
7. Skeleton packs it in message, calls local Middleware
8. Server's Middleware sends message to client's
Middleware
9. Client's Middleware gives message to client stub
10. Stub unpacks result, returns to client 11

RPCs
 Introduced by Birrell & Nelson in 1984
 Pre-RPC – Most applications were built directly over Internet
primitives
 Initial idea – Mask distributed computing system using a
“transparent” abstraction
 Looks like normal procedure call
 Hides all aspects of distributed interaction
 Supports an easy programming model
 Today, RPC is the core of many distributed systems,
e.g., Web Services
 But now acknowledge other aspects like failures &
asynchronous nature, communication latency more
12

RPCs (Cont.)
13
Source - wikipedia.org

Message Exchange Patterns
 With local calls, you call, & wait for response
 But with RPC, there are other possibilities
 Send/Receive
 Send Only (Fire & Forget)
 Send with Ack (Send Robust)
 There are 2 models for client API
 Polling
 Callback
14

RPC Concerns
 How to make RPC similar to a local procedure
call?
 Communications
 Location transparency
 Parameter passing
 Heterogeneity
 OS, arch., language
 Failures
 Failure transparency
 Stubs
15
Source: http://technet.microsoft.com/en-
us/library/cc738291%28v=ws.10%29.aspx

Data in Messages
 Data is “marshalled (encoded)” into a message &
“demarshalled (decoded)” from it
 Data structures
 Flattened on transmission
 Rebuilt upon reception
 Data types
 Base types – Ints, floats
 Flat types – structures, arrays
 Complex types – pointers
 With Web services we convert them to XML
 Issues
 Big-endian vs. little-endian, strings may require padding,
alignment (16/32/64-bits), Floating point representations 16

Asynchronous & Deferred
Synchronous RPC
17
Source: Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007

Directory Service
 Directory service or a UUDI registry
 UDDI - Universal Description, Discovery & Integration
18
Source: Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007

RPC – What Can Go Wrong?
 Failures
 Network failure
 Client failure
 Server failure
 Assuming only network idiosyncrasies for now…
 Ignoring possibility of process & machine crashes
 …Need to overcome
 Limited levels of message loss
 Message disorder/reorder
 Message duplication
19

Overcoming Only Communication
Failures
 Each message carry
 Unique sequence number
 Timestamp from a global clock (in most RPC protocols)
 Global clock assumed to return roughly same value throughout
network, up to clock synchronization limits
 Enables server to detect & discard
 Very old messages
 Duplicate copies
 RPCs use acknowledgements
 If timeout with no ack  resend packet
 Leads to the issue of replayed requests
20

Overcoming Lost Packets
21
Client Server
Time out
Sends request
Retransmit
Ack for request
Reply
Ack for reply

Overcoming Lost Packets (Cont.)
 Acks are expensive
 Retransmissions are costly
 For big messages, send
packets in bursts & Ack a
burst at a time
22

Machines Could Fail Too…
 What does a failed request mean?
 Network failure, machine failure or both!
 If a process fails, but machine remains
operational, OS will provide sufficient status
information to accurately determine “process
failure”
 Client that issued request would not know, if
server processed the request or not
23

RPC  RMI Evolution
 By expanding RPCs to invocation on remote
objects
 Object-oriented equivalent of RPC
 Distribution transparency is enhanced
 Separation between interfaces & objects
implementing these interfaces is crucial for
distributed systems
 State of remote objects is not distributed
 Only interfaces implemented by the object are
distributed (available on other machines)
24

RMI in Action
25

RMI in Action (Cont.)
26
Source: Introduction to Java Programming, Comprehensive Version (7th Edition) by Y. Daniel Liang

RMI in Action (Cont.)
 Client binds to a distributed object residing at a server
machine
 An implementation of object’s interface, called a proxy, is
loaded to the client’s address space
 Proxy – Analogous to a client stub in RPC
 Marshals message invocations into messages – object
serialization
 Unmarshals reply messages to return method invocation result to
client
 On server side
 Incoming invocation requests are passed to a server stub, called a
skeleton
 Unmarshals invocation request into proper method invocations
 Marshals replies & forwards reply messages to client side proxy27

Remote Objects & Remote Interfaces
 RMI is interface-oriented
 Remote interface implemented by remote object
 Only methods specified by remote interfaces can be
accessed via RMI
 A remote object may implement several sets of
remote interfaces simultaneously
28Source: Introduction to Java Programming,
Comprehensive Version (7th Edition) by Y. Daniel Liang

Remote Exceptions
 If a remote method throws an exception
 Remote side
 Catch all uncaught exceptions thrown from remote methods
 Find the caller
 Map to network request
 Client side
 Keep listening
 Any time an exception could be sent from a remote side
 Unmarshall network requests to exception objects
 Re-throw remote exception objects locally, as if exception
was thrown from current method
29

Remote Exceptions (Cont.)
 RMI software itself can also throw exceptions
 Network timeout
 Other system related
 They have accepted those failures & have decided to
expose it
30

Web Services
 RPC like Distributed Communication medium
that supports interoperability
 Not that much different from RMI, RPC, etc.,
except interoperability & lack of distributed
objects
 Use XML to communicate
 Request messages, description, & discovery all use
standard XML based formats
31

Web Services (Cont.)
 Publish, find, & use
32
UDDI – Universal Description,
Discovery & Integration

Web Services (Cont.)
 Are described using WSDL
 Web Service Description Language standard
 Published & found through a UUDI registry
 Invoked through SOAP standard
 XML based protocol for accessing Web Services
 Can transmit through any transport
 HTTP most common
 SMTP, JMS, UDP, TCP, VFS are known
33

Web Services Fundamentals
34

Summary
35