Communication in Distributed Systems

Research Scientist at Data61, CSIRO
Aug. 18, 2018

More Related Content


Communication in Distributed Systems

  1. Communication in Distributed Systems CS4262 Distributed Systems Dilum Bandara Some slides extracted from Dr. Srinath Perera & Dr. Rajkumar Buyya’s Presentation Deck
  2. Outline  Network Programming  Remote Procedure Calls  Remote Method Invocation  Message Oriented Communication 2
  3. Network Programming  Communication is typically over IP  HPC Clusters may use Infiniband or Myrinet  Unicast, multicast, anycast, broadcast  Only unicast is globally routable  2 main approaches  TCP – connection oriented  UDP – connection less  Lot more popular than you think! 3
  4. End-to-End Communication 4Source: Communication Networks by Alberto Leon-Garcia (Author), Indra Widjaja, 2000
  5. TCP Socket Calls 5 Source: Communication Networks by Alberto Leon- Garcia (Author), Indra Widjaja, 2000
  6. UDP Socket Calls 6 • No “handshake” • No simultaneous close • No fork() for concurrent servers Source: Communication Networks by Alberto Leon- Garcia (Author), Indra Widjaja, 2000
  7. Messages  Used to convey data among nodes  Has a limited size  Typically has a header  Multiple messages may be send through same connection  Identify message boundaries based on size, unique symbol patterns, etc.  Messages can be clear text, binary, XML, JSON, etc. 7
  8. Communication in Distributed Systems – History  Started with RPC  Went to distributed objects  Come back to RPC + Message Oriented Communication 8
  9. Remote Procedure Calls (RPCs)  Extend local procedure calls to work across computers  Call a procedure on a remote machine “just” as you would on local machine  Send the call as a message & get response/fault as a message 9 Source:
  10. RPC in Action… 10Source: Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007
  11. RPC – Steps 1. Client procedure calls client stub in normal way 2. Client stub builds message, calls Middleware 3. Middleware sends message to remote Middleware 4. Remote Middleware gives message to server Skeleton 5. Server stub unpacks parameters, calls local implementation 6. Local implementation does work, returns result to Skeleton 7. Skeleton packs it in message, calls local Middleware 8. Server's Middleware sends message to client's Middleware 9. Client's Middleware gives message to client stub 10. Stub unpacks result, returns to client 11
  12. RPCs  Introduced by Birrell & Nelson in 1984  Pre-RPC – Most applications were built directly over Internet primitives  Initial idea – Mask distributed computing system using a “transparent” abstraction  Looks like normal procedure call  Hides all aspects of distributed interaction  Supports an easy programming model  Today, RPC is the core of many distributed systems, e.g., Web Services  But now acknowledge other aspects like failures & asynchronous nature, communication latency more 12
  13. RPCs (Cont.) 13 Source -
  14. Message Exchange Patterns  With local calls, you call, & wait for response  But with RPC, there are other possibilities  Send/Receive  Send Only (Fire & Forget)  Send with Ack (Send Robust)  There are 2 models for client API  Polling  Callback 14
  15. RPC Concerns  How to make RPC similar to a local procedure call?  Communications  Location transparency  Parameter passing  Heterogeneity  OS, arch., language  Failures  Failure transparency  Stubs 15 Source: us/library/cc738291%28v=ws.10%29.aspx
  16. Data in Messages  Data is “marshalled (encoded)” into a message & “demarshalled (decoded)” from it  Data structures  Flattened on transmission  Rebuilt upon reception  Data types  Base types – Ints, floats  Flat types – structures, arrays  Complex types – pointers  With Web services we convert them to XML  Issues  Big-endian vs. little-endian, strings may require padding, alignment (16/32/64-bits), Floating point representations 16
  17. Asynchronous & Deferred Synchronous RPC 17 Source: Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007
  18. Directory Service  Directory service or a UUDI registry  UDDI - Universal Description, Discovery & Integration 18 Source: Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007
  19. RPC – What Can Go Wrong?  Failures  Network failure  Client failure  Server failure  Assuming only network idiosyncrasies for now…  Ignoring possibility of process & machine crashes  …Need to overcome  Limited levels of message loss  Message disorder/reorder  Message duplication 19
  20. Overcoming Only Communication Failures  Each message carry  Unique sequence number  Timestamp from a global clock (in most RPC protocols)  Global clock assumed to return roughly same value throughout network, up to clock synchronization limits  Enables server to detect & discard  Very old messages  Duplicate copies  RPCs use acknowledgements  If timeout with no ack  resend packet  Leads to the issue of replayed requests 20
  21. Overcoming Lost Packets 21 Client Server Time out Sends request Retransmit Ack for request Reply Ack for reply
  22. Overcoming Lost Packets (Cont.)  Acks are expensive  Retransmissions are costly  For big messages, send packets in bursts & Ack a burst at a time 22
  23. Machines Could Fail Too…  What does a failed request mean?  Network failure, machine failure or both!  If a process fails, but machine remains operational, OS will provide sufficient status information to accurately determine “process failure”  Client that issued request would not know, if server processed the request or not 23
  24. RPC  RMI Evolution  By expanding RPCs to invocation on remote objects  Object-oriented equivalent of RPC  Distribution transparency is enhanced  Separation between interfaces & objects implementing these interfaces is crucial for distributed systems  State of remote objects is not distributed  Only interfaces implemented by the object are distributed (available on other machines) 24
  25. RMI in Action 25
  26. RMI in Action (Cont.) 26 Source: Introduction to Java Programming, Comprehensive Version (7th Edition) by Y. Daniel Liang
  27. RMI in Action (Cont.)  Client binds to a distributed object residing at a server machine  An implementation of object’s interface, called a proxy, is loaded to the client’s address space  Proxy – Analogous to a client stub in RPC  Marshals message invocations into messages – object serialization  Unmarshals reply messages to return method invocation result to client  On server side  Incoming invocation requests are passed to a server stub, called a skeleton  Unmarshals invocation request into proper method invocations  Marshals replies & forwards reply messages to client side proxy27
  28. Remote Objects & Remote Interfaces  RMI is interface-oriented  Remote interface implemented by remote object  Only methods specified by remote interfaces can be accessed via RMI  A remote object may implement several sets of remote interfaces simultaneously 28Source: Introduction to Java Programming, Comprehensive Version (7th Edition) by Y. Daniel Liang
  29. Remote Exceptions  If a remote method throws an exception  Remote side  Catch all uncaught exceptions thrown from remote methods  Find the caller  Map to network request  Client side  Keep listening  Any time an exception could be sent from a remote side  Unmarshall network requests to exception objects  Re-throw remote exception objects locally, as if exception was thrown from current method 29
  30. Remote Exceptions (Cont.)  RMI software itself can also throw exceptions  Network timeout  Other system related  They have accepted those failures & have decided to expose it 30
  31. Web Services  RPC like Distributed Communication medium that supports interoperability  Not that much different from RMI, RPC, etc., except interoperability & lack of distributed objects  Use XML to communicate  Request messages, description, & discovery all use standard XML based formats 31
  32. Web Services (Cont.)  Publish, find, & use 32 UDDI – Universal Description, Discovery & Integration
  33. Web Services (Cont.)  Are described using WSDL  Web Service Description Language standard  Published & found through a UUDI registry  Invoked through SOAP standard  XML based protocol for accessing Web Services  Can transmit through any transport  HTTP most common  SMTP, JMS, UDP, TCP, VFS are known 33
  34. Web Services Fundamentals 34
  35. Summary 35

Editor's Notes

  1. Big-endian vs. little-endian -
  2. DCE RPC is a facility for calling a procedure on a remote machine as if it were a local procedure call
  3. UDDI a platform-independent, XML-based registry by which businesses worldwide can list themselves on the Internet, and a mechanism to register and locate web service applications.
  4. JMS – Java Message Service VFS – Virtual File System