c A. Kshemkalyani and M. Singhal, 2005
Limited circulation within a classroom, after seeking permission from authors.
1
DISTRIBUTED COMPUTING:
PRINCIPLES, ALGORITHMS, and SYSTEMS
Ajay D. Kshemkalyani
Department of Computer Science
University ...
“To my father Shri Digambar and my mother Shrimati Vimala.”
-Ajay D. Kshemkalyani
“To my mother Chandra Prabha Singhal, my...
Preface
Background
The field of Distributed Computing covers “all aspects of computing and information access across
multip...
recovery, authentication, global state and time, etc.) by presenting the theory behind and algo-
rithms for such problems....
Access to Resources
The following web sites will be maintained for the book. Any errors and comments should be sent
to aja...
Contents
1 Introduction 1
1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1....
2.6 Past and Future Cones of an Event . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.7 Models of Process Communi...
4.3.1 Chandy-Lamport Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3.2 Properties of the Recorded Globa...
5.5.1 Synchronous Single-Initiator Spanning Tree Algorithm Using Flooding:
Algorithm 0 . . . . . . . . . . . . . . . . . ....
6.4 Group Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6.5 Causal Order (CO) . . . . ...
7.7.4 A Static Termination Detection Algorithm . . . . . . . . . . . . . . . . . . 247
7.7.5 A Dynamic Termination Detecti...
9.4 Ricart-Agrawala Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
9.4.1 Description of the Alg...
10.4.1 The Single Resource Model . . . . . . . . . . . . . . . . . . . . . . . . . 345
10.4.2 The AND Model . . . . . . . ...
12 Distributed Shared Memory 399
12.1 Abstraction and Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....
13.5.2 Pessimistic Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459
13.5.3 Optimistic Logging . . . ....
14.5.7 Reliable Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532
14.6 Wait-free Shared Memory Con...
16.2.4 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588
16.2.5 Design Principles for Cr...
17.11The role of compilers in self-stabilization . . . . . . . . . . . . . . . . . . . . . . 649
17.11.1 Compilers for seq...
18.8 Tradeoffs between Table Storage and Route Lengths . . . . . . . . . . . . . . . . 696
18.8.1 Unifying DHT Protocols ....
Chapter 1
Introduction
1.1 Definition
A distributed system is a collection of independent entities that cooperate to solve ...
P
P P P
P
PP
M M M
MM
M M
Communication network
(WAN/ LAN)
P processor(s)
M memory bank(s)
Figure 1.1: A distributed syste...
protocols
distributed
Extent ofDistributed application
Network layer
(middleware libraries)
Application layer
Data link la...
are typically distributed across the system. For example, distributed databases such as DB2
partition the data sets across...
M
MP P P
PPP M M
M memory P processor
(b)(a)
Interconnection networkInterconnection network
PPP P
MMMM
M M
Figure 1.3: Two...
be a bus, although for greater efficiency, it is usually a multistage switch with a symmetric
and regular design.
Figure 1....
The two outgoing edges from any switch x, s are as follows. There is an edge from switch
x, s to switch y, s + 1 if (i) x ...
P0
P1
P2
P3
P4
P6
P7
101P5
000
001
M0
M1
010
011
100
101
110
111
001
101
110
111
100
111
110
100
011
010
000
M2010
000
001...
0010
0111
0011
0101
(b)(a)
processor + memory
1100
1000
1110
1010
1111
1011
1001
1101
0001
01100100
0000
Figure 1.5: Some ...
1.4.2 Flynn’s Taxonomy
Flynn identified four processing modes, based on whether the processors execute the same or
differen...
PP
CCCC
I I
II II
III
D
P
(c) MISD(b) MIMD(a) SIMD
data stream
Processing Unit
Control Unit
P
D
instruction streamI
CC
P
D...
programs. If the aggregate measure is a function of only the code, then the parallelism is indepen-
dent of the architectu...
1.5 Message Passing Systems versus Shared Memory Systems
Shared memory systems are those in which there is a (common) shar...
we will consider message-passing systems more extensively than we consider shared memory sys-
tems.
1.6 Primitives for Dis...
Send(X, destination, handlek) //handlek is a return parameter
...
...
Wait(handle1, handle2, . . ., handlek, . . . , handl...
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems  2008 (1)
Upcoming SlideShare
Loading in …5
×

Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems 2008 (1)

1,956 views
1,845 views

Published on

Distributed operating system.

Published in: Engineering
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,956
On SlideShare
0
From Embeds
0
Number of Embeds
27
Actions
Shares
0
Downloads
73
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Ajay d. kshemkalyani, mukesh singhal distributed computing principles, algorithms, and systems 2008 (1)

  1. 1. c A. Kshemkalyani and M. Singhal, 2005 Limited circulation within a classroom, after seeking permission from authors. 1
  2. 2. DISTRIBUTED COMPUTING: PRINCIPLES, ALGORITHMS, and SYSTEMS Ajay D. Kshemkalyani Department of Computer Science University of Illinois at Chicago Chicago, IL 60607 Mukesh Singhal Department of Computer Science University of Kentucky Lexington, KY 40506 January 29, 2007
  3. 3. “To my father Shri Digambar and my mother Shrimati Vimala.” -Ajay D. Kshemkalyani “To my mother Chandra Prabha Singhal, my father Brij Mohan Singhal, and my daughters Meenakshi, Malvika, and Priyanka.“ -Mukesh Singhal i
  4. 4. Preface Background The field of Distributed Computing covers “all aspects of computing and information access across multiple processing elements connected by any form of communication networks, whether local or wide-area in the coverage”. Since the advent of the Internet in the 1970s, there has been a steady growth of new applications requiring distributed processing. This was enabled by advances in networking and hardware technology, falling cost of hardware, and greater end-user awareness. These factors contributed to making distributed computing a cost-effective, high-performance, and fault-tolerant reality. Around the turn of the millenium, there has been an explosive growth in the expansion and efficiency of the Internet, and a matching outreach of access to networked resources through the World Wide Web, all across the world. Coupled with an equally dramatic growth in the wireless and mobile networking areas, and plummeting prices of bandwidth and storage devices, we are witnessing a rapid spurt in distributed applications and an accompanying interest in the field of distributed computing in universities, governments organizations, and private institutions. Advances in hardware technology have suddenly made sensor networking a reality, and em- bedded and sensor networks are rapidly becoming an integral part of each person’s life – from the home network with the interconnected gadgets to the automobile communicating by GPS (Global Positioning System), to the fully networked office with RFID monitoring. In the emerging global village, distributed computing will be the centerpiece of all computing and information access sub-disciplines within computer science. Clearly, this is a very important field. Moreover, this evolving field is characterized by a diverse range of challenges for which the solutions need to have foundations on solid principles. The field of distributed computing is very important, and there is a huge demand for a good comprehensive book. The book comprehensively covers all important topics in a great depth. This book provides both the depth and the breadth of coverage of topics in conjunction with the clarity of explanation and ease of understanding. The book will be particularly valuable to the academic community and the computer industry at large. Writing such a comprehensive book is an herculean task and is a huge undertaking. There is a deep sense of satisfaction in knowing that we were able complete this major task and perform this service to the community. Description, Approach, and Features The book will focus on fundamental principles and models underlying all aspects of distributed computing. The book will address the principles underlying the theory, algorithms, and systems aspects of distributed computing. The manner of presentation of the algorithms is very lucid, ex- plaining the main ideas and the intuition with figures and simple explanations rather than getting entangled in intimidating notations and lengthy and hard-to-follow rigorous proofs of the algo- rithms. The selection of chapter themes is broad and comprehensive, and the book covers all im- portant topics in depth. The selection of algorithms within each chapter has been done carefully to elucidate new and important techniques of algorithm design. Although the book focuses on foun- dational aspects and algorithms for distributed computing, it thoroughly addresses all practical systems-like problems (e.g., mutual exclusion, deadlock detection, termination detection, failure ii
  5. 5. recovery, authentication, global state and time, etc.) by presenting the theory behind and algo- rithms for such problems. The book is written keeping in mind the impact of emerging topics such as peer-to-peer computing and network security on the foundational aspects of distributed computing. Chapters of the book include figures, examples, exercise problems, a summary, and bibliogra- phy/references. An Index includes the index terms. Readership This book is aimed as a textbook for • Graduate students and Senior level undergraduate students in Computer Science and Com- puter Engineering. • Graduate students in Electrical Engineering and Mathematics. As wireless networks, peer- to-peer networks, and mobile computing continue to grow in importance, an increasing num- ber of students from Electrical Engineering Departments will also find this book necessary. • Practitioners, systems designers/programmers, and consultants in industry and research labs will find the book a very useful reference because it will contain the state of the art algorithms and principles to address various design issues in distributed systems, as well as the latest references. The breadth and depth of coverage, accompanied by clarity of explanation and ease of under- standing that the book offers will make it a very widely adopted textbook. Hard and soft prerequisites for the use of this book include • An undergraduate course in algorithms is required. • Undergraduate courses in operating systems and computer networks would be useful. • A reasonable familiarity with programming. We have aimed for a very comprehensive book that will be the single source that covers dis- tributed computing models and algorithms. The book will have both depth and breadth of coverage of topics, and will be characterized by clear and easy explanations. None of the existing textbooks on distributed computing provides all of these features. Acknowledgements This book grew from the notes used in the graduate courses on Distributed Computing at the Ohio State University, the University of Illinois at Chicago and at University of Kentucky. We would like to thank the graduate students at both these schools for their contributions to the book in many ways. The book is based on the published research results of numerous researchers in the field. We have made all efforts to present the material in our language and have given credit to original source of the information. We would like to thank all researchers whose work has been reported in this book. Finally, we would like to thank the staff of the Cambridge University Press for providing us with an excellent support for publication of the book. iii
  6. 6. Access to Resources The following web sites will be maintained for the book. Any errors and comments should be sent to ajayk@cs.uic.edu or singhal@cs.uky.edu. Further information about the book can be obtained from authors’ web pages. • http://www.cs.uic.edu/∼ajayk/DCS-Book • http://www.cs.uky.edu/∼singhal/DCS-Book Ajay D. Kshemkalyani Mukesh Singhal iv
  7. 7. Contents 1 Introduction 1 1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Relation to Computer System Components . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 Relation to Parallel Multiprocessor/Multicomputer Systems . . . . . . . . . . . . . 5 1.4.1 Characteristics of Parallel Systems . . . . . . . . . . . . . . . . . . . . . . 5 1.4.2 Flynn’s Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.3 Coupling, Parallelism, Concurrency, and Granularity . . . . . . . . . . . . 10 1.5 Message Passing Systems versus Shared Memory Systems . . . . . . . . . . . . . 13 1.5.1 Emulating message-passing on shared memory system (MP → SM). . . . 13 1.5.2 Emulating shared memory on a message-passing system (SM → MP). . . 13 1.6 Primitives for Distributed Communication . . . . . . . . . . . . . . . . . . . . . . 14 1.6.1 Blocking/Nonblocking, Synchronous/Asynchronous Primitives . . . . . . 14 1.6.2 Processor Synchrony . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.6.3 Libraries and Standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.7 Synchronous versus Asynchronous Executions . . . . . . . . . . . . . . . . . . . 18 1.7.1 Emulating an asynchronous system by a synchronous system (A → S). . . 19 1.7.2 Emulating a synchronous system by an asynchronous system (S → A). . . 19 1.7.3 Emulations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 1.8 Design Issues and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 1.8.1 Distributed Systems Challenges from a System Perspective . . . . . . . . . 21 1.8.2 Algorithmic Challenges in Distributed Computing . . . . . . . . . . . . . 22 1.8.3 Applications of Distributed Computing and Newer Challenges . . . . . . . 28 1.9 Selection and Coverage of Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.10 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.11 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.12 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 2 A Model of Distributed Computations 37 2.1 A Distributed Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 2.2 A Model of Distributed Executions . . . . . . . . . . . . . . . . . . . . . . . . . . 38 2.3 Models of Communication Network . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4 Global State of a Distributed System . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4.1 Global State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 2.5 Cuts of a Distributed Computation . . . . . . . . . . . . . . . . . . . . . . . . . . 42 v
  8. 8. 2.6 Past and Future Cones of an Event . . . . . . . . . . . . . . . . . . . . . . . . . . 43 2.7 Models of Process Communications . . . . . . . . . . . . . . . . . . . . . . . . . 44 3 Logical Time 47 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.2 A Framework for a System of Logical Clocks . . . . . . . . . . . . . . . . . . . . 48 3.2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.2.2 Implementing Logical Clocks . . . . . . . . . . . . . . . . . . . . . . . . 49 3.3 Scalar Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.3.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.3.2 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.4 Vector Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.4.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.4.2 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.4.3 On the Size of Vector Clocks . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.5 Efficient Implementations of Vector Clocks . . . . . . . . . . . . . . . . . . . . . 56 3.5.1 Singhal-Kshemkalyani’s Differential Technique . . . . . . . . . . . . . . . 56 3.5.2 Fowler-Zwaenepoel’s Direct-Dependency Technique . . . . . . . . . . . . 58 3.6 Jard-Jourdan’s Adaptive Technique . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.7 Matrix Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.7.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.7.2 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.8 Virtual Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.8.1 Virtual Time Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.8.2 Comparison with Lamport’s Logical Clocks . . . . . . . . . . . . . . . . . 66 3.8.3 Time Warp Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.8.4 The Local Control Mechanism . . . . . . . . . . . . . . . . . . . . . . . . 67 3.8.5 Global Control Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . 69 3.8.6 An Example: Distributed Discrete Event Simulations . . . . . . . . . . . . 71 3.9 Physical Clock Synchronization: NTP . . . . . . . . . . . . . . . . . . . . . . . . 72 3.9.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.9.2 Definitions and Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.9.3 Clock Inaccuracies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.10 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.11 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.12 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4 Global State and Snapshot Recording Algorithms 82 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.2 System Model and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.2.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.2.2 A Consistent Global State . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.2.3 Interpretation in Terms of Cuts . . . . . . . . . . . . . . . . . . . . . . . . 86 4.2.4 Issues in Recording a Global State . . . . . . . . . . . . . . . . . . . . . . 86 4.3 Snapshot Algorithms for FIFO Channels . . . . . . . . . . . . . . . . . . . . . . . 87 vi
  9. 9. 4.3.1 Chandy-Lamport Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.3.2 Properties of the Recorded Global State . . . . . . . . . . . . . . . . . . . 89 4.4 Variations of the Chandy-Lamport Algorithm . . . . . . . . . . . . . . . . . . . . 91 4.4.1 Spezialetti-Kearns Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 91 4.4.2 Venkatesan’s Incremental Snapshot Algorithm . . . . . . . . . . . . . . . 92 4.4.3 Helary’s Wave Synchronization Method . . . . . . . . . . . . . . . . . . . 93 4.5 Snapshot Algorithms for Non-FIFO Channels . . . . . . . . . . . . . . . . . . . . 94 4.5.1 Lai-Yang Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.5.2 Li et al.’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 4.5.3 Mattern’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.6 Snapshots in a Causal Delivery System . . . . . . . . . . . . . . . . . . . . . . . . 98 4.6.1 Process State Recording . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.6.2 Channel State Recording in Acharya-Badrinath Algorithm . . . . . . . . . 99 4.6.3 Channel State Recording in Alagar-Venkatesan Algorithm . . . . . . . . . 100 4.7 Monitoring Global State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 4.8 Necessary and Sufficient Conditions for Consistent Global Snapshots . . . . . . . 102 4.8.1 Zigzag Paths and Consistent Global Snapshots . . . . . . . . . . . . . . . 103 4.9 Finding Consistent Global Snapshots in a Distributed Computation . . . . . . . . . 106 4.9.1 Finding Consistent Global Snapshots . . . . . . . . . . . . . . . . . . . . 106 4.9.2 Manivannan-Netzer-Singhal Algorithm for Enumerating Consistent Snap- shots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.9.3 Finding Z-paths in a Distributed Computation . . . . . . . . . . . . . . . . 110 4.10 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.11 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 4.12 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 5 Terminology and Basic Algorithms 118 5.1 Topology Abstraction and Overlays . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.2 Classifications and Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.2.1 Application Executions and Control Algorithm Executions . . . . . . . . . 120 5.2.2 Centralized and Distributed Algorithms . . . . . . . . . . . . . . . . . . . 120 5.2.3 Symmetric and Asymmetric Algorithms . . . . . . . . . . . . . . . . . . . 121 5.2.4 Anonymous Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.2.5 Uniform Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.2.6 Adaptive Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.2.7 Deterministic Versus Nondeterministic Executions . . . . . . . . . . . . . 122 5.2.8 Execution Inhibition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.2.9 Synchronous and Asynchronous Systems . . . . . . . . . . . . . . . . . . 124 5.2.10 Online versus Offline Algorithms . . . . . . . . . . . . . . . . . . . . . . 124 5.2.11 Failure Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.2.12 Wait-free algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.2.13 Communication Channels . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.3 Complexity Measures and Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.4 Program Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.5 Elementary Graph Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 vii
  10. 10. 5.5.1 Synchronous Single-Initiator Spanning Tree Algorithm Using Flooding: Algorithm 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 5.5.2 Asynchronous Single-Initiator Spanning Tree Algorithm Using Flooding: Algorithm I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 5.5.3 Asynchronous Concurrent-Initiator Spanning Tree Algorithm Using Flood- ing: Algorithm II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 5.5.4 Asynchronous Concurrent-Initiator Depth First Search Spanning Tree Al- gorithm: Algorithm III . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 5.5.5 Broadcast and Convergecast on a Tree . . . . . . . . . . . . . . . . . . . . 137 5.5.6 Single Source Shortest Path Algorithm: Synchronous Bellman-Ford . . . . 140 5.5.7 Distance Vector Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 5.5.8 Single Source Shortest Path Algorithm: Asynchronous Bellman-Ford . . . 141 5.5.9 All Sources Shortest Paths: Asynchronous Distributed Floyd-Warshall . . . 142 5.5.10 Asynchronous and Synchronous Constrained Flooding (w/o a Spanning Tree) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 5.5.11 Minimum Weight Spanning Tree (MST) Algorithm in a Synchronous System146 5.5.12 Minimum Weight Spanning Tree (MST) in an Asynchronous System . . . 152 5.6 Synchronizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 5.7 Maximal Independent Set (MIS) . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 5.8 Connected Dominating Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 5.9 Compact Routing Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 5.10 Leader Election . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 5.11 Challenges in Designing Distributed Graph Algorithms . . . . . . . . . . . . . . . 164 5.12 Object Replication Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.12.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5.12.2 Algorithm Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.12.3 Reads and Writes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 5.12.4 Converging to an Replication Scheme . . . . . . . . . . . . . . . . . . . . 167 5.13 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.14 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 5.15 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 6 Message Ordering and Group Communication 179 6.1 Message Ordering Paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.1.1 Asynchronous Executions . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.1.2 FIFO Executions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.1.3 Causally Ordered (CO) Executions . . . . . . . . . . . . . . . . . . . . . 181 6.1.4 Synchronous Execution (SYNC) . . . . . . . . . . . . . . . . . . . . . . . 184 6.2 Asynchronous Execution with Synchronous Communication . . . . . . . . . . . . 184 6.2.1 Executions Realizable with Synchronous Communication (RSC) . . . . . . 185 6.2.2 Hierarchy of Ordering Paradigms . . . . . . . . . . . . . . . . . . . . . . 188 6.2.3 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 6.3 Synchronous Program Order on an Asynchronous System . . . . . . . . . . . . . . 190 6.3.1 Rendezvous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 6.3.2 Algorithm for Binary Rendezvous . . . . . . . . . . . . . . . . . . . . . . 192 viii
  11. 11. 6.4 Group Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 6.5 Causal Order (CO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 6.5.1 The Raynal-Schiper-Toueg Algorithm . . . . . . . . . . . . . . . . . . . . 197 6.5.2 The Kshemkalyani-Singhal Optimal Algorithm . . . . . . . . . . . . . . . 198 6.6 Total Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 6.6.1 Centralized Algorithm for Total Order . . . . . . . . . . . . . . . . . . . . 205 6.6.2 Three-Phase Distributed Algorithm . . . . . . . . . . . . . . . . . . . . . 206 6.7 A Nomenclature For Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 6.8 Propagation Trees For Multicast . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 6.9 Classification of Application-Level Multicast Algorithms . . . . . . . . . . . . . . 214 6.10 Semantics of Fault-Tolerant Group Communication . . . . . . . . . . . . . . . . . 216 6.11 Distributed Multicast Algorithms At The Network Layer . . . . . . . . . . . . . . 218 6.11.1 Reverse Path Forwarding (RPF) For Constrained Flooding . . . . . . . . . 219 6.11.2 Steiner Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 6.11.3 Multicast Cost Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 6.11.4 Delay-Bounded Steiner Trees . . . . . . . . . . . . . . . . . . . . . . . . 221 6.11.5 Core-Based Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 6.12 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 6.13 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 6.14 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 7 Termination Detection 230 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230 7.2 System Model of a Distributed Computation . . . . . . . . . . . . . . . . . . . . . 231 7.3 Termination Detection Using Distributed Snapshots . . . . . . . . . . . . . . . . . 232 7.3.1 Informal Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 7.3.2 Formal Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 7.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 7.4 Termination Detection by Weight Throwing . . . . . . . . . . . . . . . . . . . . . 234 7.4.1 Formal Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 7.4.2 Correctness of the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 235 7.5 A Spanning-Tree-Based Termination Detection Algorithm . . . . . . . . . . . . . 236 7.5.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 7.5.2 A Simple Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 7.5.3 The Correct Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 7.5.4 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 7.5.5 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 7.6 Message-Optimal Termination Detection . . . . . . . . . . . . . . . . . . . . . . . 242 7.6.1 The Main Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 7.6.2 Formal Description of the Algorithm . . . . . . . . . . . . . . . . . . . . 243 7.6.3 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 7.7 Termination Detection in a Very General Distributed Computing Model . . . . . . 245 7.7.1 Model Definition and Assumptions . . . . . . . . . . . . . . . . . . . . . 246 7.7.2 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 7.7.3 Termination Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 ix
  12. 12. 7.7.4 A Static Termination Detection Algorithm . . . . . . . . . . . . . . . . . . 247 7.7.5 A Dynamic Termination Detection Algorithm . . . . . . . . . . . . . . . . 249 7.8 Termination Detection in the Atomic Computation Model . . . . . . . . . . . . . . 251 7.8.1 The Atomic Model of Execution . . . . . . . . . . . . . . . . . . . . . . . 252 7.8.2 A Naive Counting Method . . . . . . . . . . . . . . . . . . . . . . . . . . 252 7.8.3 The Four Counter Method . . . . . . . . . . . . . . . . . . . . . . . . . . 253 7.8.4 The Sceptic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 7.8.5 The Time Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 7.8.6 Vector Counters Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 7.8.7 A Channel Counting Method . . . . . . . . . . . . . . . . . . . . . . . . . 258 7.9 Termination Detection in a Faulty Distributed System . . . . . . . . . . . . . . . . 260 7.9.1 Flow Detecting Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 7.9.2 Taking Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 7.9.3 Description of the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 263 7.9.4 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266 7.10 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 7.11 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 8 Reasoning with Knowledge 271 8.1 The Muddy Children Puzzle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 8.2 Logic of Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 8.2.1 Knowledge Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 8.2.2 The Muddy Children Puzzle Again . . . . . . . . . . . . . . . . . . . . . 273 8.2.3 Kripke Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 8.2.4 Muddy Children Puzzle using Kripke Structures . . . . . . . . . . . . . . 275 8.2.5 Properties of Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 8.3 Knowledge in Synchronous Systems . . . . . . . . . . . . . . . . . . . . . . . . . 277 8.4 Knowledge in Asynchronous Systems . . . . . . . . . . . . . . . . . . . . . . . . 278 8.4.1 Logic and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 8.4.2 Agreement in Asynchronous Systems . . . . . . . . . . . . . . . . . . . . 279 8.4.3 Variants of Common Knowledge . . . . . . . . . . . . . . . . . . . . . . . 280 8.4.4 Concurrent Common Knowledge . . . . . . . . . . . . . . . . . . . . . . 281 8.5 Knowledge Transfer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 8.6 Knowledge and Clocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 8.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 8.8 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 8.9 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 9 Distributed Mutual Exclusion Algorithms 293 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 9.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 9.2.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 9.2.2 Requirements of Mutual Exclusion Algorithms . . . . . . . . . . . . . . . 294 9.2.3 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 9.3 Lamport’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 x
  13. 13. 9.4 Ricart-Agrawala Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 9.4.1 Description of the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 300 9.5 Singhal’s Dynamic Information-Structure Algorithm . . . . . . . . . . . . . . . . 303 9.5.1 The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 9.5.2 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 9.5.3 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308 9.5.4 Adaptivity in Heterogeneous Traffic Patterns . . . . . . . . . . . . . . . . 309 9.6 Lodha and Kshemkalyani’s Fair Mutual Exclusion Algorithm . . . . . . . . . . . . 309 9.6.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 9.6.2 The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 9.6.3 Safety, Fairness and Liveness . . . . . . . . . . . . . . . . . . . . . . . . 313 9.6.4 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313 9.6.5 Message Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313 9.7 Quorum-Based Mutual Exclusion Algorithms . . . . . . . . . . . . . . . . . . . . 316 9.8 Maekawa’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 9.8.1 The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 9.8.2 Problem of Deadlocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 9.9 Agarwal-El Abbadi Quorum-Based Algorithm . . . . . . . . . . . . . . . . . . . . 320 9.9.1 Constructing a tree-structured quorum . . . . . . . . . . . . . . . . . . . . 320 9.9.2 Analysis of the algorithm for constructing tree-structured quorums . . . . . 321 9.9.3 Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 9.9.4 Examples of Tree-Structured Quorums . . . . . . . . . . . . . . . . . . . 322 9.9.5 The Algorithm for Distributed Mutual Exclusion . . . . . . . . . . . . . . 323 9.9.6 Correctness proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 9.9.7 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 9.10 Token-Based Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324 9.11 Suzuki-Kasami’s Broadcast Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 325 9.12 Raymond’s Tree-Based Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 327 9.12.1 The HOLDER Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 9.12.2 The Operation of the Algorithm . . . . . . . . . . . . . . . . . . . . . . . 329 9.12.3 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 9.12.4 Cost and Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . 335 9.12.5 Algorithm Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 9.12.6 Node Failures and Recovery . . . . . . . . . . . . . . . . . . . . . . . . . 336 9.13 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 9.14 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 10 Deadlock Detection in Distributed Systems 342 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 10.2 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 10.2.1 Wait-For-Graph (WFG) . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 10.3 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 10.3.1 Deadlock Handling Strategies . . . . . . . . . . . . . . . . . . . . . . . . 343 10.3.2 Issues in Deadlock Detection . . . . . . . . . . . . . . . . . . . . . . . . . 344 10.4 Models of Deadlocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 xi
  14. 14. 10.4.1 The Single Resource Model . . . . . . . . . . . . . . . . . . . . . . . . . 345 10.4.2 The AND Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 10.4.3 The OR Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 10.4.4 The AND-OR Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 10.4.5 The p q Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 10.4.6 Unrestricted Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 10.5 Knapp’s Classification of Distributed Deadlock Detection Algorithms . . . . . . . 348 10.5.1 Path-Pushing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 10.5.2 Edge-Chasing Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 348 10.5.3 Diffusing Computations Based Algorithms . . . . . . . . . . . . . . . . . 348 10.5.4 Global State Detection Based Algorithms . . . . . . . . . . . . . . . . . . 349 10.6 Mitchell and Merritt’s Algorithm for the Single-Resource Model . . . . . . . . . . 349 10.7 Chandy-Misra-Haas Algorithm for the AND Model . . . . . . . . . . . . . . . . . 352 10.8 Chandy-Misra-Haas Algorithm for the OR Model . . . . . . . . . . . . . . . . . . 353 10.9 Kshemkalyani-Singhal Algorithm for P-out-of-Q Model . . . . . . . . . . . . . . 355 10.9.1 Informal Description of the Algorithm . . . . . . . . . . . . . . . . . . . . 357 10.9.2 The Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 10.9.3 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 10.10Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 10.11Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 10.12Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365 11 Global Predicate Detection 370 11.1 Stable and Unstable Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 11.1.1 Stable Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 11.1.2 Unstable Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372 11.2 Modalities on Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 11.2.1 Complexity of Predicate Detection . . . . . . . . . . . . . . . . . . . . . . 374 11.3 Centralized Algorithm for Relational Predicates . . . . . . . . . . . . . . . . . . . 374 11.4 Conjunctive Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 378 11.4.1 Interval-based Centralized Algorithm for Conjunctive Predicates . . . . . . 379 11.4.2 Global State based Centralized Algorithm for Possibly(φ), where φ is conjunctive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381 11.5 Distributed Algorithms for Conjunctive Predicates . . . . . . . . . . . . . . . . . . 384 11.5.1 Distributed State-based Token Algorithm for Possibly(φ), where φ is Con- junctive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 11.5.2 Distributed Interval-based Token Algorithm for Definitely(φ), where φ is Conjunctive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 386 11.5.3 Distributed Interval-based Piggybacking Algorithm for Possibly(φ), where φ is Conjunctive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 390 11.6 Further Classification of Predicates . . . . . . . . . . . . . . . . . . . . . . . . . . 393 11.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 11.8 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 11.9 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 xii
  15. 15. 12 Distributed Shared Memory 399 12.1 Abstraction and Advantages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399 12.2 Memory Consistency Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 12.2.1 Strict consistency/Atomic consistency/Linearizability . . . . . . . . . . . . 403 12.2.2 Sequential Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406 12.2.3 Causal Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 12.2.4 PRAM (Pipelined RAM) or Processor Consistency . . . . . . . . . . . . . 411 12.2.5 Slow Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 12.2.6 Hierarchy of Consistency Models . . . . . . . . . . . . . . . . . . . . . . 413 12.2.7 Other Models based on Synchronization Instructions . . . . . . . . . . . . 414 12.3 Shared Memory Mutual Exclusion . . . . . . . . . . . . . . . . . . . . . . . . . . 416 12.3.1 Lamport’s Bakery Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 416 12.3.2 Lamport’s WRWR Mechanism and Fast Mutual Exclusion . . . . . . . . . 418 12.3.3 Hardware support for mutual exclusion . . . . . . . . . . . . . . . . . . . 421 12.4 Wait-freedom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422 12.5 Register Hierarchy and Wait-free Simulations . . . . . . . . . . . . . . . . . . . . 423 12.5.1 Construction 1: SRSW Safe to MRSW Safe . . . . . . . . . . . . . . . . . 426 12.5.2 Construction 2: SRSW Regular to MRSW Regular . . . . . . . . . . . . . 427 12.5.3 Construction 3: Boolean MRSW Safe to integer-valued MRSW Safe . . . . 427 12.5.4 Construction 4: Boolean MRSW Safe to boolean MRSW Regular . . . . . 427 12.5.5 Construction 5: Boolean MRSW Regular to integer-valued MRSW Regular 428 12.5.6 Construction 6: Boolean MRSW Regular to integer-valued MRSW Atomic 429 12.5.7 Construction 7: Integer MRSW Atomic to integer MRMW Atomic . . . . 432 12.5.8 Construction 8: Integer SRSW Atomic to integer MRSW Atomic . . . . . 433 12.6 Wait-free Atomic Snapshots of Shared Objects . . . . . . . . . . . . . . . . . . . 435 12.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 12.8 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440 12.9 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440 13 Checkpointing and Rollback Recovery 445 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 13.2 Background and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446 13.2.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446 13.2.2 A Local Checkpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 13.2.3 Consistent System States . . . . . . . . . . . . . . . . . . . . . . . . . . . 447 13.2.4 Interactions with the Outside World . . . . . . . . . . . . . . . . . . . . . 448 13.2.5 Different Types of Messages . . . . . . . . . . . . . . . . . . . . . . . . . 449 13.3 Issues in Failure Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450 13.4 Checkpoint Based Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452 13.4.1 Uncoordinated Checkpointing . . . . . . . . . . . . . . . . . . . . . . . . 452 13.4.2 Coordinated Checkpointing . . . . . . . . . . . . . . . . . . . . . . . . . 454 13.4.3 Impossibility of Min Process Non-blocking Checkpointing . . . . . . . . . 456 13.4.4 Communication-Induced Checkpointing . . . . . . . . . . . . . . . . . . . 456 13.5 Log-based Rollback Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458 13.5.1 Deterministic and Nondeterministic Events . . . . . . . . . . . . . . . . . 458 xiii
  16. 16. 13.5.2 Pessimistic Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 459 13.5.3 Optimistic Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 13.5.4 Causal Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462 13.6 Koo-Toueg Coordinated Checkpointing Algorithm . . . . . . . . . . . . . . . . . 463 13.6.1 The Checkpointing Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 463 13.6.2 The Rollback Recovery Algorithm . . . . . . . . . . . . . . . . . . . . . . 465 13.7 Juang and Venkatesan Algorithm for Asynchronous Checkpointing and Recovery . 466 13.7.1 System Model and Assumptions . . . . . . . . . . . . . . . . . . . . . . . 466 13.7.2 Asynchronous Checkpointing . . . . . . . . . . . . . . . . . . . . . . . . 467 13.7.3 The Recovery Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 13.7.4 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 13.8 Manivannan-Singhal Quasi-Synchronous Checkpointing Algorithm . . . . . . . . 470 13.8.1 Checkpointing Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 471 13.8.2 Recovery Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473 13.8.3 Comprehensive Message Handling . . . . . . . . . . . . . . . . . . . . . . 476 13.9 Peterson-Kearns Algorithm Based on Vector Time . . . . . . . . . . . . . . . . . . 479 13.9.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 13.9.2 Informal Description of the Algorithm . . . . . . . . . . . . . . . . . . . . 480 13.9.3 Formal Description of the RollBack Protocol . . . . . . . . . . . . . . . . 482 13.9.4 An Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 13.9.5 Correctness Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484 13.10Helary-Mostefaoui-Netzer-Raynal Communication-induced Protocol . . . . . . . . 486 13.10.1 Design Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487 13.10.2 The Checkpointing Protocol . . . . . . . . . . . . . . . . . . . . . . . . . 491 13.11Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 13.12Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495 14 Consensus and Agreement Algorithms 500 14.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500 14.1.1 The Byzantine Agreement and Other Problems . . . . . . . . . . . . . . . 502 14.1.2 Equivalence of the Problems and Notations . . . . . . . . . . . . . . . . . 503 14.2 Overview of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504 14.3 Agreement in a Failure-Free System (Synchronous or Asynchronous) . . . . . . . 505 14.4 Agreement in (Message-Passing) Synchronous Systems with Failures . . . . . . . 506 14.4.1 Consensus Algorithm for Crash Failures (Synchronous System) . . . . . . 506 14.4.2 Consensus Algorithms for Byzantine Failures (Synchronous System) . . . 507 14.4.3 Upper Bound on Byzantine Processes . . . . . . . . . . . . . . . . . . . . 507 14.4.4 Byzantine Agreement Tree Algorithm: Exponential (Synchronous System) 509 14.5 Agreement in Asynchronous Message-Passing Systems with Failures . . . . . . . . 518 14.5.1 Impossibility Result for the Consensus Problem . . . . . . . . . . . . . . . 518 14.5.2 Terminating Reliable Broadcast . . . . . . . . . . . . . . . . . . . . . . . 520 14.5.3 Distributed Transaction Commit . . . . . . . . . . . . . . . . . . . . . . . 521 14.5.4 k-set consensus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521 14.5.5 Approximate Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . 522 14.5.6 Renaming Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527 xiv
  17. 17. 14.5.7 Reliable Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 532 14.6 Wait-free Shared Memory Consensus in Asynchronous Systems . . . . . . . . . . 533 14.6.1 Impossibility Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533 14.6.2 Consensus Numbers and Consensus Hierarchy . . . . . . . . . . . . . . . 536 14.6.3 Universality of Consensus Objects . . . . . . . . . . . . . . . . . . . . . . 540 14.6.4 Shared Memory k-set Consensus . . . . . . . . . . . . . . . . . . . . . . . 545 14.6.5 Shared Memory Renaming . . . . . . . . . . . . . . . . . . . . . . . . . . 545 14.6.6 Shared Memory Renaming using Splitters . . . . . . . . . . . . . . . . . . 547 14.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 549 14.8 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 550 14.9 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 552 15 Failure Detectors 555 15.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555 15.2 Unreliable Failure Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556 15.2.1 The System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 556 15.2.2 Failure Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 557 15.2.3 Completeness and Accuracy Properties . . . . . . . . . . . . . . . . . . . 557 15.2.4 Types of Failure Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . 560 15.2.5 Reducibility of Failure Detectors . . . . . . . . . . . . . . . . . . . . . . . 560 15.2.6 Reducing Weak Failure Detector W to a Strong Failure Detector S . . . . . 561 15.2.7 Reducing an Eventually Weak Failure Detector ♦W to an Eventually Strong Failure Detector ♦S . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563 15.3 The Consensus Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565 15.3.1 Solutions to the Consensus Problem . . . . . . . . . . . . . . . . . . . . . 566 15.3.2 A Solution Using Strong Failure Detector S . . . . . . . . . . . . . . . . . 566 15.3.3 A Solution Using Eventually Strong Failure Detector ♦S . . . . . . . . . . 568 15.4 Atomic Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 570 15.5 A Solution to Atomic Broadcast . . . . . . . . . . . . . . . . . . . . . . . . . . . 571 15.6 The Weakest Failure Detectors to Solve Fundamental Agreement Problems . . . . 573 15.6.1 Realistic Failure Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . 574 15.6.2 The weakest failure detector for consensus . . . . . . . . . . . . . . . . . 575 15.6.3 The Weakest Failure Detector for Terminating Reliable Broadcast . . . . . 576 15.7 An Implementation of a Failure Detector . . . . . . . . . . . . . . . . . . . . . . . 576 15.8 An Adaptive Failure Detection Protocol . . . . . . . . . . . . . . . . . . . . . . . 578 15.8.1 Lazy Failure Detection Protocol (FDL) . . . . . . . . . . . . . . . . . . . 579 15.9 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582 15.10Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 582 16 Authentication in Distributed System 586 16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586 16.2 Background and Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586 16.2.1 Basis of Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587 16.2.2 Types of Principals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 587 16.2.3 A Simple Classification of Authentication Protocols . . . . . . . . . . . . 588 xv
  18. 18. 16.2.4 Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 588 16.2.5 Design Principles for Cryptographic Protocols . . . . . . . . . . . . . . . 589 16.3 Protocols Based on Symmetric Cryptosystems . . . . . . . . . . . . . . . . . . . . 590 16.3.1 Basic Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 590 16.3.2 Modified Protocol with Nonce . . . . . . . . . . . . . . . . . . . . . . . . 591 16.3.3 Wide-Mouth Frog Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . 592 16.3.4 A Protocol Based On an Authentication Server . . . . . . . . . . . . . . . 593 16.3.5 One-Time Password Scheme . . . . . . . . . . . . . . . . . . . . . . . . . 594 16.3.6 Otway-Rees Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596 16.3.7 Kerberos Authentication Service . . . . . . . . . . . . . . . . . . . . . . . 597 16.4 Protocols Based on Asymmetric Cryptosystems . . . . . . . . . . . . . . . . . . . 602 16.4.1 The Basic Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 602 16.4.2 A Modified Protocol with a Certification Authority . . . . . . . . . . . . . 602 16.4.3 Needham and Schroeder Protocol . . . . . . . . . . . . . . . . . . . . . . 603 16.4.4 SSL Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605 16.5 Password-based Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609 16.5.1 Encrypted Key Exchange (EKE) Protocol . . . . . . . . . . . . . . . . . . 609 16.5.2 Secure Remote Password (SRP) Protocol . . . . . . . . . . . . . . . . . . 610 16.6 Authentication Protocol Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . 611 16.7 Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613 16.8 Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613 17 Self-Stabilization 619 17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619 17.2 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620 17.3 Definition of Self-Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 622 17.4 Issues in the design of self-stabilization algorithms . . . . . . . . . . . . . . . . . 624 17.4.1 The Number of States in Each of the Individual Units . . . . . . . . . . . . 625 17.4.2 Uniform Vs. Non-uniform Networks . . . . . . . . . . . . . . . . . . . . 631 17.4.3 Central and Distributed Demons . . . . . . . . . . . . . . . . . . . . . . . 632 17.4.4 Reducing the number of states in a token ring . . . . . . . . . . . . . . . . 633 17.4.5 Shared memory Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 633 17.4.6 Mutual Exclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634 17.4.7 Costs of self-stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . 634 17.5 Methodologies for designing self-stabilizing systems . . . . . . . . . . . . . . . . 635 17.6 Communication Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637 17.7 Self-Stabilizing Distributed Spanning Trees . . . . . . . . . . . . . . . . . . . . . 638 17.8 Self-Stabilizing Algorithms for Spanning-tree Construction . . . . . . . . . . . . . 640 17.8.1 Dolev, Israeli, and Moran Algorithm . . . . . . . . . . . . . . . . . . . . . 640 17.8.2 Afek, Kutten, and Yung Algorithm for Spanning-tree Construction . . . . . 642 17.8.3 Arora and Gouda Algorithm for Spanning-tree Construction . . . . . . . . 643 17.8.4 Huang et al. Algorithms for Spanning-tree Construction . . . . . . . . . . 643 17.8.5 Afek and Bremler Algorithm for Spanning-tree Construction . . . . . . . . 644 17.9 An anonymous self-stabilizing algorithm for 1-maximal independent set in trees . . 645 17.10A Probabilistic Self-Stabilizing Leader Election Algorithm . . . . . . . . . . . . . 646 xvi
  19. 19. 17.11The role of compilers in self-stabilization . . . . . . . . . . . . . . . . . . . . . . 649 17.11.1 Compilers for sequential programs . . . . . . . . . . . . . . . . . . . . . . 649 17.11.2 Compilers for asynchronous message passing systems . . . . . . . . . . . 650 17.11.3 Compilers for asynchronous shared memory systems . . . . . . . . . . . . 651 17.12Self stabilization as a Solution to Fault Tolerance . . . . . . . . . . . . . . . . . . 652 17.13Factors Preventing Self-Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . 654 17.14Limitations of Self-Stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . . 655 17.15Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 17.16Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 17.17Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658 18 Peer-to-Peer Computing and Overlay Graphs 666 18.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 666 18.1.1 Napster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 667 18.1.2 Application Layer Overlays . . . . . . . . . . . . . . . . . . . . . . . . . 667 18.2 Data Indexing and Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 668 18.2.1 Distributed Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669 18.3 Unstructured Overlays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 670 18.3.1 Unstructured Overlays: Properties . . . . . . . . . . . . . . . . . . . . . . 670 18.3.2 Gnutella . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 670 18.3.3 Search in Gnutella and Unstructured Overlays . . . . . . . . . . . . . . . . 671 18.3.4 Replication Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 673 18.3.5 Implementing Replication Strategies. . . . . . . . . . . . . . . . . . . . . 675 18.4 Chord Distributed Hash Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676 18.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 676 18.4.2 Simple lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 677 18.4.3 Scalable Lookup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678 18.4.4 Managing Churn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 678 18.4.5 Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 682 18.5 Content Addressible Networks: CAN . . . . . . . . . . . . . . . . . . . . . . . . 683 18.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 683 18.5.2 CAN Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684 18.5.3 CAN Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 685 18.5.4 CAN Maintainence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 686 18.5.5 CAN Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 688 18.5.6 CAN Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689 18.6 Tapestry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689 18.6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689 18.6.2 Overlay and Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 689 18.6.3 Object Publication and Object Search . . . . . . . . . . . . . . . . . . . . 692 18.6.4 Node Insertion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 693 18.6.5 Node Deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 694 18.7 Some Other Challenges in P2P System Design . . . . . . . . . . . . . . . . . . . . 695 18.7.1 Fairness: A Game Theory Application . . . . . . . . . . . . . . . . . . . . 695 18.7.2 Trust or Reputation Management . . . . . . . . . . . . . . . . . . . . . . 696 xvii
  20. 20. 18.8 Tradeoffs between Table Storage and Route Lengths . . . . . . . . . . . . . . . . 696 18.8.1 Unifying DHT Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . 696 18.8.2 Bounds on DHT Storage and Routing Distance . . . . . . . . . . . . . . . 697 18.9 Graph Structures of Complex Networks . . . . . . . . . . . . . . . . . . . . . . . 699 18.10Internet graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 701 18.10.1 Basic Laws and their Definitions . . . . . . . . . . . . . . . . . . . . . . . 701 18.10.2 Properties of the Internet . . . . . . . . . . . . . . . . . . . . . . . . . . . 702 18.10.3 Error and Attack Tolerance of Complex Networks . . . . . . . . . . . . . 704 18.11Random Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707 18.11.1 Graph Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707 18.11.2 Graph Degree Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 708 18.11.3 Graph Diameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 708 18.11.4 Graph Clustering Coefficient . . . . . . . . . . . . . . . . . . . . . . . . . 708 18.11.5 Generalized Random Graph Networks . . . . . . . . . . . . . . . . . . . . 709 18.12Small-world Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 709 18.13Scale-free Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 710 18.13.1 Master-equation approach . . . . . . . . . . . . . . . . . . . . . . . . . . 710 18.13.2 Rate-equation approach . . . . . . . . . . . . . . . . . . . . . . . . . . . 711 18.14Evolving Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 712 18.14.1 Extended Barabasi-Albert Model . . . . . . . . . . . . . . . . . . . . . . 713 18.15Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 714 18.16Exercise Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716 18.17Bibliographic Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716 xviii
  21. 21. Chapter 1 Introduction 1.1 Definition A distributed system is a collection of independent entities that cooperate to solve a problem that cannot be individually solved. Distributed systems have been in existence since the start of the universe. From a school of fish to a flock of birds and entire ecosystems of microorganisms, there is communication among mobile intelligent agents in nature. With the widespread proliferation of the internet and the emerging global village, the notion of distributed computing systems as a useful and widely deployed tool is becoming a reality. For computing systems, a distributed system has been characterized in one of several ways. • You know you are using one when the crash of a computer you have never heard of prevents you from doing work. (Lamport) • A collection of computers that do not share common memory or a common physical clock, and that communicate by message passing over a communication network; and each com- puter has its own memory and runs its own operating system. Typically the computers are semi-autonomous and are loosely coupled while they cooperate to address a problem collec- tively. (Singhal-Shivaratri [10]) • A collection of independent computers that appears to the users of the system as a single coherent computer. (Tanenbaum [11]) • A term that describes a wide range of computers, from weakly coupled systems such as wide-area networks to strongly coupled systems such as local area networks to very strongly- coupled systems such as multiprocessor systems. (Goscinski [6]) A distributed system can be characterized as a collection of mostly autonomous processors communicating over a communication network and having the following features. • No common physical clock. This is an important assumption because it introduces the ele- ment of “distribution” in the system and gives rise to the inherent asynchrony amongst the processors. 1
  22. 22. P P P P P PP M M M MM M M Communication network (WAN/ LAN) P processor(s) M memory bank(s) Figure 1.1: A distributed system that connects processors by a communication network. • No shared memory. This is a key feature that requires message-passing required for commu- nication. This feature implies the absence of the common physical clock. It may be noted that a distributed system may still provide the abstraction of a common ad- dress space via the distributed shared memory abstraction. Several aspects of shared memory multiprocessor systems have also been studied in the distributed computing literature. • Geographical separation. The geographically wider apart that the processors are, the more representative is the system of a distributed system. However, it is not necessary for the pro- cessors to be on a wide-area network (WAN). Recently, the Network/Cluster of Workstations (NOW/COW) configuration connecting processors on a LAN is also being increasingly re- garded as a small distributed system. This NOW configuration is becoming popular because of the low-cost high-speed off-the-shelf processors now available. The Google search engine is based on the NOW architecture. • Autonomy and heterogeneity. The processors are “loosely coupled” in that they have different speeds and each can be running a different operating system. They are usually not part of a dedicated system, but cooperate with one another by offering services or solving a problem jointly. 1.2 Relation to Computer System Components A typical distributed system is shown in Figure 1.1. Each computer has a memory-processing unit and the computers are connected by a communication network. Figure 1.2 shows the relation- ships of the software components that run on each of the computers and use the local operating system and network protocol stack for its functioning. The distributed software is also termed as middleware. A distributed execution is the execution of processes across the distributed system to collaboratively achieve a common goal. An execution is also sometimes termed a computation or a run. The distributed system uses a layered architecture to break down the complexity of system design. The middleware is the distributed software that drives the distributed system, while pro- 2
  23. 23. protocols distributed Extent ofDistributed application Network layer (middleware libraries) Application layer Data link layer Transport layer Networkprotocolstack Distributed software system Operating Figure 1.2: Interaction of the software components at each processor. viding transparency of heterogeneity at the platform level. Figure 1.2 schematically shows the interaction of this software with these system components at each processor. Here we assume that the middleware layer does not contain the traditional application layer functions of the network protocol stack, such as http, mail, ftp, and telnet. Various primitives and calls to functions defined in various libraries of the middleware layer are embedded in the user program code. There exist several libraries to choose from to invoke primitives for the more common functions – such as reli- able and ordered multicasting, of the middleware layer. There are several standards such as Object Management Group’s (OMG) Common Object Request Broker Architecture (CORBA), and the Remote Procedure Call (RPC) mechanism. The RPC mechanism conceptually works like a local procedure call, with the difference that the procedure code may reside on a remote machine, and the RPC software sends a message across the network to invoke the remote procedure. It then awaits a reply, after which the procedure call completes from the perspective of the program that invoked it. Currently deployed commercial versions of middleware often use CORBA, DCOM (Distributed Component Object Model), Java, and RMI (Remote Method Invocation) technolo- gies. The Message-Passing Interface (MPI) developed in the research community is an example of an interface for various communication functions. 1.3 Motivation The motivation for using a distributed system is some or all of the following requirements. 1. Inherently distributed computations. In many applications such as money transfer in bank- ing, or reaching consensus among parties that are geographically distant, the computation is inherently distributed. 2. Resource sharing. Resources such as peripherals, complete data sets in databases, special libraries, as well as data (variable/files) cannot be fully replicated at all the sites because it is often neither practical nor cost-effective. Further, they cannot be placed at a single site because access to that site might prove to be a bottleneck. Therefore, such resources 3
  24. 24. are typically distributed across the system. For example, distributed databases such as DB2 partition the data sets across several servers, in addition to replicating them at a few sites for rapid access as well as reliability. 3. Access to geographically remote data and resources. In many scenarios, the data cannot be replicated at every site participating in the distributed execution because it may be too large or too sensitive to be replicated. For example, payroll data within a multinational corporation is both too large and too sensitive to be replicated at every branch office/site. It is therefore stored at a central server which can be queried by branch offices. Similarly, special resources such as supercomputers exist only in certain locations, and to access such supercomputers, users need to log in remotely. Advances in the design of resource-constrained mobile devices as well as in wireless tech- nology using which these devices communicate have given further impetus to the importance of distributed protocols and middleware. 4. Enhanced reliability. A distributed system has the inherent potential to provide increased reliability because of the possibility of replicating resources and executions, as well as the reality that geographically distributed resources are not likely to crash/malfunction at the same time under normal circumstances. Reliability entails several aspects. • Availability, i.e., the resource should be accessible at all times. • Integrity, i.e., the value/state of the resource should be correct, in the face of concurrent access from multiple processors, as per the semantics expected by the application. • Fault-tolerance, i.e., the ability to recover from system failures, where such failures may be defined to occur in one of many failure models, which we will study in Chap- ter 14. 5. Increased Performance/Cost ratio. By resource sharing and accessing geographically remote data and resources, the Performance/Cost ratio is increased. Although higher throughput has not necessarily been the main objective behind using a distributed system, nevertheless, a task can be partitioned across the various computers in the distributed system. Such a configuration provides a better Performance/Cost ratio than using special parallel machines. This is particularly true of the NOW configuration. In addition to meeting the above requirements, a distributed system also offers the following ad- vantages. 6. Scalability. As the processors are usually connected by a wide-area network, adding more processors does not pose a direct bottleneck for the communication network. 7. Modularity and incremental expandability. Heterogeneous processors may be easily added into the system without affecting the performance, as long as those processors are running the same middleware algorithms. Similarly, existing processors may be easily replaced by other processors. 4
  25. 25. M MP P P PPP M M M memory P processor (b)(a) Interconnection networkInterconnection network PPP P MMMM M M Figure 1.3: Two standard architectures for parallel systems. (a) Uniform memory access (UMA) multiprocessor system. (b) Non-uniform memory access (NUMA) multiprocessor. In both archi- tectures, the processors may locally cache data from memory. 1.4 Relation to Parallel Multiprocessor/Multicomputer Systems The characteristics of a distributed system were identified above. A typical distributed system would look as shown in Figure 1.1. However, how does one classify a system that meets some but not all of the characteristics? Is the system still a distributed system, or does it become a parallel multiprocessor system? To better answer these questions, we first examine the architecture of par- allel systems, and then examine some well-known taxonomies for multiprocessor/multicomputer systems. 1.4.1 Characteristics of Parallel Systems A parallel system may be broadly classified as belonging to one of three types. 1. A multiprocessor system is a parallel system in which the multiple processors have direct access to shared memory which forms a common address space. The architecture is shown in Figure 1.3(a). Such processors usually do not have a common clock. A multiprocessor system usually corresponds to a uniform memory access (UMA) architec- ture in which the access latency, i.e., waiting time, to complete an access to any memory location from any processor is the same. The processors are in very close physical proximity and are usually very tightly coupled (homogenous hardware and software), are connected by an interconnection network. Interprocess communication across processors is tradition- ally through read and write operations on the shared memory, although the use of message- passing primitives such as those provided by the MPI, is also possible (using emulation on the shared memory). All the processors usually run the same operating system, and both the hardware and software are very tightly coupled. The processors are usually of the same type, and are housed within the same box/container with a shared memory among them. The interconnection network to access the memory may 5
  26. 26. be a bus, although for greater efficiency, it is usually a multistage switch with a symmetric and regular design. Figure 1.4 shows two popular interconnection networks – the Omega network and the But- terfly network, each of which is a multi-stage network formed of 2x2 switching elements. Each 2x2 switch allows data on either of the two input wires to be switched to the upper or the lower output wire. However, in a single step, only one data unit can be sent on an output wire. So if the data from both the input wires is to be routed to an output wire in a single step, there is a collsion. Various techniques such as buffering or more elaborate interconnection designs can address collisions. Each 2x2 switch is represented as a rectangle in the figure. Furthermore, a n-input and n- output network uses log n stages and log n bits for addressing. Routing in the 2x2 switch at stage k uses only the kth bit, and hence can be done at clock speed in hardware. The multi- stage networks can be constructed recursively, and the interconnection pattern between any two stages can be expressed using an iterative or a recursive generating function. Besides the Omega and Butterfly (banyan) networks, other examples of multistage interconnection networks are the Benes and the shuffle-exchange networks. Each of these has very inter- esting mathematical properties that allow rich connectivity between the processor bank and memory bank. Omega interconnection function. The Omega network which connects n processors to n memory units has n 2 · log2 n switching elements of size 2x2 arranged in log2 n stages. Between each pair of adjacent stages of the Omega network, a link exists between output i of a stage and the input j to the next stage according to the following perfect shuffle pattern which is a left-rotation operation on the binary representation of i to get j. The iterative generation function is as follows. j = 2i for 0 ≤ i ≤ n/2 − 1 2i + 1 − n for n/2 ≤ i ≤ n − 1 (1.1) Consider any stage of switches. Informally, the upper (lower) input lines for each switch come in sequential order from the upper (lower) half of the switches in the earlier stage. With respect to the Omega network in Figure 1.4(a), n = 8. Hence, for any stage, for the outputs i, where 0 ≤ i ≤ 3, the output i is connected to input 2i of the next stage. For 4 ≤ i ≤ 7, the output i of any stage is connected to input 2i + 1 − n of the next stage. Omega routing function. The routing function from input line i to output line j considers only j and the stage number s, where s ∈ [0, log2n − 1]. In a stage s switch, if the s + 1th MSB of j is 0, the data is routed to the upper output wire, otherwise it is routed to the lower output wire. Butterfly interconnection function. Unlike the Omega network, the generation of the in- terconnection pattern between a pair of adjacent stages depends not only on n but also on the stage number s. The recursive expression is as follows. Let there be M = n/2 switches per stage, and let a switch be denoted by the tuple x, s , where x ∈ [0, M − 1] and stage s ∈ [0, log2n − 1]. 6
  27. 27. The two outgoing edges from any switch x, s are as follows. There is an edge from switch x, s to switch y, s + 1 if (i) x = y or (ii) x XOR y has exactly one 1 bit, which is in the (s + 1)th MSB. For stage s, apply the rule above for M/2s switches. Whether the two incoming connections go to the upper or the lower input port is not impor- tant because of the routing function, given below. Example. Consider the Butterfly network in Figure 1.4(b), n = 8 and M = 4. There are three stages, s = 0, 1, 2, and the interconnection pattern is defined between s = 0 and s = 1 and between s = 1 and s = 2. The switch number x varies from 0 to 3 in each stage, i.e., x is a 2-bit string. (Note that unlike the Omega network formulation using input and output lines given above, this formulation uses switch numbers. Exercise 5 asks you to prove a formulation of the Omega interconnection pattern using switch numbers instead of input and output port numbers.) Consider the first stage interconnection (s = 0) of a butterfly of size M, and hence having log2 2M stages. For stage s = 0, as per rule (i), the first output line from switch 00 goes to input line of switch 00 of stage s = 1. As per rule (ii), the second output line of switch 00 goes to input line of switch 10 of stage s = 1. Similarly, x, s = (01) has one output line go to an input line of switch (11) in stage s = 1. The other connections in this stage can be determined similarly. For stage s = 1 connecting to stage s = 2, we apply the rules considering only M/21 = M/2 switches, i.e., builds 2 butterflies of size M/2 - the ”upper half" and the ”lower half" switches. The recursion terminates for M/2s = 1, when there is a single switch. Butterfly routing function. In a stage s switch, if the s + 1th MSB of j is 0, the data is routed to the upper output wire, otherwise it is routed to the lower output wire. Observe that for the Butterfly and the Omega networks, the paths from the different inputs to any one output form a spanning tree. This implies that collisions will occur when data is destined to the same output line. However, the advantage is that data can be combined at the switches if the application semantics (e.g., summation of numbers) are known. 2. A multicomputer parallel system is a parallel system in which the multiple processors do not have direct access to shared memory. The memory of the multiple processors may or may not form a common address space. Such computers usually do not have a common clock. The architecture is shown in Figure 1.3(b). The processors are in close physical proximity and are usually very tightly coupled (homoge- nous hardware and software), and connected by an interconnection network. The processors communicate either via a common address space or via message-passing. A multicomputer system that has a common address space usually corresponds to a non-uniform memory ac- cess (NUMA) architecture in which the latency to access various shared memory locations from the different processors varies. Examples of parallel multicomputers are: the NYU Ultracomputer and the Sequent shared memory machines, the CM* Connection machine and processors configured in regular and symmetrical topologies such as an array or mesh, ring, torus, cube, and hypercube (message- passing machines). The regular and symmetrical topologies have interesting mathematical 7
  28. 28. P0 P1 P2 P3 P4 P6 P7 101P5 000 001 M0 M1 010 011 100 101 110 111 001 101 110 111 100 111 110 100 011 010 000 M2010 000 001 100 101 P0 P1 P2 P3 P4 P5 P6 P7 (a) 3−stage Omega network (n=8, M=4) (b) (n=8, M=4)3−stage Butterfly network 011 M3 M4 M5 M6 M7 000 001 010 011 M0 M1 M2 M3 M4 M5 M6 M7 110 111 Figure 1.4: Interconnection networks for shared memory multiprocessor systems. (a) Omega net- work for n = 8 processors P0 − P7 and memory banks M0 − M7. (b) Butterfly network for n = 8 processors P0 − P7 and memory banks M0 − M7. properties that enable very easy routing and provide many rich features such as alternate routing. Figure 1.5(a) shows a wrap-around 4x4 mesh. For a k × k mesh which will contain k2 processors, the maximum path length between any two processors is 2(k/2 − 1). Routing can be done along the Manhattan grid. Figure 1.5(b) shows a 4-dimensional hypercube. A k-dimensional hypercube has 2k processor-and-memory units. Each such unit is a node in the hypercube, and has a unique k-bit label. Each of the k dimensions is associated with a bit position in the label. The labels of any two adjacent nodes are identical except for the bit position corresponding to the dimension in which the two nodes differ. Thus, the processors are labelled such that the shortest path between any two processors is the Hamming distance (defined as the number of bit positions in which the two equal sized bit strings differ) between the processor labels. This is clearly bounded by k. Example. Nodes 0101 and 1100 have a Hamming distance of 2. THe shortest path between them has length 2. Routing in the hypercube is done hop-by-hop. At any hop, the message can be sent along any dimension corresponding to the bit position in which the current node’s address and the des- tination address differ. The 4-D hypercube shown in the figure is formed by connecting the corresponding edges of two 3-D hypercubes (corresponding to the left and the right “cubes” in the figure) along the fourth dimension; the labels of the 4-D hypercube are formed by prepending a ‘0’ to the labels of the left 3-D hypercube and prepending a ‘1’ to the labels of the right 3-D hypercube. This can be extended to construct hypercubes of higher dimen- sions. Observe that there are multiple routes between any pair of nodes, which provides fault-tolerance as well as a congestion control mechanism. The hypercube and its variant topologies have very interesting mathematical properties with implications for routing and fault-tolerance. 3. Array processors belong to a class of parallel computers that are physically co-located, are 8
  29. 29. 0010 0111 0011 0101 (b)(a) processor + memory 1100 1000 1110 1010 1111 1011 1001 1101 0001 01100100 0000 Figure 1.5: Some popular topologies for multicomputer shared-memory machines. (a) Wrap- around 2D-mesh, also known as torus. (b) Hypercube of dimension 4. very tightly coupled, and have a common system clock (but may not share memory and communicate by passing data using messages). Array processors and systolic arrays that perform tightly synchronized processing and data exchange in lock-step for applications such as DSP and image processing belong to this category. These applications usually involve a large number of iterations on the data. This class of parallel systems has a very niche market. The distinction between UMA multiprocessors on the one hand, and NUMA and message- passing multicomputers on the other hand, is important because the algorithm design and data and task partitioning among the processors must account for the variable and unpredictable latencies in accessing memory/communication. As compared to UMA systems and array processors, NUMA and message-passing multicomputer systems are less suitable when the degree of granularity of accessing shared data and communication is very fine. The primary and most efficacious use of parallel systems is for obtaining a higher throughput by dividing the computational workload among the processors. The tasks that are most amenable to higher speedups on parallel systems are those that can be partitioned into subtasks very nicely, involving much number-crunching and relatively little communication for synchronization. Once the task has been decomposed, the processors perform large vector, array and matrix computations that are common in scientific applications. Searching through large state spaces can be performed with significant speedup on parallel machines. While such parallel machines were an object of much theoretical and systems research in the 1980s and early 1990s, they have not proved to be economically viable for two related reasons. First, the overall market for the applications that can potentially attain high speedups is relatively small. Second, due to economy of scale and the high processing power offered by relatively inexpensive off-the-shelf networked PCs, specialized parallel machines are not cost-effective to manufacture. They additionally require special compiler and other system support for maximum throughput. 9
  30. 30. 1.4.2 Flynn’s Taxonomy Flynn identified four processing modes, based on whether the processors execute the same or different instruction streams at the same time, and whether or not the processors processed the same (identical) data at the same time. It is instructive to examine this classification to understand the range of options used for configuring systems. Single Instruction stream, Single Data stream (SISD). This mode corresponds to the conven- tional processing in the von Neumann paradigm with a single CPU, and a single memory unit connected by a system bus. Single Instruction stream, Multiple Data stream (SIMD). This mode corresponds to the pro- cessing by multiple homogenous processors which execute in lock-step on different data items. Applications that involve operations on large arrays and matrices, such as scientific applications, can best exploit systems that provide the SIMD mode of operation because the data sets can be partitioned easily. Several of the earliest parallel computers, such as Illiac-IV, MPP, CM2, and MasPar MP-1 were SIMD machines. Vector processors, array processors and systolic arrays also belong to the SIMD class of processing. Recent SIMD architectures include co-processing units such as the MMX units in Intel processors (e.g., Pentium with the Streaming SIMD Extensions (SSE) options) and DSP chips such as the Sharc. Multiple Instruction stream, Single Data stream (MISD). This mode corresponds to the exe- cution of different operations in parallel on the same data. This is a specialized mode of operation with limited but niche applications, e.g., visualization. Multiple Instruction stream, Multiple Data stream (MIMD). In this mode, the various proces- sors execute different code on different data. This is the mode of operation in distributed systems as well as in the vast majority of parallel systems. There is no common clock among the system processors. Sun Ultra servers, multicomputer PCs, and IBM SP machines are example machines that execute in MIMD mode. SIMD, MISD, MIMD architectures are illustrated in Figure 1.6. MIMD architectures are most general and allow much flexibility in partitioning code and data to be processed, among the proces- sors. MIMD architectures also include the classically understood mode of execution in distributed systems. 1.4.3 Coupling, Parallelism, Concurrency, and Granularity 1.4.3.1 Coupling. The degree of coupling among a set of modules, whether hardware or software, is measured in terms of the interdependency and binding and/or homogeneity among the modules. When the degree of coupling is high (low), the modules are said to be tightly (loosely) coupled. SIMD and MISD architectures generally tend to be tightly coupled because of the common clocking of the shared instruction stream or the shared data stream. Here we briefly examine various MIMD architectures in terms of coupling. 10
  31. 31. PP CCCC I I II II III D P (c) MISD(b) MIMD(a) SIMD data stream Processing Unit Control Unit P D instruction streamI CC P DD PP DD Figure 1.6: Flynn’s taxonomy of SIMD, MIMD, and MISD architectures for multiproces- sor/multicomputer systems. 1. Tightly-coupled multiprocessors (with UMA shared memory). These may be either switch- based (e.g., NYU Ultracomputer, and RP3) or bus-based (e.g., Sequent, Encore). 2. Tightly-coupled multiprocessors (with NUMA shared memory or that communicate by mes- sage passing). Examples are the SGI Origin 2000 and the Sun Ultra HPC servers (that com- municate via NUMA shared memory), and the hypercube and the torus (that communicate by message passing). 3. Loosely-coupled multicomputers (without shared memory) physically co-located. These may be bus-based (e.g., NOW connected by a LAN or Myrinet card) or using a more gen- eral communication network, and the processors may be heterogenous. In such systems, processors neither share memory nor have a common clock, and hence may be classified as distributed systems – however, the processors are very close to one another, which is char- acteristic of a parallel system. As the communication latency may be significantly lower than in wide-area distributed systems, the solution approaches to various problems may be different for such systems than for wide-area distributed systems. 4. Loosely-coupled multicomputers (without shared memory and without common clock) that are physically remote. These correspond to the conventional notion of distributed systems. 1.4.3.2 Parallelism or speedup of a program on a specific system. This is a measure of the relative speedup of a specific program, on a given machine. The speedup depends on the number of processors and the mapping of the code to the processors. It is expressed as the ratio of the time T(n) with n processors, to the time T(1) with a single processor. 1.4.3.3 Parallelism within a parallel/distributed program. This is an aggregate measure of the percentage of time that all the processors are executing CPU instructions productively, as opposed to waiting for communication (either via shared memory or message-passing) operations to complete. The term is traditionally used to characterize parallel 11
  32. 32. programs. If the aggregate measure is a function of only the code, then the parallelism is indepen- dent of the architecture. Otherwise, this definition degenerates to the definition of parallelism in Section 1.4.3.2. 1.4.3.4 Concurrency of a program. This is a broader term that means roughly the same as parallelism of a program, but is used in the context of distributed programs. The parallelism/concurrency in a parallel/distributed program can be measured by the ratio of the number of local (non-communication and non-shared memory ac- cess) operations to the total number of operations, including the communication or shared memory access operations. 1.4.3.5 Granularity of a program. The relative measure of the amount of computation to the amount of communication within the parallel/distributed program is termed as granularity. If the degree of parallelism is coarse-grained (fine-grained), there are relatively many more (fewer) productive CPU instruction executions, com- pared to the number of times the processors communicate (either via shared memory or message- passing) and wait to get synchronized with the other processors. Programs with fine-grained par- allelism are best suited for tightly coupled systems. These typically include SIMD and MISD architectures, tightly coupled MIMD multiprocessors (that have shared memory), and loosely- coupled multicomputers (without shared memory) that are physically colocated. If programs with fine-grained parallelism were run over loosely-coupled multiprocessors that are physically remote, the latency delays for the frequent communication over the WAN would significantly degrade the overall throughput. As a corollary, it follows that on such loosely-coupled multicomputers, pro- grams with a coarse-grained communication/message-passing granularity will incur substantially less overhead. Figure 1.2 showed the relationships between the local operating system, the middleware imple- menting the distributed software, and the network protocol stack. Before moving on, we identify various classes of multiprocessor/multicomputer operating systems. • The operating system running on loosely coupled processors (i.e., heterogenous and/or geo- graphically distant processors), which are themselves running loosely coupled software (i.e., software that is heterogenous) is classified as a Network Operating System. In this case, the application cannot run any significant distributed function that is not provided by the Application Layer of the network protocol stacks on the various processors. • The operating system running on loosely coupled processors, which are running tightly cou- pled software (i.e., the middleware software on the processors is homogenous) is classified as a Distributed Operating System. • The operating system running on tightly coupled processors, which are themselves running tightly coupled software is classified as a Multiprocessor Operating System. Such a parallel system can run sophisticated algorithms contained in the tightly coupled software. 12
  33. 33. 1.5 Message Passing Systems versus Shared Memory Systems Shared memory systems are those in which there is a (common) shared address space throughout the system. Communication among processors takes place via shared data variables, and control variables for synchronization among the processors. Semaphores and monitors that were origi- nally designed for shared memory uniprocessors and multiprocessors are examples of how syn- chronization can be achieved in shared memory systems. All multicomputer (NUMA as well as message-passing) systems that do not have a shared address space provided by the underlying architecture and hardware necessarily communicate by message passing. Conceptually, program- mers find it easier to program using shared memory than by message passing. For this and several other reasons that we examine later, the abstraction called shared memory is sometimes provided to simulate a shared address space. For a distributed system, this abstraction is called distributed shared memory. Implementing this abstraction has a certain cost but it simplifies the task of the application programmer. There also exists a well-known folklore result that communication via message-passing can be simulated by communication via shared memory and vice-versa. There- fore, the two paradigms are equivalent. 1.5.1 Emulating message-passing on shared memory system (MP → SM). The shared address space can be partitioned into disjoint parts, one part being assigned to each processor. “Send” and “Receive” operations can be implemented by writing to and reading from the destination/sender processor’s address space, respectively. Specifically, a separate location can be reserved as the mailbox for each ordered pair of processes. A Pi–Pj message-passing can be emulated by a Write by Pi to the mailbox and then a Read by Pj from the mailbox. In the simplest case, these mailboxes can be assumed to have unbounded size. The write and read operations need to be controlled using synchronization primitives to inform the receiver/sender after the data has been sent/received. 1.5.2 Emulating shared memory on a message-passing system (SM → MP). This involves the use of “Send” and “Receive” operations for “Write” and “Read” operations. Each shared location can be modeled as a separate process; “Write” to a shared location is emulated by sending an update message to the corresponding owner process; a “Read” to a shared location is emulated by sending a query message to the owner process. As accessing another processor’s memory requires Send and Receive operations, this emulation is expensive. Although emulating shared memory might seem to be more attractive from a programmer’s perspective, it must be remembered that in a distributed system, it is only an abstraction. Thus, the latencies involved in read and write operations may be high even when using shared memory emulation because the read and write operations are implemented by using network-wide communication under the covers. An application can of course use a combination of shared memory and message-passing. In a MIMD message-passing multicomputer system, each “processor” may be a tightly-coupled mul- tiprocessor system with shared memory. Within the multiprocessor system, the processors com- municate via shared memory. Between two computers, the communication is by message passing. As message-passing systems are more common and more suited for wide-area distributed systems, 13
  34. 34. we will consider message-passing systems more extensively than we consider shared memory sys- tems. 1.6 Primitives for Distributed Communication 1.6.1 Blocking/Nonblocking, Synchronous/Asynchronous Primitives Message send and message receive communication primitives are denoted Send() and Receive(), respectively. A Send primitive has at least two parameters - the destination, and the buffer in the user space, containing the data to be sent. Similarly, a Receive primitive has at least two parameters - the source from which the data is to be received (this could be a wildcard), and the user buffer into which the data is to be received. There are two ways of sending data when the Send primitive is invoked - the buffered option and the unbuffered option. The buffered option which is the standard option copies the data from the user buffer to the kernel buffer. The data later gets copied from the kernel buffer onto the network. In the unbuffered option, the data gets copied directly from the user buffer onto the network. For the Receive primitive, the buffered option is usually required because the data may already have arrived when the primitive is invoked, and needs a storage place in the kernel. The following are some definitions of blocking/nonblocking and synchronous/asynchronous primitives. Synchronous primitives. A Send or a Receive primitive is synchronous if both the Send() and Receive() handshake with each other. The processing for the Send primitive completes only after the invoking processor learns that the other corresponding Receive primitive has also been invoked and that the receive operation has been completed. The processing for the Receive primitive completes when the data to be received is copied into the receiver’s user buffer. Asynchronous primitives. A Send primitive is said to be asynchronous if control returns back to the invoking process after the data item to be sent has been copied out of the user-specified buffer. It does not make sense to define asynchronous Receive primitives. Blocking primitives. A primitive is blocking if control returns to the invoking process after the processing for the primitive (whether in synchronous or asynchronous mode) completes. Nonblocking primitives. A primitive is nonblocking if control returns back to the invoking pro- cess immediately after invocation, even though the operation has not completed. For a non- blocking Send, control returns to the process even before the data is copied out of the user buffer. For a nonblocking Receive, control returns to the process even before the data may have arrived from the sender. For nonblocking primitives, a return parameter on the primitive call returns a system-generated handle which can be later used to check the status of completion of the call. The process can check for the completion of the call in two ways. First, it can keep checking (in a loop or periodically) if the handle has been flagged or posted. Second, it can issue a Wait with a list 14
  35. 35. Send(X, destination, handlek) //handlek is a return parameter ... ... Wait(handle1, handle2, . . ., handlek, . . . , handlem) //Wait always blocks Figure 1.7: A nonblocking send primitive. When the Wait call returns, at least one of its parameters is posted. of handles as parameters. The Wait call usually blocks until one of the parameter handles is posted. Presumably after issuing the primitive in nonblocking mode, the process has done whatever actions it could and now needs to know the status of completion of the call, there- fore using a blocking Wait() call is usual programming practice. The code for a nonblocking Send would look as shown in Figure 1.7. If at the time that Wait() is issued, the processing for the primitive (whether synchronous or asynchronous) has completed, the Wait returns immediately. The completion of the pro- cessing of the primitive is detectable by checking the value of handlek. If the processing of the primitive has not completed, the Wait blocks and waits for a signal to wake it up. When the processing for the primitive completes, the communication subsystem software sets the value of handlek and wakes up (signals) any process with a Wait call blocked on this handlek. This is called posting the completion of the operation. There are therefore four versions of the Send primitive – synchronous blocking, synchronous nonblocking, asynchronous blocking, and asynchronous nonblocking. For the Receive primitive, there are the blocking synchronous and nonblocking synchronous versions. These versions of the primitives are illustrated in Figure 1.8 using a timing diagram. Here, three time lines are shown for each process: (1) for the process execution, (2) for the user buffer from/to which data is sent/received, and (3) for the kernel/communication subsystem. Blocking synchronous Send (Figure 1.8(a)): The data gets copied from the user buffer to the kernel buffer and is then sent over the network. After the data is copied to the receiver’s system buffer, an acknowledgement back to the sender causes control to return to the process that invoked the Send operation and completes the Send. Nonblocking synchronous Send (Figure 1.8(b)): Control returns back to the invoking process as soon as the copy of data from the user buffer to the kernel buffer is initiated. A parameter in the nonblocking call also gets set with the handle of a location that the user process can later check for the completion of the synchronous send operation. The location gets posted after an acknowledgement returns from the receiver, as per the semantics described for (a). The user process can keep checking for the completion of the nonblocking synchronous Send by testing the returned handle, or it can invoke the blocking Wait operation on the returned handle. Blocking asynchronous Send (Figure 1.8(c)): The user process that invokes the Send is blocked until the data is copied from the user’s buffer to the kernel buffer. (For the unbuffered option, the user process that invokes the Send is blocked until the data is copied from the user’s buffer to the network.) 15

×