Communication between any two nodes has to be carried through some switches.

1. Indirect (dynamic)
Networks
1

2. Indirect (dynamic) Networks
• Communication between any two nodes has to be carried through
some switches.
• Classified into:
• Crossbar network
• Multistage interconnection networks (MIN)
• Nonblocking networks
• Rearrangeable nonblocking networks
• Blocking networks
• Single-stage interconnection networks
2

3. Crossbar (nonblocking, 1-hop)
3
An N x M crossbar
The cost increases as N x M.
Switch point
(crosspoint)
M outputs
N
inputs
N:1 multiplexer

4. A large crossbar
4
0
1
2
3

5. MINs
5
Three-stage MIN
firststage
secondstage
thirdstage
N
inputs
M
outputs

6. Nonblocking MIN
• Any input port can be connected to any free output port without
affecting the existing connections.
• Same functionality as a crossbar.
• The Clos network is proposed to reduce the cost of crossbar by
decreasing the number of switches (1953).
6

7. History (telephone net.)
• Early telephone networks were built from elector-mechanical
crossbars.
• Key developments in telephone switching include the Clos network
in 1953 and the Benes network in 1962.
• Many large telephone switches today are still built from Clos or
Clos-like networks.
7

8. 3-stage Clos network (m,n,r)
8
Nonblocking: m >= 2n-1
Rearrangeable: 2n-1 > m >= n
N
inputs
N
outputs
1
2
r
1
n
1
n
1
n
1
2
m
1
2
r
1
n
1
n
1
n
n×m r×r m×n

9. Rearrangeable nonblocking MIN
• Any input port can be connected to any free output port with
rearrangement of paths for the existing connections.
• Less stages or smaller switches than a nonblocking networks.
• The best known example is the Benes network.
9

10. Benes network
10
An 8x8 Benes network (Eσ ３
－１
E σ ２
－ 1
Eσ ２ Eσ ３ E)
N(2log2N - 1) / 2 switches, 2log2N - 1 hops, E: stage
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
σ ３
－１
σ ２
－１
σ ２ σ ３
010
001
(2,2,2) clos
(2,2,2) clos
(2,4,2) Clos

11. 2x2 switch (radix: 2)
11
straight exchange lower
broadcast
upper
broadcast

12. Permutation
•Permutations are traffic patterns in which all traffic
from each source is directed to one destination.
•It can be compactly represented by a permutation
function that maps source to destination.
•Bit (digit) permutation are those in which each bit
(digit) of the destination address is a function of one
bit (digit) of the source address.
• di = sj or dx = f(sx)
12

13. Permutation functions (1/2)
Communication from node address x(an,an-1,an-2,…, a1) to
• Perfect shuffle: σ(x)=(an-1,an-2,…, a1,an)
• k sub-shuffule: σk(x)=(an,an-1,…, ak+1,ak-1, … , a1,ak)
• k supershuffule: σk
(x)=(an-1,an-2,…, an-k+1, an, an-k, … , a1)
• Inverse perfect shuffle: σ-1
(x)=(a1,an,an-1,an-2,…, a2)
• k sub-inverse shuffule: σk
-1
(x)=(an,…, ak+1, a1,ak,…, a2)
• Butterfly: β(x)=(a1,an-1,an-2,…, a2,an)
• k sub-butterfly: βk (x)=(an,…, ak+1, a1,ak-1, … , a2,ak)
• k superbutterfly: βk
(x)=(an-k+1, an-1,…, an-k+2, an,an-k, … , a1)
13

14. Permutation functions (2/2)
Communication from node address x(an,an-1,an-2,…, a1) to
• Bit reversal: ρ(x)=(a1,a2,…, an-1, an)
• k sub-bit reversal : ρk(x)=(an,an-1,…, ak+1,a1, a2,… , ak-1,ak)
• k super-bit reversal: ρk
(x)=(an-k+1,an-k+2,…, an-1, an, an-k, … , a1)
• shift: α(x)= | x+1| (2n
), where modulo by 2n
• k sub-shift: αk(x)= | x+1| (2n
) +floor(x/2k
)2k
• k super-shift: αk
(x)= | x+ 2n-k
| (2n
)
• Exchange: Ei(x)=(an,an-1, …, ~ai,…, a1)
14

15. Perfect shuffle (1/2)
15
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
Perfect shuffle
σ(x)=(an-1,an-2,…, a1,an)
Inverse perfect shuffle
σ-1
(x)=(a1,an an-1,an-2,…, a2)

16. Perfect shuffle (2/2)
16
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
3 sub-inverse shuffule
σ3
-1
(x)=(a1 ,a3 , a2)
2 sub-inverse shuffle
σ2
-1
(x)=(a3, a1, a2)

17. Butterfly
17
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
butterfly (bit reversal)
β3(x)=(a1 ,a2 , a3)
＝ ρ ３ (x)
2 sub-butterfly
β2(x)=(a3, a1, a2)

18. shift
18
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
shift
α(x)= | x+1 | (23
)
2 sub-shift
α 2(x)=| x+1 | (22
)+floor(x/ 22
)22

19. exchange
19
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
E3(x) = (~a3, a2, a1)
E2(x) = (a3, ~a2, a1)

20. Blocking MIN
• A free input/output pair is not always possible because of conflicts
with the existing connections.
• Minimizing the number of switches and stages.
• Unidirectional MIN (Cenju-3, IBM PR3)
• Bidirectional MIN (IBM SP, TMC CM5)
20

21. Unidirectional MINs
•Channels and switches are unidirectional.
•At least ceiling(logkN) stages are required for a MIN
of N input and output ports with k×k switches.
•Every path through the MIN crosses all the stages
(same length).
•Examples: baseline, butterfly, indirect cube
logN stages for N inputs/outputs with 2x2 switches
( binary logN-fly in the form of k-ary n-fly)
the number of communication patterns are (√N)N
21

22. Baseline networks (1/2)
(B = Eσn
－１
Eσn-1
－１
E ・・・ Eσ ２
－ 1
E)
22
000
001
010
011
100
101
110
111
000
001 (UUL)
010 (ULU)
011
100
101
110
111
abc cab cba
σ ２
－ 1
σ ３
－ 1
001

23. Baseline networks (2/2)
• To route (an,an-1,an-2,…, a1) to (bn,bn-1,bn-2,…, b1),
select upper link when bi=0, otherwise select lower link on (n-i+1) stage (MSB
to LSB).
(path is determined only by the destination address)
23
switch
upper link
lower link
inputs

24. Omega networks (1/2) Ω= (σE)n
24
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
σσσ
abc bca abccab

25. Omega networks (2/2)
• To route (an,an-1,an-2,…, a1) to (bn,bn-1,bn-2,…, b1),
set the switch as straight when ai^bi=0, otherwise set it as exchange on (n-i+1)
stage (MSB to LSB).
25
straight exchange

26. Indirect binary n-cube (1/2)
(C = Eβ ２ Eβ ３ ・・・ E β ｎ Eσ － 1
)
26
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
σ-1
β ３β ２
000
001
100
101
010
110
011
111
abc acb abcbca

27. Indirect binary n-cube (2/2)
• To route (an,an-1,an-2,…, a1) to (bn,bn-1,bn-2,…, b1),
set the switch as straight when ai^bi=0, otherwise set it as exchange on i-th
stage (LSB to MSB).
• A k-ary n-fly network consists of kn
source/destination terminals, n
stages of kn-1
k×k crossbar switches.
• Degree of the switches: 2k
• Diameter: n+1 or logkN +1 where N = kn
27

28. Relation among blocking
networks
• For omega, indirect n-cube and Banyan(Y):
Ω-1
=C=Yσ
So they are almost functionally equal (isomorphic).
• The communication patterns are different, but same number of
combinations ((√N)N
).
• Smaller number of communication patterns than nonblocking
networks (N!).
28

29. Bitonic sort
29
8 inputs sorter4 inputs sorter2 inputs sorter
A L
B H

30. Batcher-banyan network
30
Bitonic
Sorter
(Batcher
stage)
Ω
(banyan
stage)
Non-
blocking
5
∞
3
1
∞
･
･
∞
1
3
5
∞
∞
･
･
∞
1
3
5
receiver Ordered addresses

31. Tree saturation (e.g. on Omega)
31
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
Combining: combine two messages
hotspot

32. Fault-tolerant MINs
• Blocking MINs provide a single path between any input/output pair.
• If a link becomes congested or fails, the unique path property can
easily disrupt the communication (no path diversity).
• A solution is to provide multiple routing paths by adding extra
stages or channels.
32

33. Fault-tolerant MINs (Omega’s case)
33
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
σσσAdditional
stage

34. Fault-tolerant MINs (Baseline’s case)
34
000
001
010
011
100
101
110
111
000
001
010
011
100
101
110
111
σ ２
－ 1
σ ３
－ 1

35. Bidirectional MINs (BMINs)
•Channels and switches are bidirectional.
•Information can be transmitted simultaneously in
opposite directions between neighboring switches.
•Paths are established in BMINs by crossing stages in
forward, then turnaround, and finally backward.
•Several paths are possible in forward direction, but
a single path is available in backward direction.
35

36. Bidirectional Banyan networks
36
000
001
010
011
100
101
110
111
β ３β ２
000
001
100
101
010
110
011
111
000
001
100
101
010
110
011
111
forward
backward
turnaround

37. A fat tree BMIN
37
forward
backward

38. Hybrid networks
• Combine mechanisms from shared-medium networks and direct or
indirect networks.
• Increase bandwidth and reduce the distance between nodes.
• Examples:
• Multiple backplane buses
• Hierarchical buses
• Cluster-based networks
38

39. Multiple backplane buses
39
device device device
bus
bus
bus

40. Hierarchical buses
40
Global bus
Cluster bus
bridge

41. Cluster-based networks
41
Cluster bus
ClusterCluster Cluster
ClusterCluster Cluster
router

42. History (Clos networks, etc.)
•Clos introduced non-blocking networks in1953.
•Benes discovered rearrangeable non-blocking
networks in 1965.
•Bachter introduced the bitonic sort and the
Batcher sorting network in 1968.
•Knuth’s book in 1998 treats sorting networks in
detail.
•Multicast in Clos networks was first considered by
Masson and Jordan(1972), and has since been
studied by many researchers.
42

43. History (indirect networks)
• Processor-memory networks emerged in the late 1960s
in the form of indirect netwoks.
• The smallest machine employed crossbar, whereas
larger sytems used butterfly, etc..
• Since the early 1990s, a variant of the Clos and Benes
networks of telephony has emerged in SMP (MPP)
networks in the form of the fat tree.
• Today, the high pin bandwidth of router chips relative
to message length motivates the use of networks of
higher node degree, such as Clos and fat free.
43