Introduction to Parallel      Computing         Part Ib
Processor IntercommunicationIn part 1b we will look at the interconnectionnetwork between processors. Using theseconnectio...
Processor Topologies (1)There are several ways in which processorscan be interconnected. The most importantinclude    • Bu...
Topology IssuesBefore looking at some of the major processortopologies we have to know what makes acertain topology well o...
TerminologyCommunication is one-to-one when oneprocessor sends a message to another one.In one-to-all or broadcast, one pr...
Connection PropertiesThere are three elements that are used inBuilding the interconnection network. Theseare the wire, a r...
Bus Topology (1)      P2        P4        P…P1         P3        P5
Bus Topology (2)Hardware cost   1One-to-one      1One-to-all      1All-to-all      pProblems        Bus becomes bottleneck
Star Topology (1)           P3      P2        P4 P1        P0        P5      P8        P6           P7
Star Topology (2)Hardware cost   p–1One-to-one      1 or 2One-to-all      p–1All-to-all      2 · (p – 1)Problems        Ce...
Tree Topology (1)P1   P2   P3   P4   P5   P6   P7   P8
Tree Topology (2)Hardware cost   2p – 2, when p power of 2One-to-one      2 · 2log pOne-to-all      (2log p) · (1 + 2log p...
Tree Topology – One-to-all P1   P2   P3   P4   P5   P6   P7   P8
Fully Connected Topology (1)            P2   P3       P1             P4            P6   P5
Fully Connected Topology (2)Hardware cost p·(p – 1)/2One-to-one    1One-to-all     2log pAll-to-all    2·2log pProblem...
Ring Topology (1)      P2   P3 P1             P4      P6   P5
Ring Topology (2)Hardware cost   pOne-to-one      p / 2One-to-all      p / 2All-to-all      2·p / 2Problems        P...
2D-Mesh Topology (1) P12   P13   P14   P15 P8    P9    P10   P11 P4    P5    P6    P7 P0    P1    P2    P3
2D-Mesh Topology (2)Hardware cost   2·(√p)·(√p – 1)One-to-one      2(√p – 1)One-to-all      2(√p – 1)All-to-all      2·√p...
Mesh – All-to-allStep 0     6     5     4     3     2     1         P1     P2   P3      P4      P5   1 23 45 1 23 45 1 23 ...
2D Wrap-around Mesh (1)   P12   P13   P14   P15   P8    P9    P10   P11   P4    P5    P6    P7   P0    P1    P2    P3
2D Wrap-around Mesh (2)Hardware cost   2pOne-to-one      2·√p / 2One-to-all      2· √p / 2All-to-all      4· √p / 2R...
2D Wrap-around – One-to-all     P12   P13   P14   P15     P8    P9    P10   P11     P4    P5    P6    P7     P0    P1    P...
Hypercube Topology (1)        1D                2D   3D                         4D
Hypercube Construction   4D   3D   2D   1DHypercube
Hypercube Topology (2)Hardware cost     (p / 2) · 2log pOne-to-one      2                 log pOne-to-all      2          ...
4D Hypercube – One-to-all
Communication IssuesThere are some things left to be said aboutcommunication :• In general the time required for data tran...
Asynchronous vs. SynchronousThe major difference between asynchronousand synchronous communication is that thefirst method...
Example asynchronous comm.    Processor A      Processor B  Instruction A1  InstructionA1    Instruction B1               ...
Asynchronous Comm.• Convenient because processors do not have to  wait for each other.• However, we often need to know whe...
Example synchronous comm.   Processor A      Processor B Instruction A1 InstructionA1    Instruction B1                  I...
Synchronous comm.• Both send and receive are blocking• Processors have to wait for each other. This  reduces efficiency.• ...
Deadlocks (1)A deadlock is a situation where two or moreprocessors are waiting to for each otherinfinitely.
Deadlocks (2)         Processor A                Processor B      Send to B                  Send to A      Receive from B...
Deadlocks (3)  Processor A      Processor BSend to B        Receive from AReceive from B   Send to A
Deadlocks (4)               P2      P3        P1                   P4               P6      P5Pattern: P1 → P2 → P5 → P4 →...
End of Part IAre there any questions regarding part I
Upcoming SlideShare
Loading in...5
×

Parallel computing(1)

539

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
539
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Parallel computing(1)

  1. 1. Introduction to Parallel Computing Part Ib
  2. 2. Processor IntercommunicationIn part 1b we will look at the interconnectionnetwork between processors. Using theseconnections, various communication patternscan be used to transport information fromone or more source processors, to one or moredestination processors.
  3. 3. Processor Topologies (1)There are several ways in which processorscan be interconnected. The most importantinclude • Bus • Ring • Star • Mesh • Tree • Wraparound mesh • Fully connected • Hypercube
  4. 4. Topology IssuesBefore looking at some of the major processortopologies we have to know what makes acertain topology well or ill suited to connectprocessors.There are two aspects of topologies thatshould be looked at. These are scalability andcost (communication and hardware).
  5. 5. TerminologyCommunication is one-to-one when oneprocessor sends a message to another one.In one-to-all or broadcast, one processor sendsa message to all other processors. In all-to-onecommunication, all processors send theirmessage to one processor. Other forms ofcommunication include gather and all-to-all.
  6. 6. Connection PropertiesThere are three elements that are used inBuilding the interconnection network. Theseare the wire, a relay node, and a processornode. The latter two elements can at a giventime send or receive only one message, nomatter how many wires enter or leave theelement.
  7. 7. Bus Topology (1) P2 P4 P…P1 P3 P5
  8. 8. Bus Topology (2)Hardware cost 1One-to-one 1One-to-all 1All-to-all pProblems Bus becomes bottleneck
  9. 9. Star Topology (1) P3 P2 P4 P1 P0 P5 P8 P6 P7
  10. 10. Star Topology (2)Hardware cost p–1One-to-one 1 or 2One-to-all p–1All-to-all 2 · (p – 1)Problems Central processor becomes bottleneck.
  11. 11. Tree Topology (1)P1 P2 P3 P4 P5 P6 P7 P8
  12. 12. Tree Topology (2)Hardware cost 2p – 2, when p power of 2One-to-one 2 · 2log pOne-to-all (2log p) · (1 + 2log p)All-to-all 2 · (2log p) · (1 + 2log p)Problems Top node becomes bottleneck This can be solved by adding more wires at the top (fat tree)
  13. 13. Tree Topology – One-to-all P1 P2 P3 P4 P5 P6 P7 P8
  14. 14. Fully Connected Topology (1) P2 P3 P1 P4 P6 P5
  15. 15. Fully Connected Topology (2)Hardware cost p·(p – 1)/2One-to-one 1One-to-all 2log pAll-to-all 2·2log pProblems Hardware cost increases quadratically with respect to p
  16. 16. Ring Topology (1) P2 P3 P1 P4 P6 P5
  17. 17. Ring Topology (2)Hardware cost pOne-to-one p / 2One-to-all p / 2All-to-all 2·p / 2Problems Processors are loaded with transport jobs. But hardware and communication cost low
  18. 18. 2D-Mesh Topology (1) P12 P13 P14 P15 P8 P9 P10 P11 P4 P5 P6 P7 P0 P1 P2 P3
  19. 19. 2D-Mesh Topology (2)Hardware cost 2·(√p)·(√p – 1)One-to-one 2(√p – 1)One-to-all 2(√p – 1)All-to-all 2·√p / 2Remarks Scalable both in network and communication cost. Can be found in many architectures.
  20. 20. Mesh – All-to-allStep 0 6 5 4 3 2 1 P1 P2 P3 P4 P5 1 23 45 1 23 45 1 23 45 1 23 45 1 23 45
  21. 21. 2D Wrap-around Mesh (1) P12 P13 P14 P15 P8 P9 P10 P11 P4 P5 P6 P7 P0 P1 P2 P3
  22. 22. 2D Wrap-around Mesh (2)Hardware cost 2pOne-to-one 2·√p / 2One-to-all 2· √p / 2All-to-all 4· √p / 2Remarks Scalable both in network and communication cost. Can be found in many architectures.
  23. 23. 2D Wrap-around – One-to-all P12 P13 P14 P15 P8 P9 P10 P11 P4 P5 P6 P7 P0 P1 P2 P3
  24. 24. Hypercube Topology (1) 1D 2D 3D 4D
  25. 25. Hypercube Construction 4D 3D 2D 1DHypercube
  26. 26. Hypercube Topology (2)Hardware cost (p / 2) · 2log pOne-to-one 2 log pOne-to-all 2 log pAll-to-all 2 · 2log pRemarks The most elegant design, also when it comes down to routing algorithms. But difficult to build in hardware.
  27. 27. 4D Hypercube – One-to-all
  28. 28. Communication IssuesThere are some things left to be said aboutcommunication :• In general the time required for data transmission is of the form startup-time + transfer speed * package size. So it is more efficient to sent one large package instead of many small packages (startup-time can be high, e.g. think about internet)• Asynchronous vs. Synchronous transfer and deadlocks.
  29. 29. Asynchronous vs. SynchronousThe major difference between asynchronousand synchronous communication is that thefirst methods sends a message and continues,while the second sends a message and waitsfor the receiver program to receive themessage.
  30. 30. Example asynchronous comm. Processor A Processor B Instruction A1 InstructionA1 Instruction B1 InstructionB1 Send to B Send to B Instruction B2 Instruction B2 Instruction A2 Instruction A2 Instruction B3 Instruction B3 Instruction A3 Instruction A3 Instruction B4 Instruction B4 Instruction A4 Instruction A4 Receive from A from A Instruction A5 Instruction A5 Instruction B5 Instruction B5
  31. 31. Asynchronous Comm.• Convenient because processors do not have to wait for each other.• However, we often need to know whether or not the destination processors has received the data, this often requires some checking code later in the program.• Need to know whether the OS supports reliable communication layers.• Receive instruction may or may not be blocking.
  32. 32. Example synchronous comm. Processor A Processor B Instruction A1 InstructionA1 Instruction B1 InstructionB1 Send to B Send to B Instruction B2 Instruction B2 Instruction A2 Instruction A2 Instruction B3 Instruction B3 Instruction A3 Instruction B4 Instruction B4 Instruction A4 Receive from A from A Instruction A5 Instruction B5 Instruction B5
  33. 33. Synchronous comm.• Both send and receive are blocking• Processors have to wait for each other. This reduces efficiency.• Implicitly offers a synchronisation point.• Easy to program because fewer unexpected situations can arise.• Problem: Deadlocks may occur.
  34. 34. Deadlocks (1)A deadlock is a situation where two or moreprocessors are waiting to for each otherinfinitely.
  35. 35. Deadlocks (2) Processor A Processor B Send to B Send to A Receive from B Receive from ANote: Only occurs with synchronous communication.
  36. 36. Deadlocks (3) Processor A Processor BSend to B Receive from AReceive from B Send to A
  37. 37. Deadlocks (4) P2 P3 P1 P4 P6 P5Pattern: P1 → P2 → P5 → P4 → P6 → P3 → P1
  38. 38. End of Part IAre there any questions regarding part I
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×