This document discusses analytical modeling of parallel systems. It provides an outline for a lecture on sources of overhead in parallel programs, performance metrics, scalability, and asymptotic analysis of parallel programs. Examples are given of one-to-all broadcast on rings, meshes, and hypercubes. The different phases of broadcast and reduction operations on these network topologies are illustrated. Finally, common MPI names for parallel operations are mentioned.

1. Chapter 5Chapter 5
Analytical Modeling of ParallelAnalytical Modeling of Parallel
SystemsSystems
Dr. Muhammad Hanif Durad
Department of Computer and Information Sciences
Pakistan Institute Engineering and Applied Sciences
hanif@pieas.edu.pk
Some slides have bee adapted with thanks from some other
lectures available on Internet

2. Dr. Hanif Durad 2
Lecture Outline
 Sources of Overhead in Parallel Programs
 Performance Metrics for Parallel Systems
 Effect of Granularity on Performance
 Scalability of Parallel Systems
 Minimum Execution Time and Minimum Cost-Optimal
Execution Time
 Asymptotic Analysis of Parallel Programs
 Other Scalability Metrics

3. Dr. Hanif Durad 3

4. One-to-All Broadcast on Rings
One-to-all broadcast on an eight-node ring. Node 0 is the
source of the broadcast. Each message transfer step is shown
by a numbered, dotted arrow from the source of the message
to its destination. The number on an arrow indicates the time
step during which the message is transferred.
(3/3)

5. Broadcast and Reduction on a
Mesh: Example
One-to-all broadcast on a 16-node
mesh.
(2/2)

6. 4.1.3 Broadcast and Reduction on
a Hypercube (1/2)
 d phases on d-dimension hypercube
 phase i:
 2(i-1)
messages
 sent in parallel
 in dimension i
 source node = 000, d = 3
 Phase 1: 000 -> 100
 Phase 2: 000 -> 010 100 -> 110
 Phase 3: 000 -> 001 100 -> 101
010 -> 011 110 -> 111

7. MPI Names of various Operations
Summary (2/3)