Distributed Memory Programming with MPI

1
Distributed Memory
Programming with MPI
Slides extended from
An Introduction to Parallel Programming by Peter
Pacheco
Dilum Bandara
Dilum.Bandara@uom.lk

2
Distributed Memory Systems
 We discuss about developing programs for these
systems using MPI
 MPI – Message Passing Interface
 Are set of libraries that can be called from C, C++, &
Fortran
Copyright © 2010, Elsevier Inc. All rights Reserved

3
Why MPI?
 Standardized & portable message-passing
system
 One of the oldest libraries
 Wide-spread adoption
 Minimal requirements on underlying hardware
 Explicit parallelization
 Achieves high performance
 Scales to a large no of processors
 Intellectually demanding

4
Our First MPI Program

5
Compilation
mpicc -g -Wall -o mpi_hello mpi_hello.c
wrapper script to compile
turns on all warnings
source file
create this executable file name
(as opposed to default a.out)
produce
debugging
information

6
Execution
mpiexec -n <no of processes> <executable>
mpiexec -n 1 ./mpi_hello
run with 1 process
run with 4 processes

7
Execution
Greetings from process 0 of 1 !

8
MPI Programs
 Need to add mpi.h header file
 Identifiers defined by MPI start with “MPI_”
 1st letter following underscore is uppercase
 For function names & MPI-defined types
 Helps to avoid confusion

9
6 Golden MPI Functions

10
MPI Components
 MPI_Init
 Tells MPI to do all necessary setup
 e.g., allocate storage for message buffers, decide rank of a
process
 argc_p & argv_p are pointers to argc & argv
arguments in main( )
 Function returns error codes

11
 MPI_Finalize
 Tells MPI we’re done, so clean up anything allocated
for this program
MPI Components (Cont.)

12
Communicators
 Collection of processes that can send messages
to each other
 Messages from others communicators are ignored
 MPI_Init defines a communicator that consists of
all processes created when the program is
started
 Called MPI_COMM_WORLD

13
Communicators (Cont.)
My rank
(process making this call)
No of processes in the communicator

14
Single-Program Multiple-Data (SPMD)
 We compile 1 program
 Process 0 does something different
 Receives messages & prints them while the
other processes do the work
 if-else construct makes our program
SPMD
 We can run this program on any no of
processors
 e.g., 4, 8, 32, 1000, …

15
Communication
 msg_buf_p, msg_size, msg_type
 Determines content of message
 dest – destination processor’s rank
 tag – use to distinguish messages that are identical in
content

16
Data Types

17
Communication (Cont.)
MPI_ANY_SOURCE to receive messages (from any
source) in order for arrival

18
Message Matching
MPI_Send
src = q
MPI_Recv
dest = r
r
q

19
Receiving Messages
 Receiver can get a message without
knowing
 Amount of data in message
 Sender of message
 Tag of message
 How can those be found out?

20
How Much Data am I Receiving?

21
 Exact behavior is determined by MPI
implementation
 MPI_Send may behave differently with regard to
buffer size, cutoffs, & blocking
 Cutoff
 if message size < cutoff  buffer
 if message size ≥ cutoff  MPI_Send will block
 MPI_Recv always blocks until a matching
message is received
 Preserve message ordering from a sender
 Know your implementation
 Don’t make assumptions!
Issues With Send & Receive

22
Trapezoidal Rule

23
Trapezoidal Rule (Cont.)

24
Serial Pseudo-code

25
Parallel Pseudo-Code

26
Tasks & Communications for
Trapezoidal Rule

27
First Version

28
First Version (Cont.)

29
First Version (Cont.)

30
COLLECTIVE
COMMUNICATION

31
Collective Communication
A tree-structured global sum

32
Alternative Tree-Structured Global Sum
Which is most optimum?
Can we do better?

33
MPI_Reduce

34
Predefined Reduction Operators

35
Collective vs. Point-to-Point Communications
 All processes in the communicator must call the
same collective function
 e.g., a program that attempts to match a call to
MPI_Reduce on 1 process with a call to MPI_Recv on
another process is erroneous
 Program will hang or crash
 Arguments passed by each process to an MPI
collective communication must be “compatible”
 e.g., if 1 process passes in 0 as dest_process &
another passes in 1, then the outcome of a call to
MPI_Reduce is erroneous
 Program is likely to hang or crash

36
Collective vs. P-to-P Communications (Cont.)
 output_data_p argument is only used on
dest_process
 However, all of the processes still need to pass in an
actual argument corresponding to output_data_p,
even if it’s just NULL
 Point-to-point communications are matched on
the basis of tags & communicators
 Collective communications don’t use tags
 Matched solely on the basis of communicator & order
in which they’re called

37
MPI_Allreduce
 Useful when all processes need result of a global
sum to complete some larger computation

38
Global sum followed
by distribution of result
MPI_Allreduce (Cont.)

39
Butterfly-Structured Global Sum
Processes exchange partial results

40
Broadcast
 Data belonging to a single process is sent to all
of the processes in communicator

41
Tree-Structured Broadcast

42
Data Distributions – Compute a Vector Sum
Serial implementation

43
Partitioning Options
 Block partitioning
 Assign blocks of consecutive components to each process
 Cyclic partitioning
 Assign components in a round robin fashion
 Block-cyclic partitioning
 Use a cyclic distribution of blocks of components

44
Parallel Implementation

45
MPI_Scatter
 Can be used in a function that reads in an entire
vector on process 0 but only sends needed
components to other processes

46
Reading & Distributing a Vector

47
MPI_Gather
 Collect all components of a vector onto process 0
 Then process 0 can process all of components

48
MPI_Allgather
 Concatenates contents of each process’
send_buf_p & stores this in each process’
recv_buf_p
 recv_count is the amount of data being received from
each process

49
Summary
Source: https://computing.llnl.gov/tutorials/mpi/

Distributed Memory Programming with MPI

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Distributed Memory Programming with MPI

Similar to Distributed Memory Programming with MPI (20)

More from Dilum Bandara

More from Dilum Bandara (20)

Recently uploaded

Recently uploaded (20)

Distributed Memory Programming with MPI

Editor's Notes