Marcirio Silveira Chaves [email_address] Tutorial on Parallel Computing and  Message Passing Model Part II - Day 4 <ul><li...
Introduction <ul><li>We saw how to  send and receive messages  where  all   of the data  were of a  single type  intrinsic...
Introduction <ul><li>Data can be contiguous in memory, but that is not generally the way we will use the word &quot;contig...
Objectives <ul><li>Overview about strategies and features in MPI that enable you to transmit collections of data of  mixed...
Sending Multiple Messages <ul><li>The simplest approach for  communicating mixed datatypes and noncontiguous   data  is: <...
Sending Multiple Messages <ul><li>E.g. </li></ul><ul><ul><li>In Fortran, arrays are stored so that  successive elements of...
Sending Multiple Messages <ul><li>Fortran send might look like this: </li></ul><ul><li>do j = 1, m  </li></ul><ul><ul><li>...
Sending Multiple Messages <ul><li>do j = 1,  m   </li></ul><ul><ul><li>call  MPI_Send (a( k , l +j-1),  n , MPI_DOUBLE, de...
Sending Multiple Messages <ul><li>do j = 1, m  </li></ul><ul><ul><li>call  MPI_Send (a(k,l+j-1), n, MPI_DOUBLE, dest, tag,...
Buffering <ul><li>When the data you need to communicate is not contiguous, one approach might be to  copy them into a cont...
Buffering <ul><li>Buffering the Right Way -  Pack and Unpack </li></ul><ul><ul><li>Use the  MPI_PACK  routine to fill a bu...
Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><ul><li>With MPI_PACK, the Fortran submatrix exa...
Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><li>The  icount  variable, is initially set to  ...
Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><li>On the receiving side, you can similarly spe...
Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><ul><li>Because of translation, data may occupy ...
Buffering <ul><li>Use of MPI_PACK and MPI_UNPACK provides  great flexibility .  </li></ul><ul><li>In addition to allowing ...
Derived Datatypes <ul><li>A more efficient approach to handling  mixed datatypes  and  noncontiguous data  than sending mu...
Derived Datatypes <ul><li>Various MPI routines that can be used to create derived datatypes.  </li></ul><ul><li>For our ex...
Derived Datatypes <ul><li>In defining the derived datatype, we need to specify the  stride ,  </li></ul><ul><ul><li>Stride...
Derived Datatypes <ul><li>call MPI_TYPE_VECTOR( m ,  n ,  nn ,  MPI_DOUBLE ,  my_mpi_type , ierr)  </li></ul><ul><li>call ...
Derived Datatypes <ul><li>Sometimes  derived datatypes  are used to construct other derived datatypes, so the intermediate...
Derived Datatypes <ul><li>The derived datatype … </li></ul><ul><ul><li>only needs to be created one time ,  </li></ul></ul...
Constructing Derived Datatypes <ul><li>MPI_TYPE_VECTOR: for constructing derived datatypes.  </li></ul><ul><li>The most ge...
Constructing Derived Datatypes <ul><li>MPI_TYPE_CONTIGUOUS :  describe a contiguous sequence of values in memory.  </li></...
Constructing Derived Datatypes June - 2009 LNEC-DHA-NTI
Constructing Derived Datatypes <ul><li>MPI_TYPE_INDEXED  describes sequences that may vary both in length and in spacing. ...
Obtaining Information About Your Derived Types <ul><li>Once you have defined a derived datatype, there are several utility...
   Nano-Self Test - Derived Datatypes   http://ci-tutor.ncsa.uiuc.edu/content.php?cid=1309 Questions 1-3.
Collective Communications
Collective Communication <ul><li>Collective communication must involve all processes in the scope of a communicator.  </li...
Collective Communication <ul><li>Synchronization  - processes wait until all members of the group have reached the synchro...
Collective Communication <ul><li>Programming Considerations and Restrictions </li></ul><ul><ul><li>Collective operations a...
Collective Communication <ul><li>Broadcast operation  </li></ul><ul><ul><li>a single process sends a copy of some data to ...
Collective Communication <ul><li>Scatter and Gather Operations </li></ul><ul><ul><li>distribute data on one processor acro...
Collective Communication <ul><li>Reduction operation </li></ul><ul><ul><li>A  single process  (the root process) collects ...
Collective Communication Routines <ul><li>MPI_Barrier </li></ul><ul><ul><li>Creates a barrier synchronization in a group. ...
Collective Communication Routines <ul><li>MPI_Bcast </li></ul><ul><ul><li>Broadcasts (sends) a message from the process wi...
Collective Communication Routines <ul><li>MPI_Scatter </li></ul><ul><ul><li>Distributes distinct messages  from a single s...
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
Collective Communication Routines June - 2009 LNEC-DHA-NTI
June - 2009 LNEC-DHA-NTI
Fortran - Collective Communications Example <ul><li>Sample program output:  </li></ul><ul><li>rank= 0 Results: 1.000000 2....
<ul><li>Any questions, so far? </li></ul>June - 2009 LNEC-DHA-NTI
   Self Test - Collective Communications   http://ci-tutor.ncsa.uiuc.edu/content.php?cid=1310
Upcoming SlideShare
Loading in …5
×

Tutorial on Parallel Computing and Message Passing Model - C4

1,809 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,809
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • You will have to learn other techniques for the heavily executed portions.
  • Tutorial on Parallel Computing and Message Passing Model - C4

    1. 1. Marcirio Silveira Chaves [email_address] Tutorial on Parallel Computing and Message Passing Model Part II - Day 4 <ul><li>Communicating Non-contiguous Data or Mixed Datatypes </li></ul><ul><li>Collective Communications </li></ul>
    2. 2. Introduction <ul><li>We saw how to send and receive messages where all of the data were of a single type intrinsic to MPI and were contiguous in an array . </li></ul><ul><li>While your data may be that well behaved, it is likely that you will need to transmit collections of data of mixed types or data that are scattered within an array . </li></ul>June - 2009 LNEC-DHA-NTI
    3. 3. Introduction <ul><li>Data can be contiguous in memory, but that is not generally the way we will use the word &quot;contiguous&quot;. </li></ul><ul><li>Rather, we refer to the contiguity of elements within arrays. </li></ul><ul><ul><li>E.g. in Fortran: columns are contiguous, i.e., x(5,7) is &quot;next to&quot; x(6,7). </li></ul></ul><ul><ul><li>Although data in memory is oftentimes arranged this same way, it is not always the case. </li></ul></ul><ul><ul><li>Contiguous data refers to its arrangement in arrays, not memory . </li></ul></ul>June - 2009 LNEC-DHA-NTI
    4. 4. Objectives <ul><li>Overview about strategies and features in MPI that enable you to transmit collections of data of mixed types or data that are scattered within an array: </li></ul><ul><ul><li>Sending Multiple Messages </li></ul></ul><ul><ul><li>Buffering </li></ul></ul><ul><ul><li>Pack and Unpack </li></ul></ul><ul><ul><li>Derived Datatypes </li></ul></ul>June - 2009 LNEC-DHA-NTI
    5. 5. Sending Multiple Messages <ul><li>The simplest approach for communicating mixed datatypes and noncontiguous data is: </li></ul><ul><ul><li>to identify the largest pieces of your data that individually meet the requirements of being of homogeneous intrinsic type and contiguous , and send each of those pieces as a separate message . </li></ul></ul><ul><li>E.g. </li></ul><ul><li>You have a large matrix stored in a two-dimensional array , and you want to send a rectangular submatrix to another processor . </li></ul>June - 2009 LNEC-DHA-NTI
    6. 6. Sending Multiple Messages <ul><li>E.g. </li></ul><ul><ul><li>In Fortran, arrays are stored so that successive elements of a column are contiguous , with the last element in one column followed by the first element of the next column. </li></ul></ul><ul><ul><li>Since you are not sending full columns of the original matrix, the columns of your submatrix will not be contiguous . </li></ul></ul><ul><ul><li>However, the elements within a column will be contiguous , so you can send one message for each column of the submatrix . </li></ul></ul><ul><ul><li>Let's look at an array that is declared to be NN by MM , </li></ul></ul><ul><ul><ul><li>where you want to send the N by M portion whose upper left corner is at position (K,L) . </li></ul></ul></ul>June - 2009 LNEC-DHA-NTI
    7. 7. Sending Multiple Messages <ul><li>Fortran send might look like this: </li></ul><ul><li>do j = 1, m </li></ul><ul><ul><li>call MPI_Send (a(k,l+j-1), n, MPI_DOUBLE, dest, tag, MPI_COMM_WORLD, ierr) </li></ul></ul><ul><li>enddo </li></ul>June - 2009 LNEC-DHA-NTI
    8. 8. Sending Multiple Messages <ul><li>do j = 1, m </li></ul><ul><ul><li>call MPI_Send (a( k , l +j-1), n , MPI_DOUBLE, dest, tag, MPI_COMM_WORLD, ierr) </li></ul></ul><ul><li>enddo </li></ul><ul><li>If the receiving processor does not know the values of N , M , K , or L , they can be sent in a separate message. </li></ul><ul><li>The principal advantage of this approach is that you do not need to learn anything new to apply it. </li></ul><ul><li>The principal disadvantage is its overhead. </li></ul><ul><ul><li>A fixed overhead is associated with the sending and receiving of a message, however long or short it is. </li></ul></ul><ul><ul><li>If you replace one long message with multiple short messages, you slow down your program by greatly increasing your overhead. </li></ul></ul><ul><ul><li>This method can be error-prone. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    9. 9. Sending Multiple Messages <ul><li>do j = 1, m </li></ul><ul><ul><li>call MPI_Send (a(k,l+j-1), n, MPI_DOUBLE, dest, tag, MPI_COMM_WORLD, ierr) </li></ul></ul><ul><li>enddo </li></ul><ul><li>If you are working in a portion of your program that will be executed rarely and the number of additional messages is small, the total amount of added overhead may be small enough to ignore. </li></ul><ul><ul><li>However, for most programs there is no advantage in limiting yourself to this strategy. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    10. 10. Buffering <ul><li>When the data you need to communicate is not contiguous, one approach might be to copy them into a contiguous buffer . </li></ul><ul><li>For our submatrix example, the Fortran code for buffering the data might look like this: </li></ul><ul><li>Copying to a buffer eliminates the excessive number of messages generated by the multiple messages approach at the cost of extra memory for the buffer and extra CPU time to perform the copy into the buffer. </li></ul><ul><li>Limitation: it still handles only one type of data at a time. </li></ul>June - 2009 LNEC-DHA-NTI icount = 0 do j = l, l+m-1 do i = k, k+n-1 icount = icount + 1 p(icount) = a(i,j) enddo enddo call MPI_Send(p, n*m, MPI_DOUBLE, dest, tag, MPI_COMM_WORLD, ierr)
    11. 11. Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><ul><li>Use the MPI_PACK routine to fill a buffer. </li></ul></ul><ul><ul><li>You call MPI_PACK with arguments that describe the buffer you are filling and most of the arguments you would have provided when using MPI_SEND . </li></ul></ul><ul><ul><li>MPI_PACK copies your data into the buffer and, if necessary, translates it into a standard intermediate representation. </li></ul></ul><ul><ul><li>After all the data you want to send have been placed in the buffer by MPI_PACK, you can send the buffer (giving its type as MPI_PACKED) and no further translations will be performed. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    12. 12. Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><ul><li>With MPI_PACK, the Fortran submatrix example becomes: </li></ul></ul>June - 2009 LNEC-DHA-NTI icount = 0 do i = 1, m call MPI_PACK (a(k,l+i-1), n, MPI_DOUBLE, buffer , bufsize, & icount, MPI_COMM_WORLD, ierr) enddo call MPI_SEND( buffer , icount, MPI_PACKED , dest, tag, MPI_COMM_WORLD, ierr)
    13. 13. Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><li>The icount variable, is initially set to zero to indicate that you are starting the construction of a new message and have an empty buffer . </li></ul><ul><li>The successive calls to MPI_PACK update the count variable to reflect the data that have been added to the buffer and the final value is used in the call to MPI_Send to specify the amount of data to send. </li></ul>June - 2009 LNEC-DHA-NTI icount = 0 do i = 1, m call MPI_PACK (a(k,l+i-1), n, MPI_DOUBLE, buffer, bufsize, icount , MPI_COMM_WORLD, ierr) enddo call MPI_SEND(buffer, icount , MPI_PACKED, dest, tag, MPI_COMM_WORLD, ierr)
    14. 14. Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><li>On the receiving side, you can similarly specify type MPI_PACKED to receive a buffer without translation and then use MPI_UNPACK (which bears the same resemblance to MPI_RECV as MPI_PACK bears to MPI_SEND) to translate and copy the data from the buffer to where you really want it. </li></ul>June - 2009 LNEC-DHA-NTI icount = 0 do i = 1, m call MPI_PACK (a(k,l+i-1), n, MPI_DOUBLE, buffer, bufsize, icount, MPI_COMM_WORLD, ierr) enddo call MPI_SEND(buffer, icount, MPI_PACKED, dest, tag, MPI_COMM_WORLD, ierr)
    15. 15. Buffering <ul><li>Buffering the Right Way - Pack and Unpack </li></ul><ul><ul><li>Because of translation, data may occupy a different amount of space in the buffer than it does natively. </li></ul></ul><ul><ul><li>You can make your buffer large enough by using the routine MPI_PACK_SIZE to calculate how much buffer space is required for the different types of data you plan to place in your buffer. </li></ul></ul><ul><ul><li>Nothing in the content of a message indicates it was or was not built with MPI_PACK. </li></ul></ul><ul><ul><ul><li>If, as in our example, all the data packed into the buffer is of a single type, the message could be received in a buffer of that type rather than receiving it as MPI_PACKED and using MPI_UNPACK to decode it. </li></ul></ul></ul><ul><ul><ul><li>Conversely, a message that was sent as an ordinary intrinsic type could be received as MPI_PACKED and distributed using calls to MPI_UNPACK. </li></ul></ul></ul>June - 2009 LNEC-DHA-NTI
    16. 16. Buffering <ul><li>Use of MPI_PACK and MPI_UNPACK provides great flexibility . </li></ul><ul><li>In addition to allowing messages that include arbitrary mixtures of datatypes its incremental construction and interpretation of messages allows the values of data early in a message to affect the type, size, or destination of data appearing later in the same message. </li></ul><ul><li>Costs of this flexibility </li></ul><ul><ul><li>the memory used for the buffers </li></ul></ul><ul><ul><li>CPU time used in copying data to and from those buffers. </li></ul></ul><ul><li>If constructing a message requires a large number of calls to MPI_PACK (or interpreting a message requires a large number of calls to MPI_UNPACK), the added procedure call overhead may also be significant. </li></ul>June - 2009 LNEC-DHA-NTI <ul><li>Buffering the Right Way - Pack and Unpack </li></ul>
    17. 17. Derived Datatypes <ul><li>A more efficient approach to handling mixed datatypes and noncontiguous data than sending multiple messages and buffering (packing and unpacking). </li></ul><ul><li>Derived datatypes: </li></ul><ul><li>datatypes that are built from the basic MPI datatypes. </li></ul><ul><li>A way to get MPI to do packing and unpacking &quot;on-the-fly&quot; as part of the send and receive operations. </li></ul><ul><li>The packing and unpacking can then be done directly to, and from, its internal communications buffers, generally making your program more efficient by eliminating the need for: </li></ul><ul><ul><li>The explicit intermediate buffer used when you do the packing and unpacking </li></ul></ul><ul><ul><li>The copying between the intermediate buffer and the communications buffer </li></ul></ul>June - 2009 LNEC-DHA-NTI
    18. 18. Derived Datatypes <ul><li>Various MPI routines that can be used to create derived datatypes. </li></ul><ul><li>For our example, we will use MPI_TYPE_VECTOR , which creates a datatype consisting of uniformly spaced blocks of data , each block having the same size . </li></ul><ul><li>In our Fortran example, we need to send segments of M columns of the matrix, each containing N values. </li></ul>June - 2009 LNEC-DHA-NTI
    19. 19. Derived Datatypes <ul><li>In defining the derived datatype, we need to specify the stride , </li></ul><ul><ul><li>Stride: the distance between the starting locations of adjacent blocks of data. </li></ul></ul><ul><li>The columns of the full matrix each have NN values, so NN will be the stride between the beginning of one column segment and an adjacent column segment. </li></ul><ul><li>The Fortran routine calls are: </li></ul><ul><ul><li>call MPI_TYPE_VECTOR (m, n, nn, MPI_DOUBLE, my_mpi_type, ierr) </li></ul></ul><ul><ul><li>call MPI_TYPE_COMMIT (my_mpi_type, ierr) </li></ul></ul><ul><ul><li>call MPI_SEND (a(k,l), 1, my_mpi_type, dest, tag, MPI_COMM_WORLD, ierr) </li></ul></ul>June - 2009 LNEC-DHA-NTI
    20. 20. Derived Datatypes <ul><li>call MPI_TYPE_VECTOR( m , n , nn , MPI_DOUBLE , my_mpi_type , ierr) </li></ul><ul><li>call MPI_TYPE_COMMIT ( my_mpi_type , ierr) </li></ul><ul><li>call MPI_SEND(a(k,l), 1, my_mpi_type , dest, tag, MPI_COMM_WORLD, ierr) </li></ul><ul><li>This datatype defines the whole submatrix. </li></ul><ul><li>The call to MPI_TYPE_COMMIT makes the derived datatype available for use in communication routines. </li></ul>June - 2009 LNEC-DHA-NTI <ul><li>MPI_TYPE_VECTOR: we are creating a new derived datatype called my_mpi_type . </li></ul><ul><li>It contains m blocks of data, each containing n MPI_DOUBLE values, with a stride of nn between each block. </li></ul>
    21. 21. Derived Datatypes <ul><li>Sometimes derived datatypes are used to construct other derived datatypes, so the intermediate types do not have to be committed. </li></ul><ul><li>Finally, the MPI_SEND routine sends the data. </li></ul><ul><li>The send is quite simple, since we are sending a single instance of a my_mpi_type , starting at location (k,l), containing the whole submatrix. </li></ul><ul><li>The data are received in an analogous way, receiving a single instance of the derived datatype my_mpi_type , resulting in the values arranged properly in the submatrix. </li></ul>June - 2009 LNEC-DHA-NTI call MPI_TYPE_VECTOR( m , n , nn , MPI_DOUBLE , my_mpi_type , ierr) call MPI_TYPE_COMMIT ( my_mpi_type , ierr) call MPI_SEND (a(k,l), 1, my_mpi_type , dest, tag, MPI_COMM_WORLD, ierr)
    22. 22. Derived Datatypes <ul><li>The derived datatype … </li></ul><ul><ul><li>only needs to be created one time , </li></ul></ul><ul><ul><li>can be used any number of times in communication routines. </li></ul></ul><ul><ul><ul><li>If our submatrix must be transferred additional times throughout the code, only a single call to MPI_SEND will be required each time . </li></ul></ul></ul>June - 2009 LNEC-DHA-NTI
    23. 23. Constructing Derived Datatypes <ul><li>MPI_TYPE_VECTOR: for constructing derived datatypes. </li></ul><ul><li>The most general way to construct a derived datatype is through the use of MPI_TYPE_STRUCT, which allows the length , location , and type of each component to be specified independently. </li></ul><ul><ul><li>There are several other procedures available to describe common patterns of access, primarily within arrays. These are: </li></ul></ul><ul><ul><ul><li>MPI_TYPE_CONTIGUOUS </li></ul></ul></ul><ul><ul><ul><li>MPI_TYPE_HVECTOR </li></ul></ul></ul><ul><ul><ul><li>MPI_TYPE_INDEXED </li></ul></ul></ul><ul><ul><ul><li>MPI_TYPE_HINDEXED </li></ul></ul></ul>June - 2009 LNEC-DHA-NTI
    24. 24. Constructing Derived Datatypes <ul><li>MPI_TYPE_CONTIGUOUS : describe a contiguous sequence of values in memory. </li></ul><ul><li>Fortran </li></ul><ul><ul><li>call mpi_type_contiguous(2,mpi_double, mpi_2d_point ,ierr) </li></ul></ul><ul><ul><li>call mpi_type_contiguous(3,mpi_double, mpi_3d_point ,ierr) </li></ul></ul><ul><ul><li>Creates new type indicators mpi_2d_point and mpi_3d_point . </li></ul></ul><ul><ul><li>These type indicators allow you to treat consecutive pairs of doubles as point coordinates in a 2-dimensional space and sequences of three doubles as point coordinates in a 3-dimensional space . </li></ul></ul>June - 2009 LNEC-DHA-NTI
    25. 25. Constructing Derived Datatypes June - 2009 LNEC-DHA-NTI
    26. 26. Constructing Derived Datatypes <ul><li>MPI_TYPE_INDEXED describes sequences that may vary both in length and in spacing. </li></ul><ul><ul><li>Because the location of these sequences is measured in elements rather than bytes, it is most appropriate for identifying arbitrary parts of a single array. </li></ul></ul><ul><li>MPI_TYPE_HINDEXED is similar to MPI_TYPE_INDEXED except that the locations are specified in bytes rather than elements. </li></ul><ul><ul><li>It allows the identification of arbitrary parts of arbitrary arrays, subject only to the requirement that they all have the same type. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    27. 27. Obtaining Information About Your Derived Types <ul><li>Once you have defined a derived datatype, there are several utility procedures you can use to obtain information about it. </li></ul><ul><ul><li>MPI_TYPE_LB and MPI_TYPE_UB: provide the lower and upper bounds of the datatype. </li></ul></ul><ul><ul><li>MPI_TYPE_EXTENT: provides the extent of the datatype. </li></ul></ul><ul><ul><ul><li>In most cases, this is the amount of memory a value of the datatype will occupy. </li></ul></ul></ul><ul><ul><li>MPI_TYPE_SIZE: provides the size of the datatype in a message. </li></ul></ul><ul><ul><ul><li>If the datatype is scattered in memory, this may be significantly smaller than the extent of the datatype. </li></ul></ul></ul>June - 2009 LNEC-DHA-NTI
    28. 28.  Nano-Self Test - Derived Datatypes  http://ci-tutor.ncsa.uiuc.edu/content.php?cid=1309 Questions 1-3.
    29. 29. Collective Communications
    30. 30. Collective Communication <ul><li>Collective communication must involve all processes in the scope of a communicator. </li></ul><ul><ul><li>All processes are by default, members in the communicator MPI_COMM_WORLD . </li></ul></ul><ul><li>It is the programmer's responsibility to insure that all processes within a communicator participate in any collective operations. </li></ul>June - 2009 LNEC-DHA-NTI
    31. 31. Collective Communication <ul><li>Synchronization - processes wait until all members of the group have reached the synchronization point. </li></ul><ul><li>Data Movement - broadcast, scatter/gather, all to all. </li></ul><ul><li>Collective Computation (reductions) - one member of the group collects data from the other members and performs an operation (min, max, add, multiply, etc.) on that data. </li></ul>June - 2009 LNEC-DHA-NTI
    32. 32. Collective Communication <ul><li>Programming Considerations and Restrictions </li></ul><ul><ul><li>Collective operations are blocking. </li></ul></ul><ul><ul><li>Collective communication routines do not take message tag arguments. </li></ul></ul><ul><ul><li>Collective operations within subsets of processes are accomplished by first partitioning the subsets into new groups and then attaching the new groups to new communicators. </li></ul></ul><ul><ul><li>Can only be used with MPI predefined datatypes - not with MPI Derived Data Types. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    33. 33. Collective Communication <ul><li>Broadcast operation </li></ul><ul><ul><li>a single process sends a copy of some data to all the other processes in a group. </li></ul></ul><ul><ul><li>Each row represents a different process. </li></ul></ul><ul><ul><li>Each colored block in a column represents the location of a piece of the data. </li></ul></ul><ul><ul><li>Blocks with the same color that are located on multiple processes contain copies of the same data. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    34. 34. Collective Communication <ul><li>Scatter and Gather Operations </li></ul><ul><ul><li>distribute data on one processor across a group of processors or vice versa </li></ul></ul><ul><ul><li>Scatter operation: all of the data (an array of some type) are initially collected on a single processor. </li></ul></ul><ul><ul><li>After the scatter operation, pieces of the data are distributed on different processors. </li></ul></ul><ul><ul><li>The multicolored box reflects the possibility that the data may not be evenly divisible across the processors. </li></ul></ul><ul><ul><li>The gather operation is the inverse operation to scatter: it collects pieces of the data that are distributed across a group of processors and reassembles them in the proper order on a single processor. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    35. 35. Collective Communication <ul><li>Reduction operation </li></ul><ul><ul><li>A single process (the root process) collects data from the other processes in a group and performs an operation on that data, which produces a single value. </li></ul></ul><ul><ul><li>Use : to compute the sum of the elements of an array that is distributed across several processors. </li></ul></ul><ul><ul><li>Operations: arithmetic, maximum and minimum and various logical and bitwise operations. </li></ul></ul>June - 2009 LNEC-DHA-NTI
    36. 36. Collective Communication Routines <ul><li>MPI_Barrier </li></ul><ul><ul><li>Creates a barrier synchronization in a group. </li></ul></ul><ul><ul><li>Each task, when reaching the MPI_Barrier call, blocks until all tasks in the group reach the same MPI_Barrier call. </li></ul></ul><ul><ul><li>MPI_BARRIER (comm,ierr) </li></ul></ul>June - 2009 LNEC-DHA-NTI
    37. 37. Collective Communication Routines <ul><li>MPI_Bcast </li></ul><ul><ul><li>Broadcasts (sends) a message from the process with rank &quot;root&quot; to all other processes in the group. </li></ul></ul><ul><ul><li>MPI_BCAST (buffer,count,datatype,root,comm,ierr) </li></ul></ul>June - 2009 LNEC-DHA-NTI
    38. 38. Collective Communication Routines <ul><li>MPI_Scatter </li></ul><ul><ul><li>Distributes distinct messages from a single source task to each task in the group. </li></ul></ul><ul><ul><li>one-to-all communication . </li></ul></ul><ul><ul><li>MPI_SCATTER (sendbuf,sendcnt,sendtype,recvbuf, </li></ul></ul><ul><ul><li>...... recvcnt,recvtype,root,comm,ierr) </li></ul></ul>June - 2009 LNEC-DHA-NTI
    39. 39. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    40. 40. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    41. 41. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    42. 42. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    43. 43. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    44. 44. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    45. 45. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    46. 46. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    47. 47. Collective Communication Routines June - 2009 LNEC-DHA-NTI
    48. 48. June - 2009 LNEC-DHA-NTI
    49. 49. Fortran - Collective Communications Example <ul><li>Sample program output: </li></ul><ul><li>rank= 0 Results: 1.000000 2.000000 3.000000 4.000000 </li></ul><ul><li>rank= 1 Results: 5.000000 6.000000 7.000000 8.000000 </li></ul><ul><li>rank= 2 Results: 9.000000 10.000000 11.000000 12.000000 </li></ul><ul><li>rank= 3 Results: 13.000000 14.000000 15.000000 16.000000 </li></ul>June - 2009 LNEC-DHA-NTI
    50. 50. <ul><li>Any questions, so far? </li></ul>June - 2009 LNEC-DHA-NTI
    51. 51.  Self Test - Collective Communications  http://ci-tutor.ncsa.uiuc.edu/content.php?cid=1310

    ×