MPI_MPROBE eliminates race conditions between probing and receiving messages. It is useful for event-based and multi-threaded MPI applications where probing is used to determine message properties like size before receiving. MPI_MPROBE guarantees the corresponding receive will retrieve the probed message, even if there are delays, by removing the message from the queue after a successful probe. This protects against races where another process could receive the message first.

1. MPI_MPROBE: It’s Good for You Jeff Squyres

2. Regular MPI_PROBE Checks to see if a matching message has arrived Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 Tag: 9 Source: 67 Comm. ID: 17

3. Regular MPI_PROBE Checks to see if a matching message has arrived MPI_PROBE looking for: Tag 9, ANY_SOURCE, COMM ID 17 Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 Tag: 9 Source: 67 Comm. ID: 17

4. Regular MPI_PROBE Checks to see if a matching message has arrived MPI_PROBE looking for: Tag 9, ANY_SOURCE, COMM ID 17 Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 Match Tag: 9 Source: 67 Comm. ID: 17

5. MPI_PROBE Succeeded Now issue a receive to actually get the message MPI_RECV(…, tag=9, src=94,comm=17, …) Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 Message is removed from incoming queue Tag: 9 Source: 67 Comm. ID: 17

6. Race Condition …but what if another MPI thread issues the receive first? Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 MPI_RECV(…, tag=9, src=ANY_SOURCE,comm=17) Tag: 9 Source: 67 Comm. ID: 17

7. Race Condition In this case, your receive will end up unexpectedly blocking (!) MPI_RECV(…, tag=9, src=94,comm=17, …) Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Blocked waiting for a matching message Tag: 9 Source: 67 Comm. ID: 17

8. Race Condition If / when the receive finally completes, it’s not the message you probed MPI_RECV(…, tag=9, src=94,comm=17, …) Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 67 Comm. ID: 17 Tag: 9 Source: 94 Comm. ID: 17

9. MPI_MPROBE MPROBE = Match + probe Fixes this race condition When a message is successfully probed, it is removed from the matching queue

10. MPI_MPROBE MPI_MPROBE looking for: Tag 9, ANY_SOURCE, COMM ID 17 Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 Tag: 9 Source: 67 Comm. ID: 17

11. MPI_MPROBE When the match occurs, message is removed from the incoming queue MPI_MPROBE looking for: Tag 9, ANY_SOURCE, COMM ID 17 Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 Tag: 9 Source: 67 Comm. ID: 17

12. MPI_MPROBE Other probes / receives will no longer match this message Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 MPI_RECV(…, tag=9, src=ANY_SOURCE,comm=17) Tag: 9 Source: 67 Comm. ID: 17

13. MPI_MRECV “Matched” receive is used to receive a message that was mprobed MPI_MRECV(…, match_handle, …) Tag: 3 Source: 14 Comm. ID: 32 Incoming message queue Time Tag: 9 Source: 94 Comm. ID: 17 Guarantees that you get exactly the message you mprobed

14. Another Useful Case Probe to find the size of an incoming message MPI_PROBE(…);

15. Another Useful Case But malloc takes some time to complete MPI_PROBE(…); buf = malloc(incoming_size); Malloc takes some time

16. Another Useful Case But malloc takes some time to complete MPI_PROBE(…); buf = malloc(incoming_size); Malloc takes some time Vulnerable race condition window

17. Another Useful Case But malloc takes some time to complete MPI_PROBE(…); buf = malloc(incoming_size); MPI_RECV(…) Malloc takes some time Vulnerable race condition window Message could be stolen!

18. Another Useful Case Delays between MPROBE and MPRECV do not matter MPI_MPROBE(…); buf = malloc(incoming_size); MPI_MRECV(…) Malloc takes some time

19. Another Useful Case Delays between MPROBE and MPRECV do not matter MPI_MPROBE(…); buf = malloc(incoming_size); MPI_MRECV(…) Malloc takes some time Message cannot be stolen

20. Summary MPI_MPROBE eliminates race condition between probe and corresponding receive Good for: Event-based applications Mutli-threaded MPI applications When message lengths are unknown Strings, serialized objects, etc. E.g., bindings for Perl, Python, Boost.mpi