In this slidecast, Jeff Squyres from Cisco Systems presents: How to make MPI Awesome - MPI Sessions. As a proposal for future versions of the MPI Standard, MPI Sessions could become a powerful tool tool to improve system resiliency as we move towards exascale. Watch the video presentation: http://wp.me/p3RLHQ-f4U Learn more: http://blogs.cisco.com/performance/mpi-sessions-a-proposal-for-the-mpi-forum Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. How to make MPI Awesome:
MPI Sessions
Wesley Bland, Intel
Ryan Grant, Sandia Na=onal Laboratory
Dan Holmes, Edinburgh Parallel Compu=ng Center
Kathryn Mohror, Lawrence Livermore Laboratory
Mar=n Schulz, Lawrence Livermore Laboratory
Anthony Skjellum, Auburn University
Jeff Squyres, Cisco Systems, Inc.
^
more
Follow-on to Jeff’s crazy thoughts
discussed at EuroMPI 2015 in Bordeaux, France

2. What we want
• Fix MPI-3.1 limita=ons:
– Cannot ini=alize MPI from different en==es within
a process without a priori knowledge /
coordina=on
– Cannot ini=alize MPI more than once
– Cannot re-ini=alize MPI aer it has been finalized
– Cannot set error behavior of MPI ini=aliza=on

3. What we want
• Any thread (e.g., library) can use MPI any =me it wants
• But s=ll be able to totally clean up MPI if/when desired
MPI Process
// Library 3
MPI_Init(…);
// Library 4
MPI_Init(…);
// Library 5
MPI_Init(…);
// Library 6
MPI_Init(…);// Library 7
MPI_Init(…);
// Library 8
MPI_Init(…);
// Library 9
MPI_Init(…);
// Library 10
MPI_Init(…);
// Library 11
MPI_Init(…);
// Library 12
MPI_Init(…);// Library 1
MPI_Init(…);
// Library 2
MPI_Init(…);

4. How do we get those things?

5. New concept: “session”
• A local handle to the MPI library
– Implementa=on intent: lightweight / uses very
few resources
– Can also cache some local state
• Can have mul=ple sessions in an MPI process
– MPI_Session_init(…, &session);
– MPI_Session_finalize(…, &session);

6. MPI Session
MPI Process
ocean library
MPI_SESSION_INIT(…)
atmosphere library
MPI_SESSION_INIT(…)
MPI library

7. MPI Session
MPI Process
ocean library
atmosphere library
MPI library
ocean
session
atmos-
phere
session
Unique handles to the underlying MPI library

8. MPI Session
MPI Process
ocean library
atmosphere library
MPI library
ocean
Errors
return
atmos-
phere
Errors
abort
Unique error handlers, info, local state, etc.

9. Great. I have a session.
Now what?

10. Fair warning
• The MPI run=me has
long-since been a
bastard stepchild
– Barely acknowledged in
the standard
– Mainly in the form of
non-norma=ve
sugges=ons
• It’s &me to change that

11. Work with the run=me
• General scheme:
– Query the underlying
run-=me system
• Get a “set” of processes
– Determine the processes
you want
• Create an MPI_Group
– Create a communicator
with just those
processes
• Create an MPI_Comm
Query run=me
for set of processes
MPI_Group
MPI_Comm
MPI_Session

12. Run =me exposes sets of processes
• Sets are iden=fied by string name
• Two sets are mandated
– “mpi://WORLD”
– “mpi://SELF”
• Other implementa=on-defined sets can be
reported, too

13. Examples of sets
MPI process 0 MPI process 1 MPI process 2 MPI process 3
mpi://WORLD

14. Examples of sets
MPI process 0 MPI process 1 MPI process 2 MPI process 3
mpi://WORLD
arch://x86_64

15. Examples of sets
MPI process 0 MPI process 1 MPI process 2 MPI process 3
mpi://WORLD
job://12942
arch://x86_64

16. Examples of sets
MPI process 0 MPI process 1 MPI process 2 MPI process 3
loca=on://rack/17 loca=on://rack/23

17. Well, that all sounds great.
…but who calls MPI_INIT?
And what session does
MPI_COMM_WORLD /
MPI_COMM_SELF belong to?

18. New concept: no longer require
MPI_INIT / MPI_FINALIZE

19. The overall theme
• Just use MPI func=ons whenever you want
– Ini=aliza=on essen=ally becomes an
implementa=on detail
• Finaliza=on will occur whenever all user-
defined handles are destroyed

20. Example
int main() {
// Create a datatype – initializes MPI
MPI_Type_contiguous(2, MPI_INT, &mytype);
The crea=on of the first user-
defined MPI object ini=alizes MPI
Ini=aliza=on can be a local ac=on!

21. Example
int main() {
// Create a datatype – initializes MPI
MPI_Type_contiguous(2, MPI_INT, &mytype);
// Free the datatype – finalizes MPI
MPI_Type_free(&mytype);
}
The destruc=on of the last user-
defined MPI object finalizes MPI.

22. Example
int main() {
// Create a datatype – initializes MPI
MPI_Type_contiguous(2, MPI_INT, &mytype);
// Free the datatype – finalizes MPI
MPI_Type_free(&mytype);
// Re-initialize MPI!
MPI_Type_dup(MPI_INT, &mytype);
We can also re-ini=alize MPI!
(it’s transparent to the user – so why not?)

23. Key insight:
Split MPI APIs into two sets
Performance doesn’t
ma/er (as much)
• Func=ons that create / query /
destroy:
– MPI_Comm
– MPI_File
– MPI_Win
– MPI_Info
– MPI_Op
– MPI_Errhandler
– MPI_Datatype
– MPI_Group
– MPI_Session
– Arributes
– Processes
• MPI_T
Performance
absolutely ma/ers
• Point to point
• Collec=ves
• I/O
• RMA
• Test/Wait
• Handle language xfer

24. Key insight:
Split MPI APIs into two sets
Performance doesn’t
ma/er (as much)
• Func=ons that create / query /
destroy:
– MPI_Comm
– MPI_File
– MPI_Win
– MPI_Info
– MPI_Op
– MPI_Errhandler
– MPI_Datatype
– MPI_Group
– MPI_Session
– Arributes
– Processes
• MPI_T
Performance
absolutely ma/ers
• Point to point
• Collec=ves
• I/O
• RMA
• Test/Wait
• Handle language xfer
Ensure that MPI is
ini=alized (and/or
finalized) by these
func=ons
These func=ons s=ll can’t
be used unless MPI is
ini=alized

25. Key insight:
Split MPI APIs into two sets
Performance doesn’t
ma/er (as much)
• Func=ons that create / query /
destroy:
– MPI_Comm
– MPI_File
– MPI_Win
– MPI_Info
– MPI_Op
– MPI_Errhandler
– MPI_Datatype
– MPI_Group
– MPI_Session
– Arributes
– Processes
• MPI_T
Performance
absolutely ma/ers
• Point to point
• Collec=ves
• I/O
• RMA
• Test/Wait
• Handle language xfer
These func=ons init / finalize
MPI transparently
These func=ons can’t be called
without a handle created from
the le-hand column

26. Wait a minute –
What about MPI_COMM_WORLD?
int main() {
// Can’t I do this?
MPI_Send(…, MPI_COMM_WORLD);
This would be calling a
“performance marers”
func=on before a
“performance doesn’t marer”
func=on
I.e., MPI has not ini=alized yet

27. Wait a minute –
What about MPI_COMM_WORLD?
int main() {
// This is valid
MPI_Init(NULL, NULL);
MPI_Send(…, MPI_COMM_WORLD);
Re-define MPI_INIT and MPI_FINALIZE:
constructor and destructor for
MPI_COMM_WORLD and MPI_COMM_SELF

28. INIT and FINALIZE
• INIT/FINALIZE create an implicit session
– You cannot extract an MPI_Session handle for the
implicit session created by MPI_INIT[_THREAD]
• Yes, you can use INIT/FINALIZE in the same
MPI process as other sessions

29. (Abbreviated) Summary
• MPI session: a unit of isola=on
– Allow “ocean” and “atmosphere” scenarios
• Define some interac=ons with the run=me
– Query process sets, make groups, make
communicators
• No longer require MPI_INIT / FINALIZE
– Key insight: cannot invoke “performance marers”
func=ons without a handle

30. More ideas and details
in the full slide deck
hrp://blogs.cisco.com/performance/
“MPI Sessions:
A proposal to the MPI Forum”
Blog entry from March 2, 2016