- McMPI is a managed-code (C#) MPI library that was implemented to remove overhead of inter-language function calls compared to using existing C-based MPI libraries in other languages like C++. - It uses object-oriented design patterns like abstract factory and bridge to isolate concerns and enable extensibility. - Performance results show its shared-memory latency and bandwidth is comparable to other MPI implementations, while its distributed-memory latency is higher but bandwidth is comparable.

1. McMPI
Managed-code MPI library in Pure C#
Dr D Holmes, EPCC
dholmes@epcc.ed.ac.uk

2. Outline
• Yet another MPI library?
• Managed-code, C#, Windows
• McMPI, design and implementation details
• Object-orientation, design patterns,
communication performance results
• Threads and the MPI Standard
• Pre-“End Points proposal” ideas

3. Why Implement MPI Again?
• Parallel program, distributed memory => MPI library
• Most (all?) MPI libraries written in C
• MPI Standard provides C and FORTRAN bindings
• C++ can use the C functions
• Other languages can follow the C++ model
• Use the C functions
• Alternatively, MPI can be implemented in that language
• Removes inter-language function call overheads but …
• May not be possible to achieve comparable performance

4. Why Did I Choose C#?
• Experience and knowledge I gained from my career in
software development
• My impression of the popularity of C# in commercial
software development
• My desire to bridge the gap between high-performance
programming and high-productivity programming
• One of the UK research councils offered me funding for a
PhD that proposed to use C# to implement MPI

5. C# Myths
• C# only runs on Windows
• Not such a bad thing – 3 of the Top500 machines use Windows
• Not actually true – Mono works on multiple operating systems
• C# is a Microsoft language
• Not such a bad thing – resources, commitment, support, training
• Not actually true – C# follows ECMA and ISO standards
• C# is slow like Java
• Not such a bad thing – expressivity, readability, re-usability
• Not actually true – no easy way to prove this conclusively
• C# and its ilk are not things we need to care about
• Not such a bad thing – they will survive/thrive, or not, without us
• Not actually true – popularity trumps utility

6. McMPI Design & Implementation
• Desirable features of code
• Isolation of concerns -> easier to understand
• Human readability -> easier to maintain
• Compiler readability -> easier to get good performance
• Object-orientation can help with isolation of concerns
• So can modularisation and judiciously reducing LOC per code file
• Design patterns can help with human readability
• So can documentation and useful in-code comments
• Choice of language & compiler can help with performance
• So can coding style and detailed examination of compiler output
• What is the best compromise?

7. Communication Layer
• Abstract class factory design pattern
• Similar to plug-ins
• Enables addition of new functionality without re-compilation of the
rest of the library
• All communication modules:
• Implement the same Abstract Device Interface (ADI)
• Isolate the details of their implementation from other layers
• Provide the same semantics and capabilities
• Reliable delivery
• Ordering of delivery
• Preservation of message boundaries
• Message = fixed size envelope information and variable size user data

8. Communication Layer – UML

9. Protocol Layer
• Bridge design pattern
• Enables addition of new functionality without re-compilation of the
rest of the library
• All protocol messages:
• Implement inherit from the same base class
• Isolate the details of their implementation from other layers
• Modify state of internal shared data structures independently
• Shared data structures (message ‘queues’)
• Unexpected queue – message envelope at receiver before receive
• Request queue – receive called before message envelope arrival
• Matched queue – at receiver waiting for message data to arrive
• Pending queue – message data waiting at sender

10. Protocol Layer – UML

11. Interface Layer
• Simple façade design pattern
• Translates MPI Standard-like syntax into protocol layer syntax
• Will become adapter design pattern
• For example, when custom data-types are implemented
• Current version of McMPI covers parts of MPI 1 only
• Initialisation and finalisation
• Administration functions, e.g. to get rank and size of communicator
• Point-to-point communication functions
• ready, synchronous, standard (not buffered)
• blocking, non-blocking, persistent
• Previous version had collectives
• Implemented on top of point-to-point
• Using hypercube or binary tree algorithms

12. McMPI Implementation Overview

13. Performance Results – Introduction 1
• Shared-memory results – hardware details
• Number of Nodes: 1 Armari Magnetar server
• CPUs per Node: 2 Intel Xeon E5420
• Threads per CPU: 4 Quad-core, no hyper-threading
• Core Clock Speed: 2.5GHz Front-side bus 1333MHz
• Level 1 Cache: 4x2x32KB Data & instruction per core
• Level 2 Cache: 2x6MB One per pair of cores
• Memory per Node: 16GB DDR2 667MHz
• Network Hardware: 2xNIC Intel 82575EB Gigabit Ethernet
• Operating System: WinXP Pro 64bit with SP3 version 5.2.3790

14. Performance Results – Introduction 2
• Distributed-memory results – hardware details
• Number of Nodes: 18 Dell PowerEdge 2900
• CPUs per Node: 2 Intel Xeon 5130 Fam 6 mod 15 step 6
• Threads per CPU: 2 Dual-core, no hyper-threading
• Core Clock Speed: 2.0GHz Front-side bus 1333MHz
• Level 1 Cache: 2x2x32KB Data & instruction per core
• Level 2 Cache: 1x4MB One per CPU
• Memory per Node: 4GB DDR2 533MHz
• Network Hardware: 2xNIC BCM5708C NetXtreme II GigE
• Operating System: Win2008 Server x64, SP2 version 6.0.6002

15. Shared-memory – Latency
0
1
2
3
4
5
6
1 2 4 8 16 32 64 128 256 512 1,024 2,048 4,096 8,192 16,384 32,768
Latency(µs)
Message Size (bytes)
MPICH2 Shared Memory
MS-MPI Shared Memory
McMPI thread-to-thread

16. Shared-memory – Bandwidth
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
4,096 8,192 16,384 32,768 65,536 131,072 262,144 524,288 1,048,576
Bandwidth(Mbit/s)
Message Size (bytes)
McMPI thread-to-thread
MPICH2 shared-memory
MS-MPI shared-memory

17. Distributed-memory – Latency
0
50
100
150
200
250
300
350
400
450
500
550
600
1 2 4 8 16 32 64 128 256 512 1,024 2,048 4,096 8,192 16,384 32,768
Latency(µs)
Message Size (bytes)
McMPI Eager
MS-MPI

18. Distributed-memory – Bandwidth
0
250
500
750
1,000
4,096 8,192 16,384 32,768 65,536 131,072 262,144 524,288 1,048,576
Bandwidth(Mbit/s)
Message Size (bytes)
McMPI Rendezvous
McMPI Eager
MS-MPI

19. Thread-as-rank – Threading Level
• McMPI allows MPI_THREAD_AS_RANK as input for the
MPI_INIT_THREAD function
• McMPI creates new threads during initialisation
• Not needed – MPI_INIT_THREAD must be called enough times
• McMPI uses thread-local storage to store ‘rank’
• Not needed – each communicator handle can encode ‘rank’
• Thread-to-thread message delivery is zero-copy
• Direct copy from user send buffer to user receive buffer
• Any thread can progress MPI messages

20. Thread-as-rank – MPI Process
Diagram created
by Gaurav Saxena
MSc, 2013

21. Thread-as-rank – MPI Standard
• Is thread-as-rank compliant with the MPI Standard?
• Does the MPI Standard allow/support thread-as-rank?
• Ambiguous/debatable at best
• The MPI Standard assumes MPI process = OS process
• Call MPI_INIT or MPI_INIT_THREAD twice in one OS process
• Erroneous by definition or results in two MPI processes?
• MPI Standard “thread compliant” prohibits thread-as-rank
• To maintain a POSIX-process-like interface for MPI process
• End-points proposal violates this principle in exactly the same way
• Other possible interfaces exist

22. Thread-as-rank – End-points
• Similarities
• Multiple threads can communicate reliably without using tags
• Thread ‘rank’ can be stored in thread-local storage or handles
• Most common use-case likely requires MPI_THREAD_MULTIPLE
• Differences
• Thread-as-rank part of initialisation and active until finalisation
• End-points created after initialisation and can be destroyed
• Thread-as-rank has all possible ranks in MPI_COMM_WORLD
• End-points only has some ranks in MPI_COMM_WORLD
• Thread-as-rank cannot create ranks but may need to merge ranks
• End-points can create ranks and does not need to merge ranks

23. Thread-as-rank – MPI Forum Proposal?
• Short answer: no
• Long answer: not yet, it’s complicated
• More likely to be suggested amendments to end-points proposal
• Thread-as-rank is a special case of end-points
• Standard MPI_COMM_WORLD replaced with an end-points
communicator during MPI_INIT_THREAD
• Thread-safety implications are similar (possibly identical?)
• Advantages/opportunities similar
• Thread-to-thread delivery rather than process-to-process delivery
• Work-stealing MPI progress engine or per-thread message queues

24. Questions?