HybridMPI (HMPI) is an MPI implementation optimized for shared memory communication on multi-core systems. HMPI uses a process-based approach instead of thread-based to avoid locking overhead. It allocates shared memory using a heap allocator to eliminate data copying between processes on the same node. For small messages, HMPI uses an immediate transfer protocol to prevent cache misses, while for large messages it uses a synergistic transfer protocol for high bandwidth. Evaluation shows HMPI causes fewer cache misses and outperforms traditional MPI in applications like MiniMD and Lulesh.

1. HybridMPI:EfficientMessagePassingfor
Multi‑coreSystems
ReadingCircleinTauraLaboratory
B4TakuyaFukuoka
February20,2019
1

2. Aboutthispaper
Author:AndrewFriedley,GregBronevetsky,TorstenHoefler,and
AndrewLumsdaine.
ThispaperispublishedinSC'13.
ItintroducesHybridMPI(HMPI),whichadoptsprocess‑based
approachthatenablessharedmemorycommunication.
Theimplementationispublishedin
https://code.google.com/archive/p/hmpi.
Thefirstauthor'sdoctoraldissertionexplainsmoredetailsabout
thisimplementation.
2

3. Introduction
MPIisdesignedforsystemswheresingle‑corecomputenodes
wereconnectedbyaninter‑nodenetwork
ThechallengeofMPIimplementationsisprovideefficient
abstractionforsharedmemory
ThispaperproposedHybridMPI(HMPI)thatisoptimizedfor
intra‑nodesharedmemorycommunication
HMPIadoptsprocess‑basedapproachinsteadofthread‑based
approach
3

4. Background
4

5. Traditionalprocess‑basedMPI
Noapplication‑visiblememoryisshared
Withinthesamesharedmemorynode,two‑copyprotocolis
adopted.
5

6. Two‑copyprotocol
fromAndrew'sdoctoraldissertion
Thesendercopiesoneblockintothesharedregion,thenthe
receivercopiesittothereceivebuffer
Topiplinethetwocopiesbybreakingthemessageintoblocks 6

7. Thread‑basedMPI
AnyMPIusingmultiplethreadsmustprotectallinter‑node
communicationusingalockanditcausespoorperformance
Nothread‑basedMPIhasbeenwidelyadoptedinpractice
7

8. ImplementationofHMPI
8

9. HybridMPI(HMPI)
LayeringHMPIontopofanyexistingMPIlibrary
Thisarchitectureassuresportabilityandtransparency
Basically,HMPIadoptssingle‑copymessagepassingtechnique
insteadoftwo‑copyprotocol(explainlater)
9

10. SharedMemoryHeapAllocator(1)
HMPIoverridesthesystem'sdefaultmemoryallocator(malloc)to
allocatememoryfromaspeciallycraftedsharedmemorypool
Therefore,thereisnoneedofinstallationofkernelextentions,
modificationofsystemlibraries,oradministrativepermissions
10

11. SharedMemoryHeapAllocator(2)
fromAndrew'sdoctoraldissertion
Eachprocessmapstheentireregionatthesameaddress,then
allocatesitsmemoryonlyfromitsportion
Thisapproacheliminatestheneedforanysynchronization
betweenprocesses 11

12. MessageMatching
SharedQueuecanbeaccessedwithlockbyallprocessesinone
node
ThesenderinsertsmessageintoSharedQueue
EachprocesshasitsownPrivateQueue
ThereceiverdrainsSharedQueueandaddincomingsendsto
PrivateQueue 12

13. Single‑copyProtocol(1)
13

14. Single‑copyProtocol(2)
14

15. Single‑copyProtocol(3)
15

16. CommunicationProtocol
Withthesharedmemoryheapallocator,youneedtocopyonly
onceinintra‑nodecommunication
Therearemoreeffectiveprotocolsforsmallmessagesizeand
largemessagesizerespectively
16

17. ImmediateTransferProtocol(1)
fromAndrew'sdoctoraldissertion
Forsmallmessagesize,HMPIadoptimmediatetransferprotocol
Themessagedataislocatedwiththemessagematching
information
Thisprotocolneedsthatmessageshouldbecopiedtwice,itcan
preventthesecondcachemiss 17

18. ImmediateTransferProtocol(2)
18

19. ImmediateTransferProtocolResult
Choosethethresholdof256bytesand
ifthemessagesizeisunderit,youuse
immediatetransferprotocol
Whenthemessagesizeisunderthis
threshold,theimmediateprotocol
performbetterthanMemcpy(single‑
copyprotocol)
19

20. SynergisticTransferProtocol(1)
fromAndrew'sdoctoraldissertion
Forlargemessages,HMPIadoptsynergistictransferprotocolsince
bandwidthismostimportant
Notonlythereceiverbutalsothesendercopiesblocksfromsend
buffertoreceivebuffer
Useacounterandanatomicoperationforcooperation 20

21. SynergisticTransferProtocol(2)
21

22. SynergisticTransferProtocolResult
Forthemessagegreaterthantwice
theblocksize,synergistictransfer
protocolwasadopted
Synergistictransferprotocolperform
betterthanMemcpy(single‑copy
protocol)intermsofbandwidth
22

23. Evaluation
23

24. CommunicationAnalysis(1)
ThisexperimentisconductedtoverifylesscachemissinHMPI
Readdata_buf,andcommunicateandthenreaddata_bufagain
Twokindsofexperimentsareconductedchangingwhatbufferto
useincommunication
24

25. CommunicationAnalysis(2)
Icannotunderstandwhycommunicationbuffersizeisassignedas
verticalaxis...(inthecaseofCommonBuffer,communication
bufferisnotused)
ThepalecolormeanslesscachemissesinHMPI
Forsmallerdatabuffer,HMPIcauseslesscachemisses 25

26. ApplicationAnalysis(1)
MiniMDisbasedonmoleculardynamic
(MD)codeanditperformsneighbor
exchangetwiceperworkiteration
ComparedwithoriginalMPI,HMPI
achievescommunicationspeedupand
applicationspeedup
26

27. ApplicationAnalysis(2)
LuleshsimulatestheSedovblastwave
problemonauniform3Dmesh
decomposedspatially
Ineachtimestep,multipleexchanges
withupto27neighborsareperformed
ComparedwithoriginalMPI,HMPI
achievescommunicationspeedupand
applicationspeedup
27

28. ApplicationAnalysis(3)
ComparedwithoriginalMPI,HMPI
achieveslessL2andL3cachemissesin
MiniMDandLulesh
28

29. Conclusion
ThispaperproposesHMPIwhichenablessharedmemory
communicationwithprocess‑basedapproach
HMPIsharesmemoryusingSharedMemoryHeapAllocatorand
improvesperformancewithimmediatetransferprotocoland
synergistictransferprotocol
HMPIcauseslesscashemissesthanoriginalMPI
HMPIperformsbetterinminiMDandLuleshthanoriginalMPI
29

30. MyThought
ThispaperiswrittenwellandIwasabletounderstandeasilywhy
theproposedmethodisuseful
HMPIisexpectedtouseinMPIeverywheremodel,butIam
wonderingthatthemodelisbetterintermsofproductivity(code
complexity)orperformanceinthefirstplace
30