Reading Circle

1. OverlappingCommunicationand
ComputationbyUsingaHybrid
MPI/SMPSsApproach
ReadingCircileinTauralaboratory
B4TakuyaFukuoka
November29,2018
1

2. Aboutthispaper
Author:VladimirMarjanovic,JosepM.Perez,EduardAyguade,
JesusLabarta,MateoValero
TheyarefromBarcelonaSupercomputingCenter.
ThispaperispublishedinICS'10Proceedingsofthe24thACM
InternationalConferenceonSupercomputing
Iwillintroducethepaperwiththeversionuploadedhere
Numberofcitationsis78(atNovember2018)
ThispaperiscitedbyMPI+ULT,HCMPI(Habanero‑C+MPI),
MPIQ(MPI+Qthreads)
2

3. Introduction(1)
ThispaperfocusedonHybriduseofMPIandTaskParallelism
(SMPSs)
TherearetwokindsofMPIfunction
Blockingfunction(MPI_Send,MPI_Recv)
Non‑blockingfunction(MPI_Isend,MPI_Irecv)
Theadvantageofnon‑blockingfunction
Communication/computationoverwrap
Thedisadvantageofnon‑blockingfunction
Codecomplexity
Lessprogrammingproductivity
3

4. Introduction(2)
MPI/SMPSsapproach
Enablecommunication/computationoverwrapwithsimple
annotation(lesscodecomplexity)
Betoleratetonetworkbandwidthandpreemptions.
ArestartmechanisimisadoptedinMPI/SMPSs
EvaluatedbyHPL(LINPACKbenchmark)
4

5. Topics
SMPSs
HybridMPI/SMPSs
HPLbenchmark
Performanceresults
Mythoughts
5

6. SMPSs(1)
SMPsuperscalarprogrammingmodel
EtensionofthestandardC/Fortranprogramminglanguage
Source‑to‑sourcecompiler
Usepragmas/directivestodeclarefunctionsthatarepotential
tasks
#pragma css task input(n) output(result)
void factorial (unsigned int n, unsigned int *result)
{
*result = 1;
for (; n > 1; n--)
*result = *result * n;
}
void main(){
// executed in one piece as a task
factorial(n, result);
}
6

7. SMPSs(2)
#pragma css task [clause-list]
{ function-header | function-definition}
Therearefourkindsofclause
input(data‑reference‑list)
output(data‑reference‑list)
inout(data‑reference‑list)
highpriority
Thefirstthreeclause(input,output,inout)isusedtodeterminethe
dependencyoftasks
Thehighpriorityclauseisusedtospecifytheprioritywhen
schedulingtasks.
7

8. SMPSs(3)
# pragma css barrier
# pragma css wait on <list of variables>
TheSMPSsprogrammingmodelcanaddabarrier
Inthefirstexample,theexecutionisstoppeduntilallthetasksis
finished.
Inthesecondexample,theexecutionisstoppeduntilthevariables
in<listofvariables>arecompletelyupdated.
8

9. SMPSsinternals(1)
Thecompilerreplacecallsdefinedastaskswithcallstothe
css_addTaskfunction
specifiedthefunctiontobeexecutedanditsarguments
css_addTaskfunctionbuildadependancetaskgraph
considermemoryaddress,sizeanddirectionofeach
parameter
Onceataskisfinished,theglobalreadyqueueisupdated.
alltaskswithnopendingdependencesisinsertedintothe
globalreadyqueue
therearetwoglobalreadyqueues:forhigh‑priorityandfor
low‑priority
9

10. SMPSsinternals(2)
Eachworkerthreadshaveownreadyqueue,andlookforready
taskinglobalreadyqueue
Thereareworkstealingmechanism
fromSMPSuperscalar(SMPSs)User'sManual
10

11. SMPSsinternals(3)
Inordertoavoidfalsedependancies,therenamingmechanismis
implemented(likeregisterrenaminginsuperscalarprocessor)
11

12. HybridMPI/SMPSs(1)
EncapsulateMPIfunctionswithdirectiveastasks
Blockingfunctionissplitintoanon‑blockingcallandawaitcall
#pragma css task output(buf, req)
void recv (<type>
buf[count], MPI_Request *req){
MPI_Irecv(buf,... ,req);
}
12

13. HybridMPI/SMPSs(2)
# pragma css restart
Inordertoavoiddeadlock,therestartpragmaisintroduced
Aborttheexecutionofcurrenttaskandputitagaininthe
readyqueue
#pragma css task input(req)
void wait (MPI_Request *req){
int go;
MPI_Test (req, &go, ...);
if (go==0) #pragma css restart;
MPI_Wait (req_recv, ...);
}
void application_receive(){
recv ();
wait ();
}
13

14. HPLbenchmark(1)
Themostwidelyusedbenchmarktomeasurethefloating‑pointexecutionrateof
acomputer
Thekernelsolvesasystemoflinearequations
ImplementLUdecompositionwithpartialpivoting
Introducelook‑aheadtechniquetooverwrapcomputation/communication
14

15. HPLbenchmark(2)
Thisistheflowwithmultipleprocesses
KeepingoneMPIcallinoneprocessbyusingthebarrier
15

16. Performanceresults(1)
Basiccomparison
Inspiteoftheoverheadofabstraction,
MPI/SMPSsshowbetterperformance
thanpureMPI.
Theuseofnon‑blockingMPI
calls
Theefficientuseofmemoryby
adoptinglook‑aheadtechnique
Withtheincreasethematrizsize,the
performancerateincreases
Influenceofcommunication
overheadissmaller 16

17. Performanceresults(2)
Tolerancetolowbandwidth
ForeachmessageofsizeS,an
additionalmessageofsizef*Sis
transferredbetweentwodummybuffers
atsenderandreceiver
Normalizedeffectivebandwidthmeans
1/(1+f)
PureMPIversionismuchmoresensitive
17

18. Performanceresults(3)
TolerancetoOSnoise
Generatinganadditionalthreadper
processthatiteratesonaloopthat
alternatessleepingandcomputing
phases.
Durationofperiodicalcomputationis
fixedat0.5s
Durationofperiodicalsleepingis
changed
PureMPIversionismuchmore
sensitive
18

19. Futurework
ImplementMPIcollectiveoperationbycombiningthehybrid
MPI/SMPSsprogrammingmodelandnon‑blockingMPIcollective
operationslibrary
19

20. MyThoughts(1)
ItisunfairtocomparehybridMPI/SMPSswithpureMPI.Ithinkthe
non‑blockingMPIversionwithcodecomplexityhavetobe
measured.
AsfarasIknow,itisimpossibletonesttasksinSMPSsmodelsand
thisrestrictstheprogramability.
HybridMPI/SMPSsmodelscannotdealwithinvocationofMPI
callsfrommultiplethreads.Theprogrammerhavetoserializethe
MPIcallsusingbarrier.Inthispoint,thispaperisuselessformy
research.
20

21. MyThoughts(2)
ThemeasurementoftolerancetolowbandwidthandOSnoiseis
interestingforme.Thisideacantakeintomyresearch.
21