Reading Circle

1. Tardis:TimeTravelingCoherence
AlgorithmforDistributedSharedMemory
ReadingCircleinTauraLaboratory
M1TakuyaFukuoka
July11,2019
1

2. Aboutthispaper
Author:XiangyoYuandSrinivasDevadas
ThispaperispublishedinPACT'15(oneof4bestpapers).
Itintroduceslogicaltimestampcounterstoordermemory
operationsandenforcesequentialconsistencyinmulticore‑
processors
Aproofofcorrectnessiswritteninanotherpaper.
Apaperfortheversionofrelaxedconsistencymodelsisalso
published.
2

3. Introduction
Thispaperfocusesonthecomparisonofcachecoherencemethod
betweendirectorybasedcoherenceandTardisinmulticore‑
processors
Tardisissimpleandscalesbetterintermsoftheamountofstorage
Whiledirectorybasedcoherencedependsonphysicaltime,Tardis
introduceslogicaltimestampcoherence
ThispapershowsTardiscanachievesimilarperformanceas
directorybasedcoherence
3

4. Background
4

5. SequentialConsistency(1)
Amemoryconsistencymodeldefinesthecorrectnessofashared
memorysystem
Aparallelprogramissequentiallyconsistentif"theresultofany
executionisthesameasiftheoperationsofallprocessors(cores)
wereexecutedinsomesequentialorder,andtheoperationsof
eachindividualprocessor(core)appearinthissequenceinthe
orderspecifiedbyitsprogram"
5

6. SequentialConsistency(2)
Thisistheprogrammer'sabstractionofthememorysubsystem
undersequentialconsistencymodel
Everyprocessorappearstoissueandcompletememory
operationsataonetimeatomicallyinprogramorder
fromParallelComputerArchitectureAHardware/SoftwareApproach
6

7. DirectoryBasedCoherenceProtocol
Widelyusedprotocoltohandlecachecoherencyprotocols
Thedirectoryissoftwareorhardwarestructuretrackinghowthe
dataaresharedorownedbydifferentcores/processors
Inadirectoryprotocol,sequencialconsistencyisachievedby
invalidationmechanism
Whenacorewritestotheblockwhichisshared,alltheshared
copiesneedtobeinvalidated
7

8. MSIProtocol
Threestatesofcachesexist
Modified:onecacheexclusivelyhasmodifiedcopy.Thecore
whichhasthecopyiscalled"owner".Thememoryisstale.
Shared:morethanonecacheandmemoryhavevalidcopy.The
ownerismemory.
Invalid:onlymemoryhasvalidcopyandmemoryisowner
http://people.ee.duke.edu/~sorin/prior‑courses/ece152‑spring2009/lectures/8.3‑multicore.pdf
8

9. AnExampleProgram
WeassumetheinstructionsfromCore0areexecutedbeforethe
instructionsinCore1
WeexecutethiswithMSIdirectoryprotocol
9

10. Cycle0(MSIdirectoryprotocol)
InitialStatesofAandBare"Shared"
Nosharersinbothcachelines
10

11. Cycle1(MSIdirectoryprotocol)
Core0invalidatesacacheofAinothercores(nocorehasacache
ofA)
AfterCore0writestoAonthecache,thestateofAchangestoE
(exclusive,modified)
11

12. Cycle2(MSIdirectoryprotocol)
ThestateofBisshared,soCore0canfetchcacheofBwithout
modification
12

13. Cycle3(MSIdirectoryprotocol)
Core1invalidatesacacheofBinothercores
AfterCore1writestoB,thestateofBchangestoE(exclusive,
modified)
13

14. Cycle4(MSIdirectoryprotocol)
ThestateofAisexclusive,soCore1demandsthatCore0should
writebackthecacheofA
Afterthat,thestateofAchangestoshared,andCore1canfetch
thecacheofA.
14

15. BasicProtocolofTardis
15

16. TimestampmanagementinTardis(1)
Timestampsaremaintainedaslogicalcounters
Eachcorehasaprogramtimestamp(pts)
Eachcachelinekeepstwokindsoftimstamps,aread
timestamp(rts)andawritetimestamp(wts)
Basically,themeaingsofrtsaretheendtimestampsofthe
reservationofthecaches
Themeeningsofwtsarewhenthevalueiswritten
Ifptsorwtsisgreaterthanorequaltorts,thecachelinegets
invalid
16

17. LoadOperationinTardis
Ifthecacheisvalid,thecoreloadsthedatafromexistingcache.
Ifthecacheisinvalidandshared,thecorerequeststotimestamp
managerforextendingtherts.Thevalueof"thelease"ishow
manytimestampsitextends.
Ifthecacheisinvalidandexcludedbyanothercore,itsends"write
back"requesttotheownercorethroughtimestampmanagerand
fetchnewcache.Atthistime,pts,wtsandrtsareupdated
17

18. StoreOperationinTardis
Ifthecacheisexcludedbyitsowncore,thecorecanwritefreely
Ifthecacheisshared,thecoreupdatespts,rtsandwtswiththe
valuegreaterthanrtsofothercores.Afterthat,itchangesthe
statusto"excluded".ThepointisTardisdoesnotinvalidatecaches
ofothercoresinwriting.
Ifthecacheisexcludedbyanothercore,itsends"writeback"
requesttotheownercorethroughthetimestampmanager,anddo
thesamethingasinacaseofthecacheisshared
18

19. Cycle0(Tardis)
Thevalueoftheleaseissetto10
Bothofthecachesaresharedstates
19

20. Cycle1(Tardis)
Becausethecacheisshared,thecoreupdatesrtsandwtsto1(the
valuegreaterthan0)
Changethestateofthecachefrom"shared"to"excluded"
20

21. Cycle2(Tardis)
Becausecacheissharedandinvalid(notwts<rts),updatetherts
with0(originalwts)+10(lease)
21

22. Cycle3(Tardis)
Becausethecacheisshared,updatepts,rts,andwtswiththe
valuegreaterthan11(originalrts)
Changethestateofthecachefrom"shared"to"excluded"
ThepointisthatnotinvalitatingthecacheofCore0
22

23. Cycle4(Tardis)
Becausethecacheisexcludedbyothercore(Core0),Core0
writesbackthecachefirst
Thenupdatethertswith12(pts)+10(lease)
23

24. TardisvsDirectoryBasedCoherence
InTardis,whenwritingtoonesharedcacheoccurs,the
invalidationsofcachesofothercoresareunnecessary
Tardishasanadvantageofscalabilityoverdirectorybased
coherence
Indirectorybasecoherence,eachcacheblockshouldhave
thesharer'sinformationandrequireO(N)storage
Ontheotherhand,whatinformationTardisrequiresineach
cacheblocksisonlyoneowner'sID,anditrequiresO(logN)
storage
24

25. AvoidingLivelock
InTardis,acoremaycontinuetoreadtheoldcacheddataevenif
anothercorehasupdateditaslongasthecachelinehasnot
expired
Forexample,ifacorespinsonavariablewhichissetbyanother
core,theptsofthespinningcoredoesnotincreaseandthecore
mayspinforever
Tardisintroducesselfincrementationofpts
25

26. Out‑of‑OrderExecution
Forout‑of‑ordercores,amemoryinstructioncanbeexecuted
beforepreviousinstructionsfinishandthusthecurrentptsarenot
known
Changethemeaningofptstomeanthetimestampofthelast
committedinstruction
Whencommittinganinstruction,ifptsisgreaterthanrts,the
instructionabortsandisrestartedwiththelatestpts
26

27. Evaluation
27

28. Environments
UseGraphitemulti‑coresimulatorandSplash‑2benchmarks
ThebaselineissimplestdirectoryMSIprotocol
AckwiseisdirectoryMSIProtoclwhichkeepsalimitednumberof
sharersandbroadcastinvalidationstoallcoreswhenthenumber
ofsharersexceedsthelimit
28

29. Performanceonin‑ordercores
ThoughputofTardisiswithin0.5%ofthebaselineMSIand
Ackwise
Tardisincurs21.2%morenetworktrafficanditismainlycausedby
synchronization
29

30. Performanceonout‑of‑ordercores
ThoughputofTardisiswithin0.2%
Thenetworktrafficisnotmuchdifferentfromin‑ordercores
30

31. Scalability(1)
InTardis,theamountofstrageisthesamewithdifferentnumbers
ofcores
31

32. Scalability(2)
Comparetheperformanceof16coresand256cores
Tardisperiodmeanstheselfincrementperiodtoavoidlivelock
Itisgenerallyconsideredbadpracticetousespinningathighcore
count 32

33. MyThought
Ithinktheauthorisgeniousandthecoreideaissimpleandbrilliant
BecausethestorageoverheadofTardisislargerthanAckwisein
64cores,soIthinkTardisonlycanbeeffectiveinverymany
countsofcores
33