Reading Circle

1. LITEKernelRDMASupportforDatacenter
Applications
ReadingCircle
M2TakuyaFukuoka
September9,2020
1

2. AboutThisPaper
Author:Shin-YehTsaiandYiyingZhang
TheyaremembersofWukLabinUniversityofCalifornia(asof
today),whichaimstobuildnext-generationdatacenter
systems.
Thegroupisverygoodatmanagingresearchwhichrequiresa
lotoftimeandeffort.
PublishedinSOSP'17.
ProposeLITE,aLocalIndirectionTiErforRDMAintheLinuxkernel.
Sourcecodeandofficialslideisavailable
Iborrowedmanyfiguresfromtheofficialslideinthis
presentation
2

3. RDMA(RemoteDirectMemoryAccess)
Lowlatency,highthroughputandlowCPUutilization
Kernel-bypassingandzero-copytransfer
OriginallyutilizedinHPC(ex.InfiniBand)
3

4. HowRNICachieveskernel-bypassing
RDMANIC(RNIC)shouldbeabletotranslatefromanvirtual
addresstoanphysicaladdress
RNIChascachedpagetableentries(PTEs)
Permissioncheckingisnecessary
RNICkeepsprotectionkeys(rkeyandlkey)
4

5. DatacenterApplicationswithRDMA
RDMAhasbeenutilizedrecentlyindatacenternetworkaswell
BecauseofmorematurehardwaresupportforRDMA(ex.
RoCE,iWarp)
Manykindsofdatacenterapplications
Key-valuestore,databaseandtransactionalsystems,graph
storesystem,consensussystemanddistributedNVM
systems.
Resourcesharingandisolationisrequired
FundamentalmismatchbetweenwhatnativeRDMAprovidesand
whatdatacenterapplicationsdesire.
Especiallyforthreereasons
5

6. Issue1:Mismatchinabstractions
NativeRDMAprovidesalow-levelabstraction
NativeRDMAshouldexplicitlymanagevarioustypesof
resources(QP,MRand...)
Gothroughseveralnon-intuitivesteps
Optimizingtheperformanceisalsodifficult
VariousRDMAoperations,tuningoptionsandoptimizations
Ontheotherhand,datacenterapplicationsdesireahigh-level,
easy-to-use,andflexibleabstraction
6

7. Issue2:Unscalableperformance
ThescalabitityofRDMAislimitedbythememorysizeofRNIC
SRAMbutdatacenterapplicationsrequirethreetypesofscalability
TheamountofMemoryRegion(MR)
StoreeachprotectionkeysinRNICforeachMR
ThetotalsizeofMR
StorelargeamountofcachedPTEsinRNIC
ThetotalnumberofQPs(communicationresources)
StoremetadatainRNICforeachQP
https://www.sigops.org/s/conferences/sosp/2017/slides/lite-sosp17-slides.pdf 7

8. Issue3:Lackofresourcesharing,
isolation,andprotection
NativeRDMAdoesnotprovideanymechanismstoshare
resourcesacrossdifferentapplications
Furtherleadtotheperformanceissuedescribedinissue2
Hardtoisolateperformanceanddeliverqualityofservice(QoS)
betweenapplications
8

9. Contributionofthispaper
IdentifythreemainissuesofusingnativeRDMAindatacenter
environments
ProposeRDMAabstrationwithakernel-levelindirectionlayerfor
datacenterapplications.
Designasetofmechanismtominimizetheperformanceoverhead
ofkernel-levelindirection
BuildtheLITEsystemwhichsolvesallthethreeissuesofnative
RDMAfordatacenterapplications
DevelopfourdatacenterapplicationsonLITEanddoevaluation
9

10. Proposedsystem:LITE
LITEstandsforLocalIndirectionTiErforRDMA
Loadablekernelmodulewitharound15Klinesofcode.
Avoidhardwareordriverchange
Provideaflexibleandeasy-to-useabstraction
One-sidedoperations
RPCoperations(seethepaper)
Cansafelymanageallprivilegedresources
AchieveresourcesharingandguaranteeQoS 10

11. LITEAPIs
LMR(LITEMemoryRegion)isvirtualmemoryregionsthatLITEhandles
LMRcanmaptooneormorephysicalmemoryadressranges
UserscanhandleLMRonlythroughlh(LITEhandler)
LITEhidesallthedetailsaboutthetargetnode,virtualaddress,andprotection
keyofanMRfromsendersbehindthelhabstraction
11

12. EasytowriteaprogramwithLITE
Bothimplementremotememset
https://www.sigops.org/s/conferences/sosp/2017/slides/lite-sosp17-slides.pdf
12

13. TheBiggestDesignChallenge
HowtopreservetheperformancebenefitsofRDMAwhileaddingthe
indirectionneededtosupportdatacenterapplications?
Solution
.Indirectiononlyatlocalforone-sidedRDMA
.Aoidhardwareindirection
13

14. 1.Indirectiononlyatlocalnode
Thebenefitofone-sidedRDMAistheeliminationofCPU
utilizationattheremotenodecompletely
LITEaddstheindirectionlayeronlyatlocalnodeinordertoretain
thisbenefit
https://www.sigops.org/s/conferences/sosp/2017/slides/lite-sosp17-slides.pdf
14

15. 2.Avoidhardwareindirection
OnloadwhatRNICisdoingtothekernel
Addressmappingandpermissionchecking
Isitreallypossiblewithoutmodifyinghardware?
15

16. 2.Avoidhardwareindirection
YoucandoitbyaninfrequentlyusedAPIthatRDMAverbs
supports!
RegisteringthewholephysicalmemorytoRNICdirectlyas
onelargeMRusingphysicalmemoryaddresses
Omittingtheuseofvirtualaddresses!
LITEcutoutapartofthememoryandassignit
https://www.sigops.org/s/conferences/sosp/2017/slides/lite-sosp17-slides.pdf 16

17. 2.Avoidhardwareindirection
Benefits:reducingtheusageofRNICSRAMdramatically
NocachedPTEsisneededbecauseRNICdoesnotneedto
dotheaddressmapping
OnlyoneglobalprotectionkeyhastobestoredinRNIC
17

18. ResourceSharingandQoS
Evenwhenyourunmultipleapplicationswithmultiplethreads,
LITEhastopreparetheconstantnumberofQPspernode.
K×NQPswhereNisthetotalnumberofnodesandKisa
configurablefactor
LITEisflexibleenoughtoguaranteeQoSwithvariouskindsof
approach.
ThispaperintroducestwoapproachforQoSasexamples
HW-Sep:Hardwareresourceisolation
SW-Pri:Softwarebasedpriority-basedflowandcongestion
controlatthesenderside
18

19. Evaluation
Onenodeequips
TwoIntelXeonE5-26202.40GHzCPUs
128GBDRAM
40GbpsMellanoxConnectX-3NIC
19

20. Basiclatencyandthroughput
LITE'sperformanceisclosetoorsometimesbetterthannative
RDMA(verbs)dispitetheoverheadofpermissioncheckingand
addresstranslationinthekernel
20

21. EffectofnumberofMRsandTotalsize
NativeRDMAdegradesperformancewiththelargescale
DuetothelimitationofRNICSRAMsize
Ontheotherhand,LITEscaleswell
21

22. RegisterandDe-registerProcesses
LITE_mapissomethinglikeanoperationthatacquirestheaccesspermission
Thecostofregister/de-registerofLITEissmallerthannativeRDMA
BecauseLITEhandlesphysicaladdressdirectlyanddoesnotneedmemory
registration
LITE_unmapseemstotakemoretimethanLITE_mapbecauseitrequiresthe
notificationtothemasternode(honestly,Iamnotsure)
22

23. LITEApplications
DeveloppedfourLITEsystems
LITEissosimpletousethatthelinesofcoderelatedtothe
communicationaresmall
23

24. MapReduce
PortedLITE-MRfromPhoenix,asingle-nodemulti-threaded
implementationofMapReduce
ComparetheWordCountruntimeoftheWikimediaworkloads
withsinglenodePhoenixandHadoop
LITE-MRoutperformsHadoopby4.3×to5.3×
IthinkthatHadoopistooweakasystem
24

25. GraphEngine/Kernel-LevelDSM
LITE-GraphisbasedonPowerGraph
LITE-Graph-DSMisasimilargraphengineontopofLITEDSM
ComparetheruntimeofPageRankontheTwitterdataset(1M
vertices,1Bdirectededges)withPowerGraphandGrappa(PGAS
system)
LITE-Graphperformsbetterthanothersystems
LITE-Graph-DSMstillperformssimilarorbetterthanother
systems 25

26. Conclusion
ThispaperproposesLITE,aLocalIndirectionTiErintheOSto
virtualizeandmanageRDMAfordatacenterapplications.
LITEsolvesthreekeyissuesofnativeRDMAforthedatacenter
environments.
Mismatchedabstraction
Unscalableperformance
Lackofresourcemanagement
LITEdemonstratesthatusingakernel-levelindirectionlayercan
preservenativeRDMAʼsgoodperformance,whilesolvingits
issues.
TheevaluationoffourdatacenterapplicationsshowsthatLITEis
botheasytouseandperformswell.
26

27. Mythought(1)
Thispaperisexcellentsystemsoftwarepaperandeverythingis
beautifullyconstructedinaconsistentway
Clearfyingthemotivationandissueswithconcreteexamples
Reasonabledesignchoicewithdetailedexplanation
Plentifulevaluationtoassisttheirclaims
DeveloppingfourapplicationswithLITEisowesome
Icannotfindmajorflawofthepapersofar
ButthenodemanagementsystemissomewhatcomplexsoI
amnotsuprisedifitleadstothescalabilityissue(thereisonly
upto8nodesevaluationinthispaper)
Perhapstherivalsystemsmaybealittlebitweak(?)
27

28. Mythought(2)
JustcurioushowtousetheLITEAPIstoconstructsystems
ItdoesnotseemtobeasimpleSPMDmodel
ThedocumentofAPIsexistbutnotmanyapplicationcodes
arepublic
ThesamegrouppublishStorm 1 ,whichinsiststhatthesame
problemnolongerapplywhenusingnewerRDMANICs
ThedifferencebetweenRNICgenerationsisalsointeresting
tome
1 Novakovic,S.,Shan,Y.,Kolli,A.,Cui,M.,Zhang,Y.,Eran,H.,Pismenny,B.,Liss,L.,Wei,M.,Tsafrir,D.,&Aguilera,M.
(2019).StoRM:Afasttransactionaldataplaneforremotedatastructures.SYSTOR2019-Proceedingsofthe12thACM
InternationalSystemsandStorageConference,97–108.https://doi.org/10.1145/3319647.3325827
28

29. Mythought(3)
Iguessthattheresearchprogressesinthisway
Focusonthewell-knownproblemoflimitationofRNICSRAM
ComeupwiththesimpleideaofonloadingtheburdenofRNICto
thekernelbyregisteringawholememoryasoneMRwithphysical
addresses(andfindverbsAPI!)
ThinkaboutthebasicdesignofLITEforachivingtheindirection
onlyatthelocalsideandkeepingperformance
DevelopconcreteAPIswhichareusefulforimplementing
applications(suchasRPCandmemoryoperations)
Makeadetailedplanforimplementingapplications(they
measuredhowmanystudentdaysittakesforeachapplicationto
becompleted!)
FindthismechanismalsoworksforkeepingQoSanddothe
relatedevaluation 29