Reading Circle

1. UsingRDMAEfficientlyforKey‑Value
Services
ReadingCircleinTauraLaboratory
M1TakuyaFukuoka
January23,2020
1

2. Aboutthispaper
Author:AnujKalia,MichaelKaminskyandDavidG.Andersen
Thefirstauthoriscurrentlyafinal‑yearPhDstudentwithmany
publicationsrelatedtoRDMAandRPC.
ThispaperispublishedinSIGCOMM'14.
ItintroducesHERD,newdesignofRDMA‑basedkey‑value
systems.
Theauthorswroteanotherpaper[1],whichdescribeshowto
improvetheperformanceofHERD
Usingdoorbellmechanismtopostabatchofrequests
SourcecodeofHERDisavailablehere
[1]Kalia,A.,Kaminsky,M.,Labs,I.,Andersen,D.G.,Kaminsky,M.,&Andersen,D.G.(2016).DesignGuidelinesforHighPerformanceRDMA
Systems.InUSENIXATC’16(p.437).
2

3. Introduction
Thispaperfucusonanewkey‑valuestoresystemcalledHERD
withdetailedanaysistomakebestofRDMA
HERDadopttwounconventionaldecisionstoavoidroundtrips
ItdoesnotuseRDMAreads
ItusesamixofRDMAandtwo‑sidedverbs
Toverifytheidea,plentifulpreliminaryexperimentsareconducted
TheresultsshowHERDachivesthroughputsimilartonativeRDMA
readthroughput,whichisabout2xthanexistingsystems
HERDborrowsexistingback‑endkey‑valuestructuresfromMICA
[1]
Inthispresentation,Idon'tfocusonitsomuch.
[1]Lim,H.,Han,D.,Andersen,D.G.,&Kaminsky,M.(2014).MICA:Aholisticapproachtofastin‑memorykey‑valuestorage.InProceedings
ofthe11thUSENIXSymposiumonNetworkedSystemsDesignandImplementation,NSDI2014(pp.429–444).
3

4. Background
Key‑valuestore(KVS)
RDMA
4

5. Key‑valuestore(KVS)
KVSisoneofNoSQLswhichfitswellfortheeraofBigData
Redisandmemcachedisexamplesofsuchsystemusedfor
cachesofdatabases
Ifocusonin‑memoryKVSssystemwithasingleservermachine
5

6. Back‑endofkey‑valuestoresystem
(MICA)
Hereiskey‑valuedatastructuresofMICA,whichHERDborrows
ItisonlyoneexampleinmanydesignchoiceforKVSbackend
implementations
Theyarecomposeofhashindexand(circular)log
Youneedtoaccesstwice
Afigureisfromhttps://www.usenix.org/sites/default/files/conference/protected‑files/nsdi14_slides_lim.pdf 6

7. RDMA(RemoteDirectMemoryAccess)
One‑sidedoperationtoread/writeremotememorywhilethe
remoteCPUsareunawareofcommunication
ReducelatencyandCPUoverhead
Typicaloperations
RDMAread(READ)
RDMAwrite(WRITE)
(<=>two‑sidedoperation,notRDMA)
SEND
RECEIVE
7

8. RDMAPath
WRITEhavethreechancesofoptimizations
WhileREADneedsaroundtrip,optimizedWRITEonlyneedsone‑
waytrip 8

9. RDMApathofoptimizedWRITE
Afigureisfromhttps://conferences.sigcomm.org/sigcomm/2014/doc/slides/51.pdf
9

10. ExistingRDMA‑basedkey‑valuestore
DesigningKVSsystemsinwhicheachGETrequestrequiresonly1
smallRDMAreadisverydifficult
Typically,youneedtoaccesstheservertwicetogetavalue
Getapointerfromtheindex
Getavaluefromthelogs
ItisbelievedthatdoubleREADsisefficientbecauseitnever
involvesremoteCPUs
Isittruelyefficient?? 10

11. HERD'sdesign
11

12. HERD'smainidea
UseWRITEinsteadofREAD
ButthisdesignmustinvolveremoteCPUs
SoitshouldbeverifiedthatWRITEissuperiorthanREADnot
onlyinlatencyandbutalsointhroughput
Forresponse,useSENDinsteadofWRITE
Buttwo‑sidedverbsisknowntoperformworsethanone‑
sided
SotheremustbesomevirtueinSENDwhichWRITEdoesnot
have
Scalabilityforresponse
12

13. HowHERDworks
WRITEforrequestandSENDforresponse
ServerCPUsdolookupfromakeytoavalue
13

14. Preliminaryexperiments
WRITEperformsbetterthanREAD
Latency
InboundThroughput
OutboundThroughput
Forresponse,SENDperformsbetterthanWRITE
Scalability
14

15. LatencyofREADand(optimized)WRITE
Youcansee(ECHO/2)togetfull‑optimizedWRITElatency
ThesendercannotrecognizewhenoptimizedWRITEfinishes
(ECHO/2)performsbetterthanREAD
Itisduetothediffrenceofthelengthofcommunicationpath
15

16. RDMApath(again)
16

17. InboundthroughputofREADandWRITE
WRITErepresents(maybe)inlinedWRITE
(Nodiscriptionaboutinliningonthepaper)
WRITEperformsbetterthanREADininboundthroughput
Theauthorexplainthereasonas"inboundREADsrequireaDMA
readbytheRNICfollowedbyapackettransmission,whereas
inboundWRITEsonlyrequireaDMAwrite"
17

18. RDMApath(again)
(IfWRITEisnotinlined,thesamephenomenonshouldoccurand
performworse)
(TherearenodescriptionaboutinliningofWRITE,butIassume
inliningisadopted) 18

19. OutboundthroughputofREADandWRITE
Forsmallsizes,inlinedWRITEhashigherthroughputthanREAD
ItisexplainedbythedifferenceofpathsofinlinedWRITEand
READ
Forlargesizes,inlineWRITEhaslowerthroughputthanREAD
Thesharpdecreasesisexplainedbytheuseofwrite‑
combiningbuffersforPIOacceleration.
Inmyopinion,youcannotsaythatasingleWRITEperformsbetter
thanasingleREADfromthisresult
Ithinkthesizepayloadinoutboundwritetendstobelarger 19

20. PreliminaryExperiments
WRITEperformsbetterthanREAD
Latency
InboundThroughput
OutboundThroughput
Forresponse,SENDperformsbetterthanWRITE
Scalability
20

21. ScalabilityofWRITEandSEND
Inall‑to‑allcommunication
Clientprocessesselectaserverprocessatrandomandissue
aWRITEtoit
OutboundWRITE(Out‑write‑UC)doesnotscalewiththe
increasingnumberofclientprocesses
Thisisbecauseoftheoverheadofconnections
Thenumberofconnectionsis(N_client)x(N_server)
YoucansolveitbyusingUD(UnreliableDatagram)modewhich
onlySENDsupports 21

22. Evaluation
22

23. Experimentalenvironment
17clientmachinesrunupto3clientprocesseseach
oneservermachineruns6serverprocesses
EmulatingexistingRDMA‑basedKVSsystems
Pilaf[1]
FaRM[2]
Itonlysupportsfixedvaluesize
FaRM‑VAR
FaRMwhichcansupportvariablevaluesizes
Basically,GETsofPilafandFaRM‑VARusesmultipleREADs
[1]Mitchell,C.,Geng,Y.,&Li,J.(2013).Usingone‑sidedRDMAreadstobuildafast,CPU‑efficientkey‑valuestore.InProceedingsofthe
2013USENIXAnnualTechnicalConference,USENIXATC2013(pp.103–114).
[2]Dragojević,A.,Narayanan,D.,Castro,M.,&Hodson,O.(2014).FaRM:FastRemoteMemory.Proceedingsofthe11thUSENIXSymposium
onNetworkedSystemsDesignandImplementation(NSDI14),401–414. 23

24. Thoughputcomparion
HERDachieves26Mopswhichisslightlylargerthanthe
throughputofnativeRDMAreads
ThethroughputofPilafandFaRMisdeterminedbythethroughput
ofREADs
GETthroughputis9.9MopsinPilaf,whichneeds2.6READs
GETthroughputis17.2MopsinFaRM,whichisnearthemax
throughputwith288byteREADs 24

25. Throughputcomparisonwithvarioussizes
95%GET,5%PUT
HERDperformsbetterthanothersystems
25

26. Latencycomparsion
95%GET,5%PUT
Increasetheloadontheserverbyaddingmoreclientsuntilthe
serverissaturated.
FaRMprovidesthelowestlatencybecauseitrequiresonlyoneone
networkroundtripandnocomputationattheserver
However,FaRMsupportsonlyfixedvaluesize. 26

27. Scalability
Onemachinetorun6serverprocessesandremaining186
machinesforclientsprocesses
ThedeclineisduetothecachemissesinserverRNIC
Theauthorsclaimsthatitcanbesolvedwiththeintroductionof
DynamicallyConnectedTransportinthenewConnect‑IBcards
27

28. HERDCPUUse
DrawbackofHERDismoreCPUinvolvement
However,theresultshowsexistingsystemsalsohavesignificant
sourcesofCPUusage
Ialsowanttoseetheresultsinread‑intensive
28

29. Resistancetoskew
AworkloadwherethekeysaredrawnfromaZipfdistributionwhen
Zipfparameteris.99
95%GET,5%PUT,48bytesitems
HERD'sresistancetoskewcomesfromtwofactors
Back‑endMICAarchitecture
CPUcoressharetheRNICandloadbalancingworks
29

30. Conclusion
ThispaperproposedHERD,newdesignofKVSsystemwhichuse
asingleroundtripcombiningRDMAwriteforrequestandtwo‑
sidedsendforresponse
TheresultsshowHERDachivesthroughputsimilartonativeRDMA
readthroughput,whichisabout2xthanexistingsystems
ThisworkimprovedunderstandingofhowtouseRDMAtocostruct
notonlyforKVSsystemsbutalsoforfutureDRAM‑basedstorage
services
30

31. Mythought
Ilikethestory‑tellingofthispaperwithstrongtone
Ifeeltheauthors'spritstoinfluencetheusegeofRDMAinthe
world
However,Ithinkwesholudexaminewhetherthepreliminary
experimentstruelyfitforHERDsystem.Especially,Iwanttosee
theresultsofmicro‑benchmarksofthroughputinsuchasituation
All‑to‑allcommunication
Useallcoresinclientmachines
WRITEswithfull‑optimization
(ButImightthinkitisnotnecessarybecausetheyshowedHERD
performedtothelimitofRNIC)
IalsowanttoseethecomparisonbetweenHERDandstate‑of‑
the‑artKVSsystemswithnoRDMA
IthinkHERDcannotproveitsmeritswithlargevaluesizes 31