Reading Circle

1. PageFaultSupportforNetwork
Controllers
ReadingCircle
M2TakuyaFukuoka
May27,2020
1

2. AboutThisPaper
Author:IlyaLesokhin,HaggaiEran,ShacharRaindel,GuyShapiro,
SagiGrimberg,LiranLiss,MuliBen‑Yehuda,NadavAmitandDan
Tsafrir
ThecollaborationofTechnisonIsraelInstituteofTechnology,
MellanoxTechnologiesandVMwareResearch
PublishedinASPLOS'17
Proposedesignandimplementationofnetworkpagefault(NPF)
supportforInfiniBandandEthernetNICs
NPFsupportofInfiniBandisalreadyavailableforusason‑demand
paging(ODP)[1]
[1]https://community.mellanox.com/s/article/understanding‑on‑demand‑paging‑‑odp‑x
2

3. BriefSummary
DirectI/Odoesnotmakeuseoftwobenefitsofvirtualmemory
Simplifiedprogrammingmodel
Canonicalmemoryoptimization
Networkpagefaults(NPFs)supportcansolvetheproblem
HowtohandlereceiveNPFs(rNPFs)isachallengingtask
ThispaperproposesdesignandimplementationofNPFssupport
forInfiniBandandEthernetNICwithrNPFshandling
Theevaluationshowsproposedmethodsachievememory
efficienceandgoodperformance
3

4. ThreeBenefitsofVertualMemory
4

5. TheRiseofDirectNetworkI/O
OS‑providedabstractionandisolationmightdegradeperformance
CustomizedIOchannelssoftwarecanbe4‑20xmore
performantthanawell‑tunedPOSIX‑compliantprocess
Overheadscanbeavoidedifhardwaresupportabstractionand
isolationdirectly
5

6. ExamplesofDirectNetworkI/O
RemoteDMA(RDMA)
Single‑RootI/OVirautalization(SRIOV)
DataPlaneDevlopmentKit(DPDK)
https://ru.mellanox.com/products/adapter‑ethernet‑SW/RDMA‑RoCE‑Ethernet‑Network‑Efficiency
https://access.redhat.com/documentation/ja‑
jp/red_hat_enterprise_linux/6/html/virtualization_host_configuration_and_guest_installation_guide/chap‑
virtualization_host_configuration_and_guest_installation_guide‑sr_iov
http://www.mashfords.com/2018/08/21/dpdk‑data‑plane‑development‑kit‑for‑linux‑vms‑now‑generally‑available/
6

7. TheProblemofDirectNetworkNIC
NICscannotcopewithpagefaultswhichDMAdevicesexperience
Thereforelosingtwovirtualmemorybenefits
7

8. HowtoMakeUseofVirtualMemory
BenefitswithDMA
Twoexistingmethods
Staticallypinningtheentireaddressspacetophysicalmemory
Dynamicallypin/unpinningDMAtargetbuffers
Simplifiedprogramming
model
CanonicalMemory
Optimization
Static Yes No
Dynamic No Yes
NPF
(Proposed) Yes Yes
8

9. ExistingDMAPageFaultsSupport
Recently,PCIestandardprovidessimilarfunctionalityofNPFs
ButtheproblemofreceiveNPFs(rNPFs)causedbyincoming
externaldataisoutsidethescope
ThispaperproposesimplementationwithsupportofrNPFs
9

10. Implementation
▶InfiniBandPageFaultSupport
Basicidea:relyonRCprotocolofInfiniBand
EthernetPageFaultSupport
Basicidea:preparebackupring
10

11. InfiniBandImplementationofNPFs
ImplementtheNPFsbymodifyingthedriverandfirmwareof
InfiniBand
NPFflowisbasicallyasfollows
1.NICconsultstheIOMMUpagetablesandcheckifNPFisneeded
2.Thefarmware[1]detectsthisfaultandraisesanNPFinterrupt
3.Thedrives'sNPFinterrputhandlerqueriestheOS
4.ThedriverupdatestheIOMMUpagetableandinformsthe
farmware
[1]farmwareissoftwareinNIC 11

12. OverheadofNPFsinInfiniBand
MellanoxConnect‑IB56Gb/sInfiniBandNIC
Send4KBand4MBmessageswithnodiskaccess
Theoverheadisdominatedbymodifiedhardware(firmware) 12

13. HowtoHandlerNPFsinInfiniBand
Makeuseofreliableconnection(RC)protocolofInfiniBandwhich
assuresthemessagedelivery
RNRNACKandretransmission
Whenthereceiverisnotready,itsendareceiver‑not‑ready(RNR)
negativeacknowledgmentpacket(NACK)toaskthesendertowait
foraspecifiedtime
Retransmitpacketsincaseofpacket‑drops
13

14. Implementation
InfiniBandPageFaultSupport
Basicidea:relyonRCprotocolofInfiniBand
▶EthernetPageFaultSupport
Basicidea:preparebackupring
14

15. EthernetImplementationofNPFs
TCPalsoassuresthepacketdelivery.DoesithandlerNPFsinthe
samewayasInfiniBand?
TheanswerisNo
Thatisbecausepacketdropsareconsideredtoasignof
congestionandcauseTCPtoreducethetransmissionrate
Thecommunicationmayeventuallyhalt
15

16. HowmuchSufferfromThisProblem
Comparethethroughputofmemcached(key‑valuestore)
Dropping(whichrelyonTCPretransmission)suffersforabout60
secondsduringwhichitsthroughputisalmostzero
16

17. Solution:PrepareBackupRing
Ifapagefaultisencountered,thepacketiswrittentoasmall,
pinnedbackupringownedbytheIOprovider(OSorhypervisor)
17

18. Evaluation
MemoryUtilization
Key‑valuestore
Storagesystem
OverheadofPinningandCopying
MPImicrobenchmark
HPCworkloadapplication
ProgrammingComplexity
What‑IfAnalysis
MorefrequentrNPFs 18

19. ExperimentalEnvironment
Ethernet
TwoDellPowerEdgeR210IIRackServers
8GB1333MHzmemory
One4‑coreIntelXeonE3‑1220CPUat3.10GHzrunningUbuntu13.10witha
Linux3.11.4
NPF‑supporting12GbpsprototypeNIC
InfiniBand
EightHPProLiantDL380pGen8servers
128GBmemory
One12‑coredualsocketXeonE5‑2697v2CPUat2.7GHzrunningRedHat7.0
withLinux3.10
NPF‑supporting56GbpsConnect‑IBNICs
19

20. MemoryUtilizationofKVS
Key‑valuestorewithEthernet
OnememcachedinstanceinoneVM
Memslapbenchmark(90%get,10%set,1KBvaluesbydefault)
Itispossibletorun4VMswithNPFswhileupto2VMswith
pinning 20

21. MemoryUtilizationofKVS
WithNPFs,youcanrunfourVMsinproductivemanner
21

22. MemoryUtilizationofKVSwithDynamic
Reallocation
Changethesizeofworkingsetafter50seconds
InoneVM,100MBto900MBwhileinotherVM,900MBto
100MB 22

23. MemoryutilizationofKVSwithDynamic
Reallocation
Withpinning,oneofthetwoinstancesalwayssuffer
Staticallyallocatedmemory(500MB)istoosmalltoholdits
workingset(900MB)
OverallperformanceofNPFsisbetterthanpinning
23

24. MemoryUtilizationofStorageSystem
AMachineemploystgt,aniSCSItargetimplementationthat
supportsiSCSIextensionsforRDMA(iSER)
Evaluatebandwidthbymeasuringtherandom512KBread
bandwidthusingfiobenchmark
Comparetwosettings:pinningandNPFs 24

25. MemoryUtilizationofStorageSystem
Inthepinnedconfiguration,thestaticmemoryallocationleavesa
smalleramountofmemorytothepagecache,whichcausesmore
cachemisses
WhatIthinkaboutisthatthebandwidthdiffersdependingonwhen
themeasurementisconducted 25

26. OverheadofPinningandCopying(1)
MicrobenchmarksinIntelMPIbenchmarkssuits
ComparebetweenthreeconfigurationsasMPIcommunication
backend
copying,pinning(astate‑of‑the‑artheuristicpin‑downcache)
andNPFs 26

27. OverheadofPinningandCopying(2)
BeffbenchmarkwhichmodelsrealHPCworkloads
NPFprovidessamebenefitaspinningwithouttheneedtopin
downmemory
27

28. ProgrammingComplexity
PortingthetgtstoragedemonwithNPFrequireonly40LOC(lines
ofcodes)
IntheMPImiddlewarelibrary,NPFenableseliminationof
thousandsofLOC
28

29. What‑ifAnalysis
SyntheticsituationwithmorefrequentrNPFs
MeasurebandwidthofsendoperationswithrNPFsatavariable
specifiedfrequency
BackupringsignificantlyimprovesperformancewithEthernet
IthinktheabsolutevalueoffrequencyshowsrNPFsdegrades
performancegreatly
29

30. Conclusion
ThispaperproposedNPFssupportforInfiniBandandEthernetNIC
InordertocopewithrNPFs
InfiniBandsupportreliesonRCprotocolwithRNRNACKand
retransmission
Ethernetsupportreliesonbackupring
TheevaluationsshowthattheirNPFssupport
Improvestheutilizationofmemory
Increasesoverallperformance
Simplifiestheprogrammodel
30

31. MyThoughts
Thispaperisagoodexampleofwell‑writtenpapersinsystemsoftware
Clearlyexplainsmotivationandrequirementwithmanycitations
fromawideperspective
Describebothoverallpicturesandthetechnicaldetailsseparetely
intheimplementationpart
Thegoodpointisthatitexplainsnotonlytheimplementation
itselfbutalsowhyoneway(orsimplesolution)doesnotwork
Plentifulevaluationassumingmanydifferentsituations
31

32. FromthePerspectiveofmyResearch
Provideproofthatproposedmethodworksinordinaryconditions
butnotinextremeconditions
TheymainlyevaluatesituationswithrarerNPFssuchasget‑
intensiveKVSandstorage
Atleast,what‑ifanalysisreportsfrequentrNPFcandegrade
performancegreatly
ProvidenoevaluationoflatencyofrNPFs
IwanttoseetheoverheadofrNPFsincludingretransmission
Iwanttoknowmoreimformationaboutwhatmethodswereused
withtheevalutionofRDMA
Isuspectmostaresendoperations
Howaboutone‑sidedoperationssuchasRDMA
READ/WRITE? 32

33. ThingsmentionedaboutRDMAOperation
ButinsomecasesRCdoesnotpermitRNRNACKsforRDMA.
Whenaninitiatorofaremotereadrequestencountersapagefault,
RCprovidesnowayforittoasktherespondertostop.Theonly
waytogetthesendertoretransmitisbyaskingittorewind,after
therNPFisresolved.Untilthen,wemustdropallincoming
packets.
Iamnotsurewhat"insomecases"specificallymeans
OnethingIgotfromthesesentencesisthattheimplementationof
NPFsforRDMAoperationsismoredifficult
ItisnowonderthattheusageofRDMAoperationsinOn‑demand
Pagingcausesunexpectedoverhead
33