Your SlideShare is downloading. ×
XS Oracle 2009 Networking 10gig
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

XS Oracle 2009 Networking 10gig

481
views

Published on

Jose Renato Santos: Achieving 10GB/s Paravirtualized

Jose Renato Santos: Achieving 10GB/s Paravirtualized

Published in: Technology, Business

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
481
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. A c h ie v in g 1 0 G b /s U s in g X e n P a r a -v ir tu a liz e d N e tw o r k D r iv e r s K a u s h ik K u m a r R a m *, J. R enato S antos + , Y o s h io T u r n e r + , A la n L . C o x *, S c o tt R ix n e r * + HP Labs *R ic e U n iv e r s ity © 2 0 0 8 H e w le tt -P a c k a r d D e v e lo p m e n t C o m p a n y , L .P . T h e in f o r m a tio n c o n t a in e d h e r e in is s u b je c t to c h a n g e w it h o u t n o t ic e
  • 2. X e n P V D r iv e r o n 1 0 G ig N e tw o r k s T hroug hput on a s ing le T C P c onnec tion (netperf) Xen Linux 10 9 8 7 Rate (Gb/s) 6 5 4 3 2 1 0 RX TX TX sendfile • F o c u s o f th is ta lk : R X 2 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 3. N e tw o r k P a c k e t R e c e p tio n in X e n P o s t g ra nt o n 7 P us h into the 1 I /O c ha nnel netw ork s ta c k I/O D r iv e r D o m a in G u e s t D o m a in C hannel M ec ha nis m s to B ackend F r g r te n on 4 reduc e driver D r iv e r d D r iv e r dom a in c os t: dem ux B r id g e 5 g ra nt U s e o f M ulti-queue • c opy N IC P h y s ic a l − A v o id d a ta c o p y D r iv e r − P a c k e t d e m u ltip le x in 6 event h a rd w a re IR Q DM A 3 2 Xen G ra nt R eus e • M ec ha nis m − R e d u c e c o s t o f g ra n t N IC H a rd w a re o p e r a tio n s In c o m in g P kt 3 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 4. U s in g M u lti-Q u e u e N IC s P o s t g ra nt o n 1 P us h into the 9 I /O c ha nnel netw o rk s ta c k U nM a p I/O 7 D r iv e r G u e s t D o m a in buffer C h a n n e ls D o m a in B a c k e n d F r ognrte n D r iv e r d D r iv e r 3 M ap • A d v a n ta g e o f po s t buf o n buffer dev queue m u lti-q u e u e 2 • A v o id d a ta c o p y P h y s ic a l DM A 5 D r iv e r O ne R X queue • A v o id s o ftw a r e p e8r g u e s t b r id g e 6 event IR Q Xen guest MAC addr In c o m in g M Q N IC H a rd w a re 4 dem ux P kt 4 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 5. P e r fo r m a n c e Im p a c t o f M u lti-q u e u e D river D om a in C P U C os t 10000 • S a v in g s d u e to Xen 9000 m u ltiq u e u e Xen grant 8000 Linux other • g ra n t c o p y 7000 User copy 6000 Cycles/Packet Linux grant • b r id g e 5000 mm mem* • M ost of 4000 bridge 3000 r e m a in in g c o s t network 2000 netback • g r a n t h y p e r c a lls 1000 driver (g r a n t + x e n 0 fu n c tio n s ) Current Xen Multi-queue 5 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 6. U s in g G r a n ts w ith M u lti-q u e u e N IC 2 us e pa g e fo r I /O • M u lti-q u e u e r e p la c e s D r iv e r D o m a in G u e s t D o m a in o n e g r a n t h y p e r c a ll B ackend F rgorn te n d (c o p y ) w ith tw o D r iv e r D r iv e r h y p e r c a lls (m a p /u n m a p ) P h y s ic a l D r iv e r • G r a n t h y p e r c a lls a r e 1 e x p e n s iv e 3 m a p g ra nt unm a p g ra nt Xen hyperc a ll hyperc a ll • M a p /u n m a p c a lls fo r e v e r y I/O o p e r a tio n N IC 6 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 7. R e d u c in g G r a n t C o s t • G ra n t R e u s e − D o n o t r e v o k e g r a n t a fte r I/O is c o m p le te d − K e e p b u ffe r p a g e o n a p o o l o f u n u s e d I/O p a g e s − R e u s e a lr e a d y g r a n te d p a g e s a v a ila b le o n b u ffe r p o o l fo r fu tu r e I/O o p e r a tio n s − A v o id s m a p /u n m a p o n e v e r y I/O 7 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 8. R e v o k in g a G r a n t fo r w h e n th e P a g e is M a p p e d in D r iv e r D o m a in • G u e s t m a y n e e d to r e c la im I/O p a g e fo r o th e r u s e (e .g . m e m o r y p r e s s u r e o n g u e s t) • N e e d to u n m a p p a g e a t d r iv e r d o m a in b e fo r e u s in g it in g u e s t k e r n e l • T o p r e s e r v e m e m o r y is o la tio n (e .g . p r o te c t fr o m d r iv e r bugs) • N e e d h a n d s h a k e b e tw e e n fr o n te n d a n d b a c k e n d to re v o k e g ra n t • T h is m a y b e s lo w e s p e c ia lly if th e d r iv e r d o m a in is n o t r u n n in g 8 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 9. A p p r o a c h to A v o id H a n d s h a k e w h e n R e v o k in g G r a n ts • O b s e r v a tio n : N o n e e d to m a p g u e s t p a g e in to d r iv e r d o m a in w ith m u lti-q u e u e N IC • S o ftw a r e d o e s n o t n e e d to lo o k a t p a c k e t h e a d e r , s in c e d e m u x is p e r fo r m e d in th e d e v ic e • J u s t n e e d p a g e a d d r e s s fo r D M A o p e r a tio n • A p p r o a c h : R e p la c e g r a n t m a p h y p e r c a ll w ith a s h a r e d m e m o r y in te r fa c e to th e h y p e r v is o r • S h a r e d m e m o r y ta b le p r o v id e s tr a n s la tio n o f g u e s t g ra n t to p a g e a d d re s s • N o n e e d to u n m a p p a g e w h e n g u e s t n e e d s to r e v o k e g r a n t (n o h a n d s h a k e ) 9 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 10. S o ftw a r e I/O T r a n s la tio n T a b le • S IO T T : s o ftw a r e I/O 4 1 7 S end g ra nt o ve r c re a te a g ra nt fo r U s e pa g e fo r I /O tr a n s la tio n ta b le I /O c ha nne l buffe r pa g e − In d e x e d b y g r a n t D r iv e r D o m a in G u e s t D o m a in event r e fe r e n c e B ackend F rgorn te n d pg − “p g ” fie ld : g u e s t p a g e D r iv e r D r iv e r 6 a d d r e s s & p e r m is s io n g et pa g e 2 S IO T − “u s e ” fie ld in d ic a te s if s et pg us e hyperc a ll T 5 g r a n t is in u s e b y d r iv e r P h y s ic a l s e t us e 9 c lea r d o m a in D r iv e r 8 res e t us e hyperc a ll 1 #pg 0 • s e t/c le a r h y p e r c a lls Xen 3 V a lida te, pin a nd 10 c hec k us e a nd − In v o k e d b y g u e s t upda te S I O T T re vo k e − S e t v a lid a te s g r a n t, p in s DM A p a g e , a n d w r ite s p a g e N IC a d d r e s s to S IO T T − C le a r r e q u ir e s th a t “u s e ”= 0 10 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 11. G ra n t R e u s e : A v o id p in /u n p in h y p e r c a ll o n e v e r y I/O U s e pa g e fo r I /O I /O B uffer D r iv e r D o m a in G u e s t D o m a in 1 c rea te g ra nt P ool event return buffer to po o l & k ee p g ra nt 4 B ackend F rgorn te n d reus e buffe r & g ra nt fro m po o l 5 D r iv e r D r iv e r return buffer to po o l & k ee p g ra nt 6 7 k erne l m em pres s ure S IO T T pg us e 10 revo k e g ra nt 2 s et 11 return pa g e to P h y s ic a l hyperc a ll k e rnel 8 D r iv e r c le a r hyperc a ll #pg 0 Xen 3 va lida te, pin a nd 9 upda te S I O T T c lea r S I O T N IC 11 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 12. P e r fo r m a n c e Im p a c t o f G r a n t R e u s e w / S o ftw a r e I/O T r a n s la tio n T a b le D river D om a in C P U C os t 10000 Xen 9000 Xen grant 8000 Linux other 7000 Cycle / Packet User copy 6000 Linux grant 5000 mm mem* 4000 bridge 3000 network 2000 netback 1000 driver 0 Current Xen Multi-queue SIOTT w/ grant reuse c o s t s a v in g : g r a n t h y p e r c a ll 12 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 13. Im p a c t o f o p tim iz a tio n s o n th r o u g h p u t C P U utiliza tio n D a ta ra te driver domain guest linux 10 120 9 8 100 7 CPU utilization (%) Rate (Gb/s) 80 6 5 60 4 3 40 2 20 1 0 0 current Xen multi-queue w/ Linux current Xen multi-queue w/ Linux grant reuse grant reuse • M u lti-q u e u e w / g r a n t r e u s e s ig n ific a n tly r e d u c e d r iv e r d o m a in c o s t • B o ttle n e c k s h ifts fr o m d r iv e r d o m a in to g u e s t • H ig h e r c o s t in g u e s t th a n in L in u x s till lim its th r o u g h p u t in X e n 13 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 14. A d d itio n a l o p tim iz a tio n s a t g u e s t fr o n te n d d r iv e r • L R O (L a r g e R e c e iv e O fflo a d ) s u p p o r t a t fr o n te n d − C o n s e c u t iv e p a c k e ts o n s a m e c o n n e c tio n c o m b in e d in to o n e la r g e p a c k e t − R e d u c e s c o s t o f p r o c e s s in g p a c k e t in n e tw o r k s ta c k • S o ftw a r e p r e fe tc h − P r e fe t c h n e x t p a c k e t a n d s o c k e t b u ffe r s tr u c t in to C P U c a c h e w h ile p r o c e s s in g c u r r e n t p a c k e t − R e d u c e s c a c h e m is s e s a t fr o n te n d • A v o id fu ll p a g e b u ffe r s − U s e h a lf-p a g e (2 K B ) b u ffe r s (M a x p k t s iz e is 1 5 0 0 b y te s ) − R e d u c e s T L B w o r k in g s e t a n d th u s T L B m is s e s 14 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 15. P e r fo r m a n c e im p a c t o f g u e s t fr o n te n d o p tim iz a tio n s G ues t D om a in C P U C os t 5000 4500 Xen 4000 Xen grant 3500 cycles/packet Linux other 3000 User copy 2500 Linux-grant mm 2000 mem 1500 network 1000 driver 500 0 multiqueue w/ frontend w/ frontend w/ reduced Linux grant reuse LRO prefetch buffer size (2KB) • O p t im iz a tio n s b r in g C P U c o s t in g u e s t c lo s e to n a tiv e L in u x • R e m a in in g c o s t d iffe r e n c e − H ig h e r c o s t in n e tfr o n t th a n in p h y s ic a l d r iv e r − X e n fu n c tio n s to s e n d a n d d e liv e r e v e n ts 15 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 16. Im p a c t o f a ll o p tim iz a tio n s o n th r o u g h p u t 10 9 8 7 Rate (gb/s) 6 5 4 3 2 1 0 o ptim ized o ptim ized c urrent P V D irec t I /O L inux P V drive r P V driver driver (1 g ues t) (1 g ues t) (2 g ues ts ) • M u lt iq u e u e w ith s o ftw a r e o p tim iz a tio n s a c h ie v e s th e s a m e th r o u g h p u t a s d ir e c t I/O ( ~8 G b /s ) • 2 o r m o r e g u e s ts a r e a b le to s a tu r a te 1 0 g ig a b it lin k 16 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 17. C o n c lu s io n • U s e o f m u lti-q u e u e s u p p o r t in m o d e r n N IC s e n a b le s h ig h p e r fo r m a n c e n e tw o r k in g w ith X e n P V D r iv e r s − A ttr a c tiv e a lte r n a tiv e to D ir e c t I/O • S a m e th r o u g h p u t, a lth o u g h w ith s o m e a d d itio n a l C P U c y c le s a t d r iv e r d o m a in • A v o id s h a r d w a r e d e p e n d e n c e in th e g u e s ts − L ig h t d r iv e r d o m a in e n a b le s s c a la b ility fo r m u ltip le g u e s ts • D r iv e r d o m a in c a n n o w h a n d le 1 0 G b /s d a ta r a te s • M u ltip le g u e s ts c a n le v e r a g e m u ltip le C P U c o r e s a n d s a tu r a te 1 0 g ig a b it lin k 17 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 18. S ta tu s • S ta tu s − P e r fo r m a n c e r e s u lts o b ta in e d o n a m o d ifie d n e t fr o n t /n e tb a c k im p le m e n ta tio n u s in g th e o r ig in a l N e tc h a n n e l1 p r o to c o l − C u r r e n tly p o r tin g m e c h a n is m s to N e tc h a n n e l2 • B a s ic m u lti-q u e u e a lr e a d y a v a ila b le o n p u b lic n e tc h a n n e l2 tr e e • A d d itio n a l s o ftw a r e o p tim iz a tio n s s till in d is c u s s io n w ith c o m m u n ity a n d s h o u ld b e in c lu d e d in n e tc h a n n e l2 s o m e tim e soon. • T h a n k s to − M itc h W illia m s a n d J o h n R o n c ia k fr o m In te l fo r p r o v id in g s a m p le s o f In te l N IC s a n d fo r a d d in g m u lti-q u e u e s u p p o r t o n th e ir d r iv e r − Ia n P r a tt, S te v e n S m ith a n d K e ir F r a s e r fo r h e lp fu l d is c u s s io n s 18 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9
  • 19. A c h ie v in g 1 0 G b /s U s in g X e n P a r a -v ir tu a liz e d N e tw o r k D r iv e r s K a u s h ik K u m a r R a m *, J. R enato S antos + , Y o s h io T u r n e r + , A la n L . C o x *, S c o tt R ix n e r * + HP Labs *R ic e U n iv e r s ity © 2 0 0 8 H e w le t t-P a c k a r d D e v e lo p m e n t C o m p a n y , L .P . T h e in fo r m a tio n c o n ta in e d h e r e in is s u b je c t to c h a n g e w ith o u t n o tic e 1
  • 20. X e n P V D r iv e r o n 1 0 G ig N e tw o r k s T hro ug hput o n a s ing le T C P c o nnec tio n (netperf) Xen Linux 10 9 8 7 Rate (Gb/s) 6 5 4 3 2 1 0 RX TX TX sendfile • F o c u s o f th is ta lk : R X 2 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 2
  • 21. N e tw o r k P a c k e t R e c e p tio n in X e n P o s t g ra nt o n 7 P us h into the 1 I /O c ha nnel netw o rk s ta c k I/O D r iv e r D o m a in G u e s t D o m a in Channel M ec ha nis m s to B ackend F r g r te n on 4 reduc e driver D r iv e r d D r iv e r do m a in c o s t: dem ux B r id g e 5 g ra nt U s e o f M ulti-queue • c o py N IC P h y s ic a l − A v o id d a ta c o p y D r iv e r − P a c k e t d e m u ltip le x in 6 event h a rd w a re IR Q DM A 3 2 Xen G ra nt R eus e • M ec ha nis m − R e d u c e c o s t o f g ra n t N IC H a rd w a re o p e r a tio n s I n c o m in g P kt 3 0 2 /2 2 /2 5 /29 0 9 0 5 00 0 X e n e n m u mit m itF – b e b 0 9 0 9 X su s m – e F 20 20 3
  • 22. U s in g M u lti-Q u e u e N IC s P o s t g ra nt on 1 P us h into the 9 I /O c ha nne l netw ork s ta c k U nM a p I /O 7 D r iv e r G u e s t D o m a in buffer C h a n n e ls D o m a in B a c k e n d F r ognrte n D r iv e r d D r iv e r 3 M ap • A d v a n ta g e o f pos t buf on buffer de v queue m u lti-q u e u e 2 • A v o id d a ta c o p y P h y s ic a l DM A 5 D r iv e r O ne R X queue • A v o id s o ftw a r e p e8r g u e s t b r id g e 6 event IR Q Xen guest MAC addr In c o m in g M Q N IC H a rd w a re 4 de m ux P kt 4 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 4
  • 23. P e r fo r m a n c e Im p a c t o f M u lti-q u e u e D river D om a in C P U C os t 10000 • S a v in g s d u e to Xen 9000 m u ltiq u e u e Xen grant 8000 Linux other • g ra n t c o p y 7000 User copy 6000 Cycles/Packet Linux grant • b r id g e 5000 mm mem* • M ost of 4000 bridge 3000 r e m a in in g c o s t network 2000 netback • g r a n t h y p e r c a lls 1000 driver (g r a n t + x e n 0 fu n c tio n s ) Current Xen Multi-queue 5 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 5
  • 24. U s in g G r a n ts w ith M u lti-q u e u e N IC 2 us e pa g e fo r I /O • M u lti-q u e u e r e p la c e s D r iv e r D o m a in G u e s t D o m a in o n e g r a n t h y p e r c a ll B ackend F rgorn te n d (c o p y ) w ith tw o D r iv e r D r iv e r h y p e r c a lls (m a p /u n m a p ) P h y s ic a l D r iv e r • G r a n t h y p e r c a lls a r e 1 e x p e n s iv e 3 m a p g ra nt unm a p g ra nt Xen hyperc a ll hyperc a ll • M a p /u n m a p c a lls fo r e v e r y I/O o p e r a tio n N IC 6 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 6
  • 25. R e d u c in g G r a n t C o s t • G ra n t R e u s e − D o n o t r e v o k e g r a n t a fte r I/O is c o m p le te d − K e e p b u ffe r p a g e o n a p o o l o f u n u s e d I/O p a g e s − R e u s e a lr e a d y g r a n te d p a g e s a v a ila b le o n b u ffe r p o o l fo r fu tu r e I/O o p e r a tio n s − A v o id s m a p /u n m a p o n e v e r y I/O 7 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 7
  • 26. R e v o k in g a G r a n t fo r w h e n th e P a g e is M a p p e d in D r iv e r D o m a in • G u e s t m a y n e e d to r e c la im I/O p a g e fo r o th e r u s e (e .g . m e m o r y p r e s s u r e o n g u e s t) • N e e d to u n m a p p a g e a t d r iv e r d o m a in b e fo r e u s in g it in g u e s t k e r n e l • T o p r e s e r v e m e m o r y is o la tio n (e .g . p r o te c t fr o m d r iv e r bugs) • N e e d h a n d s h a k e b e tw e e n fr o n te n d a n d b a c k e n d to re v o k e g ra n t • T h is m a y b e s lo w e s p e c ia lly if th e d r iv e r d o m a in is n o t r u n n in g 8 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 8
  • 27. A p p r o a c h to A v o id H a n d s h a k e w h e n R e v o k in g G r a n ts • O b s e r v a tio n : N o n e e d to m a p g u e s t p a g e in to d r iv e r d o m a in w ith m u lti-q u e u e N IC • S o ftw a r e d o e s n o t n e e d to lo o k a t p a c k e t h e a d e r , s in c e d e m u x is p e r fo r m e d in th e d e v ic e • J u s t n e e d p a g e a d d r e s s fo r D M A o p e r a tio n • A p p r o a c h : R e p la c e g r a n t m a p h y p e r c a ll w ith a s h a r e d m e m o r y in te r fa c e to th e h y p e r v is o r • S h a r e d m e m o r y ta b le p r o v id e s tr a n s la tio n o f g u e s t g r a n t to p a g e a d d r e s s • N o n e e d to u n m a p p a g e w h e n g u e s t n e e d s to r e v o k e g r a n t (n o h a n d s h a k e ) 9 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 9
  • 28. S o ftw a r e I/O T r a n s la tio n T a b le • S IO T T : s o ftw a r e I/O 4 1 7 S end g ra nt o ver c rea te a g ra nt fo r U s e pa g e fo r I /O tr a n s la tio n ta b le I /O c ha nnel buffer pa g e − In d e x e d b y g r a n t D r iv e r D o m a in G u e s t D o m a in event r e fe r e n c e B ackend F rgorn te n d pg − “p g ” fie ld : g u e s t p a g e D r iv e r D r iv e r 6 a d d r e s s & p e r m is s io n g et p a g e 2 S IO T − “u s e ” fie ld in d ic a te s if s et pg us e hyperc a ll T 5 g r a n t is in u s e b y d r iv e r P h y s ic a l s e t us e 9 c lea r d o m a in D r iv e r 8 re s et us e hyperc a ll 1 #pg 0 • s e t/c le a r h y p e r c a lls Xen 3 V a lid a te, pin a nd 10 c hec k us e a nd − In v o k e d b y g u e s t upda te S I O T T revo k e − S e t v a lid a te s g r a n t, p in s DM A p a g e , a n d w r ite s p a g e N IC a d d r e s s to S IO T T − C le a r r e q u ir e s th a t “u s e ”= 0 10 0 2 /225/2 50/2 9 0 9 0 /2 0 0 X e X esnu m um itm–it F–e F e2b0 0 9 0 9 n sm b 20 10
  • 29. G ra n t R e u s e : A v o id p in /u n p in h y p e r c a ll o n e v e r y I/O U s e pa g e fo r I /O I /O B uffer D r iv e r D o m a in G u e s t D o m a in 1 c rea te g ra nt Pool event return b uffer to p o o l & k eep g ra nt 4 B ackend F rgorn te n d 5 re us e b uffer & g ra nt fro m po o l D r iv e r D r iv e r 6 re turn buffer to po o l & k eep g ra nt k ernel m em pre s s ure 7 S IO T T pg us e 10 re vo k e g ra nt 2 s et 11 re turn pa g e to P h y s ic a l hyperc a ll k ernel 8 c lea r D r iv e r hyperc a ll #p g 0 Xen 3 va lida te, pin a nd 9 upda te S I O T T c le a r S I O T N IC 11 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 11
  • 30. P e r fo r m a n c e Im p a c t o f G r a n t R e u s e w / S o ftw a r e I/O T r a n s la tio n T a b le D river D om a in C P U C os t 10000 Xen 9000 Xen grant 8000 Linux other 7000 Cycle / Packet User copy 6000 Linux grant 5000 mm mem* 4000 bridge 3000 network 2000 netback 1000 driver 0 Current Xen Multi-queue SIOTT w/ grant reuse c o s t s a v in g : g r a n t h y p e r c a ll 12 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 12
  • 31. Im p a c t o f o p tim iz a tio n s o n th r o u g h p u t C P U utiliza tio n D a ta ra te driver domain guest linux 10 120 9 8 100 7 CPU utilization (%) Rate (Gb/s) 80 6 5 60 4 3 40 2 20 1 0 0 current Xen multi-queue w/ Linux current Xen multi-queue w/ Linux grant reuse grant reuse • M u lti-q u e u e w / g r a n t r e u s e s ig n ific a n tly r e d u c e d r iv e r d o m a in c o s t • B o ttle n e c k s h ifts fr o m d r iv e r d o m a in to g u e s t • H ig h e r c o s t in g u e s t th a n in L in u x s till lim its th r o u g h p u t in X e n 13 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 13
  • 32. A d d itio n a l o p tim iz a tio n s a t g u e s t fr o n te n d d r iv e r • L R O (L a r g e R e c e iv e O fflo a d ) s u p p o r t a t fr o n te n d − C o n s e c u tiv e p a c k e ts o n s a m e c o n n e c tio n c o m b in e d in to o n e la r g e p a c k e t − R e d u c e s c o s t o f p r o c e s s in g p a c k e t in n e tw o r k s ta c k • S o ftw a r e p r e fe tc h − P r e fe tc h n e x t p a c k e t a n d s o c k e t b u ffe r s tr u c t in to C P U c a c h e w h ile p r o c e s s in g c u r r e n t p a c k e t − R e d u c e s c a c h e m is s e s a t fr o n te n d • A v o id fu ll p a g e b u ffe r s − U s e h a lf-p a g e (2 K B ) b u ffe r s (M a x p k t s iz e is 1 5 0 0 b y te s ) − R e d u c e s T L B w o r k in g s e t a n d th u s T L B m is s e s 14 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 14
  • 33. P e r fo r m a n c e im p a c t o f g u e s t fr o n te n d o p tim iz a tio n s G ues t D om a in C P U C os t 5000 4500 Xen 4000 Xen grant 3500 cycles/packet Linux other 3000 User copy 2500 Linux-grant mm 2000 mem 1500 network 1000 driver 500 0 multiqueue w/ frontend w/ frontend w/ reduced Linux grant reuse LRO prefetch buffer size (2KB) • O p tim iz a tio n s b r in g C P U c o s t in g u e s t c lo s e to n a tiv e L in u x • R e m a in in g c o s t d iffe r e n c e − H ig h e r c o s t in n e tfr o n t th a n in p h y s ic a l d r iv e r − X e n fu n c tio n s to s e n d a n d d e liv e r e v e n ts 15 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 15
  • 34. Im p a c t o f a ll o p tim iz a tio n s o n th r o u g h p u t 10 9 8 7 Rate (gb/s) 6 5 4 3 2 1 0 o ptim ize d o ptim ize d c urrent P V D irec t I /O L inux P V driver P V driver driver (1 g ues t) (1 g ues t) (2 g ues ts ) • M u ltiq u e u e w ith s o ftw a r e o p tim iz a tio n s a c h ie v e s th e s a m e th r o u g h p u t a s d ir e c t I/O ( ~8 G b /s ) • 2 o r m o r e g u e s ts a r e a b le to s a tu r a te 1 0 g ig a b it lin k 16 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 16
  • 35. C o n c lu s io n • U s e o f m u lti-q u e u e s u p p o r t in m o d e r n N IC s e n a b le s h ig h p e r fo r m a n c e n e tw o r k in g w ith X e n P V D r iv e r s − A ttr a c tiv e a lte r n a tiv e to D ir e c t I/O • S a m e th r o u g h p u t, a lth o u g h w ith s o m e a d d itio n a l C P U c y c le s a t d r iv e r d o m a in • A v o id s h a r d w a r e d e p e n d e n c e in th e g u e s ts − L ig h t d r iv e r d o m a in e n a b le s s c a la b ility fo r m u ltip le g u e s ts • D r iv e r d o m a in c a n n o w h a n d le 1 0 G b /s d a ta r a te s • M u ltip le g u e s ts c a n le v e r a g e m u ltip le C P U c o r e s a n d s a tu r a te 1 0 g ig a b it lin k 17 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 17
  • 36. S ta tu s • S ta tu s − P e r fo r m a n c e r e s u lts o b ta in e d o n a m o d ifie d n e tfr o n t/n e tb a c k im p le m e n ta tio n u s in g th e o r ig in a l N e tc h a n n e l1 p r o to c o l − C u r r e n tly p o r tin g m e c h a n is m s to N e tc h a n n e l2 • B a s ic m u lti-q u e u e a lr e a d y a v a ila b le o n p u b lic n e tc h a n n e l2 tr e e • A d d itio n a l s o ftw a r e o p tim iz a tio n s s till in d is c u s s io n w ith c o m m u n ity a n d s h o u ld b e in c lu d e d in n e tc h a n n e l2 s o m e tim e soon. • T h a n k s to − M itc h W illia m s a n d J o h n R o n c ia k fr o m In te l fo r p r o v id in g s a m p le s o f In te l N IC s a n d fo r a d d in g m u lti-q u e u e s u p p o r t o n th e ir d r iv e r − Ia n P r a tt, S te v e n S m ith a n d K e ir F r a s e r fo r h e lp fu l d is c u s s io n s 18 0 2 /2 5 /2 0 0 9 X e n s u m m it – F e b 2 0 0 9 18