Your SlideShare is downloading. ×
0
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Redesigning Xen Memory Sharing (Grant) Mechanism

5,776

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,776
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
60
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Redesigning Xen Memory Sharing (Grant) Mechanism<br />Kaushik Kumar Ram (Rice University)<br />Jose Renato Santos (HP Labs)<br />Yoshio Turner (HP Labs)<br />Alan L. Cox (Rice University)<br />Scott Rixner (Rice University)<br />Xen Summit<br />Aug 2nd 2011<br />
  • 2. This talk…<br />Will make a case for redesigning the grant mechanism to achieve better I/O performance and for other benefits<br />Will propose an alternate design for the grant mechanism<br />Will present an evaluation of a prototype of this new design<br />8/2/11<br />1<br />Xen Summit 2011<br />
  • 3. Outline<br />Motivation<br />Proposal <br />A grant reuse scheme<br />Evaluation<br />Conclusion<br />8/2/11<br />2<br />Xen Summit 2011<br />
  • 4. Traditional I/O Virtualization<br />frontend<br />backend<br />Guest Domain<br />Driver Domain<br />Guest domain –driver domain<br />memory sharing<br />(grant mechanism)<br />Physical Driver<br />Xen Hypervisor<br />Driver domain –device memory sharing (IOMMU)<br />Device<br />Hardware<br /> Two level memory sharing<br />8/2/11<br />3<br />Xen Summit 2011<br />
  • 5. Direct Device Assignment<br />Guest Domain<br />Guest domain –device memory sharing (IOMMU)<br />Physical Driver<br />Xen Hypervisor<br />Device<br />Hardware<br /> One level memory sharing<br />8/2/11<br />4<br />Xen Summit 2011<br />
  • 6. Grant Mechanism<br />Controlled memory sharing between domains<br />Source domain can share its memory pages with a specific destination domain<br />Destination domain can validate that the shared pages belong to the source domain via the hypervisor<br />8/2/11<br />5<br />Xen Summit 2011<br />
  • 7. Creating Shared Memory using Grant Mechanism<br />Source Domain<br />Creates grant entry in grant table<br />Destination Domain<br />Issues grant hypercall<br />Hypervisor validates grant and maps source page <br />Destination Domain<br />Source Domain<br />grant reference<br />Grant Table<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />6<br />Xen Summit 2011<br />
  • 8. Revoking Shared Memory using Grant Mechanism<br />Destination Domain<br />Issues grant hypercall<br />Hypervisor unmaps page<br />Source Domain<br />Deletes grant entry from grant table<br />Source Domain<br />Destination Domain<br />Grant Table<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />7<br />Xen Summit 2011<br />
  • 9. IOMMU<br />To safely share memory with I/O devices<br />Maintain memory isolation between domains (direct device assignment)<br />Protect against device driver bugs<br />Protect against attacks exploiting device DMA<br />Memory<br />IOMMU Table<br />I/O Device<br />Machine Address<br />I/O Address<br />8/2/11<br />8<br />Xen Summit 2011<br />
  • 10. Sharing Memory via IOMMUs<br />Para-virtualized I/O :-<br />Fine-grained sharing<br />IOMMU mapping setup during grant map hypercall and revoked during grant unmaphypercall<br />Direct Device Assignment :-<br />Only coarse-grained sharing<br />8/2/11<br />9<br />Xen Summit 2011<br />
  • 11. High Memory Sharing Overhead<br />I/O page is shared only for the duration of a single I/O<br />High cost of grant hypercalls and mapping/unmapping incurred in driver domain on every I/O operation<br />8/2/11<br />10<br />Xen Summit 2011<br />
  • 12. Reuse Scheme to Reduce Overhead<br />Take advantage of temporal and/or spatial locality in use of I/O pages<br />Reuse grants when I/O pages are reused<br />Reduce grant issue and revoke operations<br />Reduce grant hypercalls and mapping/unmapping overheads in driver domain <br />8/2/11<br />11<br />Xen Summit 2011<br />
  • 13. Reuse Under Existing Grant Mechanism<br />Grant reuse scheme requires –<br />Not revoking grants after every I/O operation<br />Persistent mapping of guest I/O pages in driver domain<br />Grants can be revoked when pages re-purposed for non-I/O operations<br />Today, there exists no way for guest domain to revoke access when its page is still mapped in driver domain <br />8/2/11<br />12<br />Xen Summit 2011<br />
  • 14. Goals<br />Enable reuse to reduce memory sharing related overheads during I/O<br />Support unilateral revocation of grants by source domains<br />Support an unified interface to share memory with I/O devices via IOMMUs<br />8/2/11<br />13<br />Xen Summit 2011<br />
  • 15. Proposal<br />Move the grant related hypercalls to the guest domains<br />Guest domains directly interact with the hypervisor to issue and revoke grants<br />Guest Domain<br />Driver Domain<br />Grant Table<br />Hypercall<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />14<br />Xen Summit 2011<br />
  • 16. Redesigned Grant Mechanism1. Initialization<br />INIT1 hypercall (para-virtualized I/O only)<br />Registers a virtual address range <br />Base address(es) and size<br />INIT2 hypercall<br />Provides a “device_id”<br />Returns the size of the “grant address space” <br />0 – size of address range<br />Guest Domain<br />Driver Domain<br />INIT2<br />Hypercall<br />INIT1<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />15<br />Xen Summit 2011<br />
  • 17. Grant (I/O) Address Space<br />8/2/11<br />Xen Summit 2011<br />16<br />0x20000<br />0x10000<br />Size of address range<br />0x40000<br />0x10000<br />0x0<br />Grant address space<br />0x30000<br />Driver domain virtual address space (page table)<br />I/O virtual address space (IOMMU table)<br />
  • 18. Redesigned Grant Mechanism2. Creating Shared Memory<br />Guest Domain :<br />Picks a “grant reference”<br />Offset within grant address space<br />Issues grant MAP hypercall<br />Hypervisor validates grant and maps guest page<br />Driver Domain :<br />Translates grant reference into virtual address and I/O address<br />Guest Domain<br />Driver Domain<br />grant reference<br />MAP<br />Hypercall<br />Xen Hypervisor<br />Setup IOMMU mapping<br />Hardware<br />8/2/11<br />17<br />Xen Summit 2011<br />
  • 19. Grant Mapping<br />8/2/11<br />Xen Summit 2011<br />18<br />0x20000<br />Grant reference<br />0x10000<br />0x40000<br />0x7000<br />0x10000<br />0x0<br />Grant address space<br />0x30000<br />Driver domain virtual address space (page table)<br />I/O virtual address space (IOMMU table)<br />
  • 20. Redesigned Grant Mechanism2. Creating Shared Memory<br />Guest Domain :<br />Picks a “grant reference”<br />Offset within grant address space<br />Issues grant MAP hypercall<br />Hypervisor validates grant and maps guest page<br />Driver Domain :<br />Translates grant reference into virtual address and I/O address<br />Guest Domain<br />Driver Domain<br />grant reference<br />MAP<br />Hypercall<br />Xen Hypervisor<br />Setup IOMMU mapping<br />Hardware<br />8/2/11<br />19<br />Xen Summit 2011<br />
  • 21. Grant Mapping<br />8/2/11<br />Xen Summit 2011<br />20<br />0x20000<br />Grant reference<br />0x10000<br />0x17000<br />0x40000<br />0x10000<br />0x37000<br />0x0<br />Grant address space<br />0x30000<br />Driver domain virtual address space (page table)<br />I/O virtual address space (IOMMU table)<br />
  • 22. Redesigned Grant Mechanism3. Revoking Shared Memory<br />Guest Domain :<br />Issues grant UNMAP hypercall<br />Provides grant reference<br />Hypervisor unmaps page<br />Guest Domain<br />Driver Domain<br />UNMAP<br />Hypercall<br />Xen Hypervisor<br />Remove IOMMU mapping<br />Hardware<br />8/2/11<br />21<br />Xen Summit 2011<br />
  • 23. Unilateral Revocation<br />Guest domains can revoke grants any time by issuing grant UNMAP hypercall<br />No driver domain participation required<br />Safe to revoke grants even when the I/O pages are in use <br />Since corresponding IOMMU mappings are also removed<br />8/2/11<br />22<br />Xen Summit 2011<br />
  • 24. Unified Interface<br /> Grant hypercall interface can be invoked from the Guest DMA library <br />Guest Domain <br />SRIOV VF Driver<br />netfront<br />DMA Library<br />Xen Hypervisor<br />Hardware<br />IOMMU<br />8/2/11<br />23<br />Xen Summit 2011<br />
  • 25. Grant Reuse<br />Take advantage of temporal and/or spatial locality in use of I/O pages<br />Reuse grants when I/O pages are reused<br />Reuse grants across multiple I/O operations<br />Guest domain issues grant<br />Driver domain uses I/O page for multiple I/O operations<br />Guest domain revokes grant<br />Guest domains can implement any scheme to reuse grants <br />Relax safety constraints<br />Security vs performance trade-off<br />Shared mappings, delayed invalidations, optimistic tear-down etc.<br />8/2/11<br />24<br />Xen Summit 2011<br />
  • 26. A Grant Reuse Scheme<br />Security compromise – prevents corruption of non-I/O pages<br />Policy – Never share a non-I/O read-write page<br />Receive read-write sharing <br />Allocate I/O buffers from a dedicated pool<br />E.g. slab cache in Linux<br />Revoke grant when pages are reaped from pool<br />I/O buffer pool also promotes temporal locality<br />Transmit read-only sharing<br />Persistent sharing<br />Grants revoked only when there are no more grant references available (or keep it mapped always)<br />8/2/11<br />25<br />Xen Summit 2011<br />
  • 27. Evaluation - Setup and Methodology<br />Server Configuration<br />HP Proliant BL460c G7 Blade server<br />Intel Xeon X5670 – 6 CPU cores<br />32 GB RAM<br />2 embedded 10 GbE ports<br />Domain Configuration<br />Domain0<br />linux 2.6.32.40 pvops kernel and 1 GB memory<br />Driver Domain<br />linux-2.6.18.8-xen0 (modified) and 512 MB memory<br />Guest Domains<br />linux-2.6.18.8-xenU (modified) and 512 MB memory<br />Driver and guest domains configured with one VCPU each (pinned)<br />Netperf TCP Streaming tests<br />8/2/11<br />26<br />Xen Summit 2011<br />
  • 28. Evaluation - Transmit Results<br />8/2/11<br />27<br />Xen Summit 2011<br /><ul><li>mapcount() logic significantly affects performance (baseline with IOMMU)</li></li></ul><li>Evaluation - Receive Results<br />8/2/11<br />28<br />Xen Summit 2011<br /><ul><li>No IOMMU overhead during RX
  • 29. Driver domain bottleneck (Baseline)</li></li></ul><li>Evaluation – Inter-guest Results<br />8/2/11<br />29<br />Xen Summit 2011<br /><ul><li>Driver domain bottleneck (Baseline)</li></li></ul><li>Discussion<br />Supporting multiple mappings in driver domain (e.g. block tap interface)<br />Driver domain can register address ranges from multiple address spaces<br />Or use hardware-assisted memory virtualization<br />Cannot support unilateral revocation without IOMMUs<br />Cannot revoke grants to in-use pages<br />8/2/11<br />30<br />Xen Summit 2011<br />
  • 30. Conclusions<br />Made a case for redesigning the grant mechanism<br />Enable grant reuse<br />Support unilateral revocations<br />Support an unified interface to program IOMMUs<br />Proposed an alternate design where the source domain interacts directly with the hypervisor<br />Implemented and evaluated a reuse scheme<br />8/2/11<br />31<br />Xen Summit 2011<br />

×