0
Redesigning Xen Memory Sharing (Grant) Mechanism<br />Kaushik Kumar Ram (Rice University)<br />Jose Renato Santos (HP Labs...
This talk…<br />Will make a case for redesigning the grant mechanism to achieve better I/O performance and for other benef...
Outline<br />Motivation<br />Proposal <br />A grant reuse scheme<br />Evaluation<br />Conclusion<br />8/2/11<br />2<br />X...
Traditional I/O Virtualization<br />frontend<br />backend<br />Guest Domain<br />Driver Domain<br />Guest domain –driver d...
Direct Device Assignment<br />Guest Domain<br />Guest domain –device memory sharing (IOMMU)<br />Physical Driver<br />Xen ...
Grant Mechanism<br />Controlled memory sharing between domains<br />Source domain can share its memory pages with a specif...
      Creating Shared Memory       using Grant Mechanism<br />Source Domain<br />Creates grant entry in        grant table...
     Revoking Shared Memory       using Grant Mechanism<br />Destination Domain<br />Issues grant hypercall<br />Hyperviso...
IOMMU<br />To safely share memory with I/O devices<br />Maintain memory isolation between domains (direct device assignmen...
Sharing Memory via IOMMUs<br />Para-virtualized I/O :-<br />Fine-grained sharing<br />IOMMU mapping setup during grant map...
High Memory Sharing Overhead<br />I/O page is shared only for the duration of a single I/O<br />High cost of grant hyperca...
Reuse Scheme to Reduce Overhead<br />Take advantage of temporal and/or spatial locality in use of I/O pages<br />Reuse gra...
Reuse Under Existing Grant Mechanism<br />Grant reuse scheme requires –<br />Not revoking grants after every I/O operation...
Goals<br />Enable reuse to reduce memory sharing related overheads during I/O<br />Support unilateral revocation of grants...
Proposal<br />Move the grant related hypercalls to the guest domains<br />Guest domains directly interact with the hypervi...
       Redesigned Grant Mechanism1. Initialization<br />INIT1 hypercall (para-virtualized I/O only)<br />Registers a virtu...
Grant (I/O) Address Space<br />8/2/11<br />Xen Summit 2011<br />16<br />0x20000<br />0x10000<br />Size of address range<br...
       Redesigned Grant Mechanism2. Creating Shared Memory<br />Guest Domain :<br />Picks a “grant reference”<br />Offset ...
Grant Mapping<br />8/2/11<br />Xen Summit 2011<br />18<br />0x20000<br />Grant reference<br />0x10000<br />0x40000<br />0x...
       Redesigned Grant Mechanism2. Creating Shared Memory<br />Guest Domain :<br />Picks a “grant reference”<br />Offset ...
Grant Mapping<br />8/2/11<br />Xen Summit 2011<br />20<br />0x20000<br />Grant reference<br />0x10000<br />0x17000<br />0x...
       Redesigned Grant Mechanism3. Revoking Shared Memory<br />Guest Domain :<br />Issues grant UNMAP hypercall<br />Prov...
Unilateral Revocation<br />Guest domains can revoke grants any time by issuing grant UNMAP hypercall<br />No driver domain...
Unified Interface<br />	Grant hypercall interface can be invoked from the Guest DMA library <br />Guest Domain <br />SRIOV...
Grant Reuse<br />Take advantage of temporal and/or spatial locality in use of I/O pages<br />Reuse grants when I/O pages a...
A Grant Reuse Scheme<br />Security compromise – prevents corruption of non-I/O pages<br />Policy – Never share a non-I/O r...
Evaluation - Setup and Methodology<br />Server Configuration<br />HP Proliant BL460c G7 Blade server<br />Intel Xeon X5670...
Evaluation - Transmit Results<br />8/2/11<br />27<br />Xen Summit 2011<br /><ul><li>mapcount() logic significantly affects...
Driver domain bottleneck (Baseline)</li></li></ul><li>Evaluation – Inter-guest Results<br />8/2/11<br />29<br />Xen Summit...
Conclusions<br />Made a case for redesigning the grant mechanism<br />Enable grant reuse<br />Support unilateral revocatio...
Redesigning Xen Memory Sharing (Grant) Mechanism
Redesigning Xen Memory Sharing (Grant) Mechanism
Upcoming SlideShare
Loading in...5
×

Redesigning Xen Memory Sharing (Grant) Mechanism

5,814

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,814
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
60
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Redesigning Xen Memory Sharing (Grant) Mechanism"

  1. 1. Redesigning Xen Memory Sharing (Grant) Mechanism<br />Kaushik Kumar Ram (Rice University)<br />Jose Renato Santos (HP Labs)<br />Yoshio Turner (HP Labs)<br />Alan L. Cox (Rice University)<br />Scott Rixner (Rice University)<br />Xen Summit<br />Aug 2nd 2011<br />
  2. 2. This talk…<br />Will make a case for redesigning the grant mechanism to achieve better I/O performance and for other benefits<br />Will propose an alternate design for the grant mechanism<br />Will present an evaluation of a prototype of this new design<br />8/2/11<br />1<br />Xen Summit 2011<br />
  3. 3. Outline<br />Motivation<br />Proposal <br />A grant reuse scheme<br />Evaluation<br />Conclusion<br />8/2/11<br />2<br />Xen Summit 2011<br />
  4. 4. Traditional I/O Virtualization<br />frontend<br />backend<br />Guest Domain<br />Driver Domain<br />Guest domain –driver domain<br />memory sharing<br />(grant mechanism)<br />Physical Driver<br />Xen Hypervisor<br />Driver domain –device memory sharing (IOMMU)<br />Device<br />Hardware<br /> Two level memory sharing<br />8/2/11<br />3<br />Xen Summit 2011<br />
  5. 5. Direct Device Assignment<br />Guest Domain<br />Guest domain –device memory sharing (IOMMU)<br />Physical Driver<br />Xen Hypervisor<br />Device<br />Hardware<br /> One level memory sharing<br />8/2/11<br />4<br />Xen Summit 2011<br />
  6. 6. Grant Mechanism<br />Controlled memory sharing between domains<br />Source domain can share its memory pages with a specific destination domain<br />Destination domain can validate that the shared pages belong to the source domain via the hypervisor<br />8/2/11<br />5<br />Xen Summit 2011<br />
  7. 7. Creating Shared Memory using Grant Mechanism<br />Source Domain<br />Creates grant entry in grant table<br />Destination Domain<br />Issues grant hypercall<br />Hypervisor validates grant and maps source page <br />Destination Domain<br />Source Domain<br />grant reference<br />Grant Table<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />6<br />Xen Summit 2011<br />
  8. 8. Revoking Shared Memory using Grant Mechanism<br />Destination Domain<br />Issues grant hypercall<br />Hypervisor unmaps page<br />Source Domain<br />Deletes grant entry from grant table<br />Source Domain<br />Destination Domain<br />Grant Table<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />7<br />Xen Summit 2011<br />
  9. 9. IOMMU<br />To safely share memory with I/O devices<br />Maintain memory isolation between domains (direct device assignment)<br />Protect against device driver bugs<br />Protect against attacks exploiting device DMA<br />Memory<br />IOMMU Table<br />I/O Device<br />Machine Address<br />I/O Address<br />8/2/11<br />8<br />Xen Summit 2011<br />
  10. 10. Sharing Memory via IOMMUs<br />Para-virtualized I/O :-<br />Fine-grained sharing<br />IOMMU mapping setup during grant map hypercall and revoked during grant unmaphypercall<br />Direct Device Assignment :-<br />Only coarse-grained sharing<br />8/2/11<br />9<br />Xen Summit 2011<br />
  11. 11. High Memory Sharing Overhead<br />I/O page is shared only for the duration of a single I/O<br />High cost of grant hypercalls and mapping/unmapping incurred in driver domain on every I/O operation<br />8/2/11<br />10<br />Xen Summit 2011<br />
  12. 12. Reuse Scheme to Reduce Overhead<br />Take advantage of temporal and/or spatial locality in use of I/O pages<br />Reuse grants when I/O pages are reused<br />Reduce grant issue and revoke operations<br />Reduce grant hypercalls and mapping/unmapping overheads in driver domain <br />8/2/11<br />11<br />Xen Summit 2011<br />
  13. 13. Reuse Under Existing Grant Mechanism<br />Grant reuse scheme requires –<br />Not revoking grants after every I/O operation<br />Persistent mapping of guest I/O pages in driver domain<br />Grants can be revoked when pages re-purposed for non-I/O operations<br />Today, there exists no way for guest domain to revoke access when its page is still mapped in driver domain <br />8/2/11<br />12<br />Xen Summit 2011<br />
  14. 14. Goals<br />Enable reuse to reduce memory sharing related overheads during I/O<br />Support unilateral revocation of grants by source domains<br />Support an unified interface to share memory with I/O devices via IOMMUs<br />8/2/11<br />13<br />Xen Summit 2011<br />
  15. 15. Proposal<br />Move the grant related hypercalls to the guest domains<br />Guest domains directly interact with the hypervisor to issue and revoke grants<br />Guest Domain<br />Driver Domain<br />Grant Table<br />Hypercall<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />14<br />Xen Summit 2011<br />
  16. 16. Redesigned Grant Mechanism1. Initialization<br />INIT1 hypercall (para-virtualized I/O only)<br />Registers a virtual address range <br />Base address(es) and size<br />INIT2 hypercall<br />Provides a “device_id”<br />Returns the size of the “grant address space” <br />0 – size of address range<br />Guest Domain<br />Driver Domain<br />INIT2<br />Hypercall<br />INIT1<br />Hypercall<br />Xen Hypervisor<br />Hardware<br />8/2/11<br />15<br />Xen Summit 2011<br />
  17. 17. Grant (I/O) Address Space<br />8/2/11<br />Xen Summit 2011<br />16<br />0x20000<br />0x10000<br />Size of address range<br />0x40000<br />0x10000<br />0x0<br />Grant address space<br />0x30000<br />Driver domain virtual address space (page table)<br />I/O virtual address space (IOMMU table)<br />
  18. 18. Redesigned Grant Mechanism2. Creating Shared Memory<br />Guest Domain :<br />Picks a “grant reference”<br />Offset within grant address space<br />Issues grant MAP hypercall<br />Hypervisor validates grant and maps guest page<br />Driver Domain :<br />Translates grant reference into virtual address and I/O address<br />Guest Domain<br />Driver Domain<br />grant reference<br />MAP<br />Hypercall<br />Xen Hypervisor<br />Setup IOMMU mapping<br />Hardware<br />8/2/11<br />17<br />Xen Summit 2011<br />
  19. 19. Grant Mapping<br />8/2/11<br />Xen Summit 2011<br />18<br />0x20000<br />Grant reference<br />0x10000<br />0x40000<br />0x7000<br />0x10000<br />0x0<br />Grant address space<br />0x30000<br />Driver domain virtual address space (page table)<br />I/O virtual address space (IOMMU table)<br />
  20. 20. Redesigned Grant Mechanism2. Creating Shared Memory<br />Guest Domain :<br />Picks a “grant reference”<br />Offset within grant address space<br />Issues grant MAP hypercall<br />Hypervisor validates grant and maps guest page<br />Driver Domain :<br />Translates grant reference into virtual address and I/O address<br />Guest Domain<br />Driver Domain<br />grant reference<br />MAP<br />Hypercall<br />Xen Hypervisor<br />Setup IOMMU mapping<br />Hardware<br />8/2/11<br />19<br />Xen Summit 2011<br />
  21. 21. Grant Mapping<br />8/2/11<br />Xen Summit 2011<br />20<br />0x20000<br />Grant reference<br />0x10000<br />0x17000<br />0x40000<br />0x10000<br />0x37000<br />0x0<br />Grant address space<br />0x30000<br />Driver domain virtual address space (page table)<br />I/O virtual address space (IOMMU table)<br />
  22. 22. Redesigned Grant Mechanism3. Revoking Shared Memory<br />Guest Domain :<br />Issues grant UNMAP hypercall<br />Provides grant reference<br />Hypervisor unmaps page<br />Guest Domain<br />Driver Domain<br />UNMAP<br />Hypercall<br />Xen Hypervisor<br />Remove IOMMU mapping<br />Hardware<br />8/2/11<br />21<br />Xen Summit 2011<br />
  23. 23. Unilateral Revocation<br />Guest domains can revoke grants any time by issuing grant UNMAP hypercall<br />No driver domain participation required<br />Safe to revoke grants even when the I/O pages are in use <br />Since corresponding IOMMU mappings are also removed<br />8/2/11<br />22<br />Xen Summit 2011<br />
  24. 24. Unified Interface<br /> Grant hypercall interface can be invoked from the Guest DMA library <br />Guest Domain <br />SRIOV VF Driver<br />netfront<br />DMA Library<br />Xen Hypervisor<br />Hardware<br />IOMMU<br />8/2/11<br />23<br />Xen Summit 2011<br />
  25. 25. Grant Reuse<br />Take advantage of temporal and/or spatial locality in use of I/O pages<br />Reuse grants when I/O pages are reused<br />Reuse grants across multiple I/O operations<br />Guest domain issues grant<br />Driver domain uses I/O page for multiple I/O operations<br />Guest domain revokes grant<br />Guest domains can implement any scheme to reuse grants <br />Relax safety constraints<br />Security vs performance trade-off<br />Shared mappings, delayed invalidations, optimistic tear-down etc.<br />8/2/11<br />24<br />Xen Summit 2011<br />
  26. 26. A Grant Reuse Scheme<br />Security compromise – prevents corruption of non-I/O pages<br />Policy – Never share a non-I/O read-write page<br />Receive read-write sharing <br />Allocate I/O buffers from a dedicated pool<br />E.g. slab cache in Linux<br />Revoke grant when pages are reaped from pool<br />I/O buffer pool also promotes temporal locality<br />Transmit read-only sharing<br />Persistent sharing<br />Grants revoked only when there are no more grant references available (or keep it mapped always)<br />8/2/11<br />25<br />Xen Summit 2011<br />
  27. 27. Evaluation - Setup and Methodology<br />Server Configuration<br />HP Proliant BL460c G7 Blade server<br />Intel Xeon X5670 – 6 CPU cores<br />32 GB RAM<br />2 embedded 10 GbE ports<br />Domain Configuration<br />Domain0<br />linux 2.6.32.40 pvops kernel and 1 GB memory<br />Driver Domain<br />linux-2.6.18.8-xen0 (modified) and 512 MB memory<br />Guest Domains<br />linux-2.6.18.8-xenU (modified) and 512 MB memory<br />Driver and guest domains configured with one VCPU each (pinned)<br />Netperf TCP Streaming tests<br />8/2/11<br />26<br />Xen Summit 2011<br />
  28. 28. Evaluation - Transmit Results<br />8/2/11<br />27<br />Xen Summit 2011<br /><ul><li>mapcount() logic significantly affects performance (baseline with IOMMU)</li></li></ul><li>Evaluation - Receive Results<br />8/2/11<br />28<br />Xen Summit 2011<br /><ul><li>No IOMMU overhead during RX
  29. 29. Driver domain bottleneck (Baseline)</li></li></ul><li>Evaluation – Inter-guest Results<br />8/2/11<br />29<br />Xen Summit 2011<br /><ul><li>Driver domain bottleneck (Baseline)</li></li></ul><li>Discussion<br />Supporting multiple mappings in driver domain (e.g. block tap interface)<br />Driver domain can register address ranges from multiple address spaces<br />Or use hardware-assisted memory virtualization<br />Cannot support unilateral revocation without IOMMUs<br />Cannot revoke grants to in-use pages<br />8/2/11<br />30<br />Xen Summit 2011<br />
  30. 30. Conclusions<br />Made a case for redesigning the grant mechanism<br />Enable grant reuse<br />Support unilateral revocations<br />Support an unified interface to program IOMMUs<br />Proposed an alternate design where the source domain interacts directly with the hypervisor<br />Implemented and evaluated a reuse scheme<br />8/2/11<br />31<br />Xen Summit 2011<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×