• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Personal Pervasive Telemetry
 

Personal Pervasive Telemetry

on

  • 485 views

 

Statistics

Views

Total Views
485
Views on SlideShare
485
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Personal Pervasive Telemetry Personal Pervasive Telemetry Presentation Transcript

    • Achieving 10 Gb/s Using Xen Para-virtualized Network Drivers Kaushik Kumar Ram*, J. Renato Santos + , Yoshio Turner + , Alan L. Cox*, Scott Rixner* + HP Labs *Rice University
    • Xen PV Driver on 10 Gig Networks Xen summit – Feb 2009 02/25/2009
      • Focus of this talk: RX
      Throughput on a single TCP connection (netperf)
    • Network Packet Reception in Xen Xen summit – Feb 2009 02/25/2009 Driver Domain Guest Domain Xen Physical Driver Hardware NIC I/O Channel Incoming Pkt IRQ Bridge grant copy event DMA demux Push into the network stack Post grant on I/O channel
      • Mechanisms to reduce driver domain cost:
      • Use of Multi-queue NIC
        • Avoid data copy
        • Packet demultiplex in hardware
      • Grant Reuse Mechanism
        • Reduce cost of grant operations
      Backend Driver Frontend Driver 2 3 4 5 6 7 1 gr
    • Using Multi-Queue NICs Xen summit – Feb 2009 02/25/2009 Driver Domain Guest Domain Xen Physical Driver Hardware MQ NIC I/O Channels Incoming Pkt IRQ event Post grant on I/O channel Map buffer post buf on dev queue DMA UnMap buffer Push into the network stack
      • Advantage of multi-queue
        • Avoid data copy
        • Avoid software bridge
      One RX queue per guest guest MAC addr demux Backend Driver Frontend Driver 6 8 1 5 7 9 gr 3 2 4
    • Performance Impact of Multi-queue Xen summit – Feb 2009 02/25/2009
      • Savings due to multiqueue
        • grant copy
        • bridge
      • Most of remaining cost
        • grant hypercalls (grant + xen functions)
      Driver Domain CPU Cost
    • Using Grants with Multi-queue NIC Xen summit – Feb 2009 02/25/2009 Driver Domain Guest Domain Xen Physical Driver NIC map grant hypercall unmap grant hypercall
      • Multi-queue replaces one grant hypercall (copy) with two hypercalls (map/unmap)
      • Grant hypercalls are expensive
        • Map/unmap calls for every I/O operation
      use page for I/O Frontend Driver Backend Driver 1 gr 3 2
    • Reducing Grant Cost Xen summit – Feb 2009 02/25/2009
      • Grant Reuse
        • Do not revoke grant after I/O is completed
        • Keep buffer page on a pool of unused I/O pages
        • Reuse already granted pages available on buffer pool for future I/O operations
          • Avoids map/unmap on every I/O
    • Revoking a Grant for when the Page is Mapped in Driver Domain Xen summit – Feb 2009 02/25/2009
      • Guest may need to reclaim I/O page for other use (e.g. memory pressure on guest)
      • Need to unmap page at driver domain before using it in guest kernel
        • To preserve memory isolation (e.g. protect from driver bugs)
      • Need handshake between frontend and backend to revoke grant
        • This may be slow especially if the driver domain is not running
    • Approach to Avoid Handshake when Revoking Grants Xen summit – Feb 2009 02/25/2009
      • Observation: No need to map guest page into driver domain with multi-queue NIC
        • Software does not need to look at packet header, since demux is performed in the device
        • Just need page address for DMA operation
      • Approach: Replace grant map hypercall with a shared memory interface to the hypervisor
        • Shared memory table provides translation of guest grant to page address
        • No need to unmap page when guest needs to revoke grant (no handshake)
    • Software I/O Translation Table Xen summit – Feb 2009 02/25/2009 Driver Domain Guest Domain Xen Physical Driver NIC create a grant for buffer page Send grant over I/O channel set hypercall Validate, pin and update SIOTT clear hypercall get page reset use SIOTT #pg use pg 0 1 Use page for I/O DMA event check use and revoke set use
      • SIOTT: software I/O translation table
        • Indexed by grant reference
        • “ pg” field: guest page address & permission
        • “ use” field indicates if grant is in use by driver domain
      • set/clear hypercalls
        • Invoked by guest
        • Set validates grant, pins page, and writes page address to SIOTT
        • Clear requires that “use”=0
      Backend Driver Frontend Driver 2 9 6 8 1 3 7 10 gr 4 5 pg
    • Grant Reuse: Avoid pin/unpin hypercall on every I/O Xen summit – Feb 2009 02/25/2009 Driver Domain Guest Domain Xen Physical Driver NIC create grant set hypercall validate, pin and update SIOTT SIOTT #pg use pg 0 event I/O Buffer Pool reuse buffer & grant from pool return buffer to pool & keep grant kernel mem pressure clear hypercall clear SIOT return page to kernel return buffer to pool & keep grant revoke grant Use page for I/O Backend Driver Frontend Driver 2 1 3 5 4 8 9 7 11 gr gr 6 10
    • Performance Impact of Grant Reuse w/ Software I/O Translation Table Xen summit – Feb 2009 02/25/2009 cost saving: grant hypercall Driver Domain CPU Cost
    • Impact of optimizations on throughput Xen summit – Feb 2009 02/25/2009 Data rate CPU utilization
      • Multi-queue w/ grant reuse significantly reduce driver domain cost
      • Bottleneck shifts from driver domain to guest
      • Higher cost in guest than in Linux still limits throughput in Xen
    • Additional optimizations at guest frontend driver Xen summit – Feb 2009 02/25/2009
      • LRO (Large Receive Offload) support at frontend
        • Consecutive packets on same connection combined into one large packet
        • Reduces cost of processing packet in network stack
      • Software prefetch
        • Prefetch next packet and socket buffer struct into CPU cache while processing current packet
        • Reduces cache misses at frontend
      • Avoid full page buffers
        • Use half-page (2KB) buffers (Max pkt size is 1500 bytes)
        • Reduces TLB working set and thus TLB misses
    • Performance impact of guest frontend optimizations Xen summit – Feb 2009 02/25/2009
      • Optimizations bring CPU cost in guest close to native Linux
      • Remaining cost difference
        • Higher cost in netfront than in physical driver
        • Xen functions to send and deliver events
      Guest Domain CPU Cost
    • Impact of all optimizations on throughput Xen summit – Feb 2009 02/25/2009
      • Multiqueue with software optimizations achieves the same throughput as direct I/O ( ~8 Gb/s)
      • 2 or more guests are able to saturate 10 gigabit link
      current PV driver optimized PV driver (1 guest) optimized PV driver (2 guests) Direct I/O (1 guest) Linux
    • Conclusion Xen summit – Feb 2009 02/25/2009
      • Use of multi-queue support in modern NICs enables high performance networking with Xen PV Drivers
        • Attractive alternative to Direct I/O
          • Same throughput, although with some additional CPU cycles at driver domain
          • Avoids hardware dependence in the guests
        • Light driver domain enables scalability for multiple guests
          • Driver domain can now handle 10 Gb/s data rates
          • Multiple guests can leverage multiple CPU cores and saturate 10 gigabit link
    • Status Xen summit – Feb 2009 02/25/2009
      • Status
        • Performance results obtained on a modified netfront/netback implementation using the original Netchannel1 protocol
        • Currently porting mechanisms to Netchannel2
          • Basic multi-queue already available on public netchannel2 tree
          • Additional software optimizations still in discussion with community and should be included in netchannel2 sometime soon.
      • Thanks to
        • Mitch Williams and John Ronciak from Intel for providing samples of Intel NICs and for adding multi-queue support on their driver
        • Ian Pratt, Steven Smith and Keir Fraser for helpful discussions