Successfully reported this slideshow.

VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications

7

Share

Upcoming SlideShare
Cache & CPU performance
Cache & CPU performance
Loading in …3
×
1 of 32
1 of 32

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Related Audiobooks

Free with a 14 day trial from Scribd

See all

VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications

  1. 1. Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications Benoit Hudzia Sr. Researcher; SAP Research CEC Belfast (UK) Aidan Shribman Sr. Researcher; SAP Research Israel With the contribution of Roei Tell, Steve Walsh, Peter Izsak and Chaim Bendelac
  2. 2. Agenda Introduction Live Migration Memory Intensive Apps Pre-Copy Live Migration Post-Copy Live Migration Summary © 2012 SAP AG. All rights reserved. 2
  3. 3. Introduction The challenge of live migrating memory intensive applications
  4. 4. What is live migration? Moving of a guest VM running on host A (the source host) to host B (the destination host) without noticeable service disruption. Typically used for dynamic resource load balancing in cloud environments. So the faster to migrate and the less the service disruption the better. . © 2012 SAP AG. All rights reserved. 4
  5. 5. What is VM migration? CPU state and device driver state migration  MBs of information can be transferred while VM is stopped Memory state migration  GBs of information must (at least mostly) be transferred while VM is live  Requirement: is that memory can be migrated during guest execution Storage state migration  TBs of information (requiring minutes up to hours to move)  Recommendation: use shared storage (accessed via NFS, etc) NIC relocation  ARP announcement that IP address has been taken over  Requirement: both hosts reside on the same LAN © 2012 SAP AG. All rights reserved. 5
  6. 6. Memory Intensive Applications An application dominated by high rate of memory read/write accesses Why must memory be transferred mid execution? 16 GBs takes 12 seconds to migrate over 10 GbE (1.25 GB/s) interconnects 12 seconds downtime  would cause severe service disruption The challenge (of live migration implementations) is to transfer memory a mid high rates of memory shurn © 2012 SAP AG. All rights reserved. 6
  7. 7. Pre-Copy Live Migration Reducing the amount of page re-sends and the cost of page re-sends
  8. 8. Pre-Copy Memory Transfer Memory is transferred before VM relocation The problem is how to copy memory while it is re-dirtied over and over again by the guest VM Solved by first copying all the memory followed by intervals of copying newly dirtied pages until remaining state is small enough Implemented by most (all) hypervisors VMWare, Xen, Qemu Challenged by fast memory dirtying applications © 2012 SAP AG. All rights reserved. 8
  9. 9. Pre-Copy Live Migration (pre-migration) Guest VM Host A Host B Iterative Stop Commit Pre-migrate Reservation Pre-copy X and Rounds Copy Live on A Degraded on A Downtime Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 9
  10. 10. Pre-Copy Live Migration (reservation) Guest VM Guest VM Host A Host B Iterative Stop Commit Pre-migrate Reservation Pre-copy X and Rounds Copy Live on A Degraded on A Downtime Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 10
  11. 11. Pre-Copy Live Migration (pre-copy) Guest VM Guest VM Host A Host B Iterative Stop Commit Pre-migrate Reservation Pre-copy X and Rounds Copy Live on A Degraded on A Downtime Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 11
  12. 12. Pre-Copy Live Migration (stop and copy) Guest VM Guest VM } Host A Host B Iterative Stop Commit Pre-migrate Reservation Pre-copy X and Rounds Copy Live on A Degraded on A Downtime Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 12
  13. 13. Pre-Copy Live Migration (commit) Guest VM Host A Host B Iterative Stop Commit Pre-migrate Reservation Pre-copy X and Rounds Copy Live on A Degraded on A Downtime Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 13
  14. 14. LRU Page Reordering (reducing number of pages transferred) An LRU queue is used for tracking all historic page dirty events (i.e. application wrote to memory) QEMU (responsible for the live migration process) queries the Linux Kernel KVM hypervisor for dirty page events. It then inserts dirty page events into the LRU queue (after a random shuffle) Dirty pages (maintained in a separate LRU queue) are inserted according to historic LRU queue and transmitted head first LRU reordering may have an adverse effect for some workloads requiring a smarter cache (ARC?) © 2012 SAP AG. All rights reserved. 14
  15. 15. XBZRLE Delta Encoder (reducing cost of page resent) Cache page with highest dirtying rate Classic Pre-Copy during send operation If old page is found in cache  delta compress it; Else  transfer new Delta Compression Pre-Copy page uncompressed Implemented using an LRU cache configured to fit working set size © 2012 SAP AG. All rights reserved. 15
  16. 16. XBZRLE Delta Encoder - Example 0 1 2 3 4 5 6 7 Size Old ‘a’ ‘b’ ‘c’ ‘d’ ‘e’ ‘f’ ‘g’ ‘h’ 8 New ‘a’ ‘b’ ‘c’ 0x00 0x01 ‘f’ ‘g’ ‘h’ 8 XOR 0x00 0x00 0x00 ‘d’ ‘f’ 0x00 0x00 0x00 8 RLE 0x00 0x03 ‘d’ 0x01 ‘f’ 0x01 0x00 0x03 8 ZRLE 0x00 0x03 ‘d’ ‘f’ 0x00 0x03 6 XBRLE (XOR + RLE) presented in VEE 2011: Evaluation of Delta Compression Techniques for Efficient Live Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth XBZRLE (XOR + ZRLE) the current effort already released to QEMU community © 2012 SAP AG. All rights reserved. 16
  17. 17. XBZRLE Delta Encoder - Micro Benchmark Sparse Medium Dense Very Dense Jump 1111 701 203 121 Dirty 12 33 41 43 © 2012 SAP AG. All rights reserved. 17
  18. 18. Pre-copy Live Migration Evaluation Host: 1 core; 2 GB Host: 2 core; 8 GB; Bandwidth 240 Mbps Benchmark: synthetic dirtying 512 MB; 1 T Benchmark: appmembench dirtying 512 MB at 1 GB/s 8 T © 2012 SAP AG. All rights reserved. 18
  19. 19. Post-Copy Live Migration Fast live migration … but at a cost?
  20. 20. Post-Copy Memory Transfer Memory is transferred only after VM relocation The problem is how to ensure that VM performance is not degraded (after relocation) due to expensive network bound page faults Solved by using fast interconnects and improved page fault mechanism References: Yobusame PoC for KVM Linux (using char device user mode transfer) Challenged by fast memory reading applications © 2012 SAP AG. All rights reserved. 20
  21. 21. Post-Copy Live Migration (pre-migration) Guest VM Host A Host B Stop Page Pushing Commit Pre-migrate Reservation and 1 Copy Round Live on A Downtime Degraded on B Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 21
  22. 22. Post-Copy Live Migration (reservation) Guest VM Guest VM Host A Host B Stop Page Pushing Commit Pre-migrate Reservation and 1 Copy Round Live on A Downtime Degraded on B Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 22
  23. 23. Post-Copy Live Migration (stop and copy) Guest VM Guest VM Host A Host B Stop Page Pushing Commit Pre-migrate Reservation and 1 Copy Round Live on A Downtime Degraded on B Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 23
  24. 24. Post-Copy Live Migration (post-copy) Guest VM Guest VM Page fault Page push Host A Host B Stop Page Pushing Commit Pre-migrate Reservation and 1 Copy Round Live on A Downtime Degraded on B Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 24
  25. 25. Post-Copy Live Migration (commit) Guest VM Host A Host B Stop Page Pushing Commit Pre-migrate Reservation and 1 Copy Round Live on A Downtime Degraded on B Live on B Total Migration Time © 2012 SAP AG. All rights reserved. 25
  26. 26. Reducing Effects of Network Bound Page Faults Low latency RDMA page transfer protocol (reducing latency/cost of page faults)  Implemented fully in kernel mode OFED VERBS  Can use the fastest RDMA hardware available (IB, IWARP, RoCE) Demand pre-paging (pre-fetching) mechanism (reducing the number of page faults)  Currently only a simple fetching of pages surrounding page on which fault occured Full Linux MMU integration (reducing the system-wide effects/cost of page fault)  Enabling to perform page fault transparency (only pausing the requesting thread) Hybrid post-copy live migration (reducing the number of page faults)  Moving some of the pages to the destination in a bounded pre-copy phase prior to relocation © 2012 SAP AG. All rights reserved. 26
  27. 27. Evaluation Pre-Copy vs. Post-Copy Specification  Benchmark: Google Stress Application Test (SAT) with a 1 GB working set  Communication: softIWARP over 1 GbE  Guest VM: 2 x vCPU; 1 GB – 14 GB memory Results Roughly the total migration time between pre-copy to post-copy is 1:10 © 2012 SAP AG. All rights reserved. 27
  28. 28. Summary
  29. 29. Key takeaways Pre-Copy and Post-Copy optimizations enhace the support of live migrating memory intessive applications To be released as open source under GPLv2 and LGPL licenses to Qemu and Linux communities Developed by SAP Research Technology Infrastructure (TI) Practice © 2012 SAP AG. All rights reserved. 30
  30. 30. Thank you Benoit Hudzia; Sr. Researcher; SAP Research CEC Belfast benoit.hudzia@sap.com Aidan Shribman; Sr. Researcher; SAP Research Israel aidan.Shribman@sap.com
  31. 31. Legal Disclaimer The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP. This presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation and SAP's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This document is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions. © 2012 SAP AG. All rights reserved. 32

×