VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications

Pre-Copy and Post-Copy VM Live Migration
for Memory Intensive Applications
Benoit Hudzia Sr. Researcher; SAP Research CEC Belfast (UK)
Aidan Shribman Sr. Researcher; SAP Research Israel

With the contribution of Roei Tell, Steve Walsh, Peter Izsak and Chaim Bendelac

Agenda

Introduction
Live Migration
Memory Intensive Apps

Pre-Copy Live Migration

Post-Copy Live Migration

Summary

© 2012 SAP AG. All rights reserved. 2

Introduction
The challenge of live migrating memory intensive applications

What is live migration?

Moving of a guest VM running on host A (the source host) to
host B (the destination host) without noticeable service
disruption.
Typically used for dynamic resource load balancing in cloud
environments. So the faster to migrate and the less the service
disruption the better.
.


What is VM migration?

CPU state and device driver state migration
 MBs of information can be transferred while VM is stopped

Memory state migration
 GBs of information must (at least mostly) be transferred while VM is live
 Requirement: is that memory can be migrated during guest execution

Storage state migration
 TBs of information (requiring minutes up to hours to move)
 Recommendation: use shared storage (accessed via NFS, etc)

NIC relocation
 ARP announcement that IP address has been taken over
 Requirement: both hosts reside on the same LAN

Memory Intensive Applications

An application dominated by high rate of memory read/write accesses

Why must memory be transferred mid execution?
16 GBs takes 12 seconds to migrate over 10 GbE (1.25 GB/s) interconnects
12 seconds downtime  would cause severe service disruption

The challenge (of live migration implementations) is to transfer memory a mid high rates of
memory shurn


Pre-Copy Live Migration
Reducing the amount of page re-sends and the cost of page re-sends

Pre-Copy Memory Transfer

Memory is transferred before VM relocation

The problem is how to copy memory while it is re-dirtied over and
over again by the guest VM

Solved by first copying all the memory followed by intervals of
copying newly dirtied pages until remaining state is small enough

Implemented by most (all) hypervisors VMWare, Xen, Qemu

Challenged by fast memory dirtying applications


Pre-Copy Live Migration (pre-migration)

Guest VM

Host A Host B

Iterative Stop
Commit
Pre-migrate Reservation Pre-copy X and
Rounds Copy

Live on A Degraded on A Downtime Live on B

Total Migration Time


Pre-Copy Live Migration (reservation)

Guest VM Guest VM

Host A Host B

Iterative Stop
Commit
Rounds Copy




Pre-Copy Live Migration (pre-copy)

Guest VM Guest VM

Host A Host B

Iterative Stop
Commit
Rounds Copy




Pre-Copy Live Migration (stop and copy)

Guest VM Guest VM

}
Host A Host B

Iterative Stop
Commit
Rounds Copy




Pre-Copy Live Migration (commit)

Guest VM

Host A Host B

Iterative Stop
Commit
Rounds Copy




LRU Page Reordering (reducing number of pages transferred)

An LRU queue is used for tracking all historic page
dirty events (i.e. application wrote to memory)

QEMU (responsible for the live migration process)
queries the Linux Kernel KVM hypervisor for dirty
page events. It then inserts dirty page events into
the LRU queue (after a random shuffle)

Dirty pages (maintained in a separate LRU queue)
are inserted according to historic LRU queue and
transmitted head first

LRU reordering may have an adverse effect for
some workloads requiring a smarter cache (ARC?)

XBZRLE Delta Encoder (reducing cost of page resent)

Cache page with highest dirtying rate Classic Pre-Copy

during send operation
If old page is found in cache  delta
compress it; Else  transfer new
Delta Compression Pre-Copy
page uncompressed
Implemented using an LRU cache
configured to fit working set size


XBZRLE Delta Encoder - Example

0 1 2 3 4 5 6 7 Size
Old ‘a’ ‘b’ ‘c’ ‘d’ ‘e’ ‘f’ ‘g’ ‘h’ 8
New ‘a’ ‘b’ ‘c’ 0x00 0x01 ‘f’ ‘g’ ‘h’ 8
XOR 0x00 0x00 0x00 ‘d’ ‘f’ 0x00 0x00 0x00 8
RLE 0x00 0x03 ‘d’ 0x01 ‘f’ 0x01 0x00 0x03 8
ZRLE 0x00 0x03 ‘d’ ‘f’ 0x00 0x03 6

XBRLE (XOR + RLE) presented in VEE 2011: Evaluation of Delta Compression Techniques for
Efficient Live Migration of Large Virtual Machines by Benoit, Svard, Tordsson and Elmroth

XBZRLE (XOR + ZRLE) the current effort already released to QEMU community


XBZRLE Delta Encoder - Micro Benchmark

Sparse Medium Dense Very Dense
Jump 1111 701 203 121
Dirty 12 33 41 43


Pre-copy Live Migration Evaluation

Host: 1 core; 2 GB Host: 2 core; 8 GB; Bandwidth 240 Mbps
Benchmark: synthetic dirtying 512 MB; 1 T Benchmark: appmembench dirtying 512 MB at 1 GB/s 8 T

Post-Copy Live Migration
Fast live migration … but at a cost?

Post-Copy Memory Transfer

Memory is transferred only after VM relocation

The problem is how to ensure that VM performance is not degraded (after
relocation) due to expensive network bound page faults

Solved by using fast interconnects and improved page fault mechanism

References: Yobusame PoC for KVM Linux (using char device user mode
transfer)

Challenged by fast memory reading applications


Post-Copy Live Migration (pre-migration)

Guest VM

Host A Host B

Stop Page Pushing
Commit
Pre-migrate Reservation and 1
Copy Round

Live on A Downtime Degraded on B Live on B



Post-Copy Live Migration (reservation)

Guest VM Guest VM

Host A Host B

Stop Page Pushing
Commit
Copy Round




Post-Copy Live Migration (stop and copy)

Guest VM Guest VM

Host A Host B

Stop Page Pushing
Commit
Copy Round




Post-Copy Live Migration (post-copy)

Guest VM Guest VM

Page fault
Page push

Host A Host B

Stop Page Pushing
Commit
Copy Round




Post-Copy Live Migration (commit)

Guest VM

Host A Host B

Stop Page Pushing
Commit
Copy Round




Reducing Effects of Network Bound Page Faults

Low latency RDMA page transfer protocol (reducing latency/cost of page faults)
 Implemented fully in kernel mode OFED VERBS
 Can use the fastest RDMA hardware available (IB, IWARP, RoCE)

Demand pre-paging (pre-fetching) mechanism (reducing the number of page faults)
 Currently only a simple fetching of pages surrounding page on which fault occured

Full Linux MMU integration (reducing the system-wide effects/cost of page fault)
 Enabling to perform page fault transparency (only pausing the requesting thread)

Hybrid post-copy live migration (reducing the number of page faults)
 Moving some of the pages to the destination in a bounded pre-copy phase prior to relocation


Evaluation Pre-Copy vs. Post-Copy

Specification
 Benchmark: Google Stress Application Test (SAT)
with a 1 GB working set
 Communication: softIWARP over 1 GbE
 Guest VM: 2 x vCPU; 1 GB – 14 GB memory

Results
Roughly the total migration time between pre-copy
to post-copy is 1:10


VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications

Key takeaways

Pre-Copy and Post-Copy optimizations enhace the
support of live migrating memory intessive applications

To be released as open source under GPLv2 and LGPL
licenses to Qemu and Linux communities

Developed by SAP Research Technology Infrastructure
(TI) Practice


Thank you
Benoit Hudzia; Sr. Researcher;
SAP Research CEC Belfast
benoit.hudzia@sap.com

Aidan Shribman; Sr. Researcher;
SAP Research Israel
aidan.Shribman@sap.com

Legal Disclaimer

The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of
SAP. This presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP
has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or
release any functionality mentioned therein. This document, or any related presentation and SAP's strategy and possible future
developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at
any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to
deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied,
including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This
document is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or
omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.
All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially
from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as
of their dates, and they should not be relied upon in making purchasing decisions.


VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications

VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications

Recommended

More Related Content

Viewers also liked

Viewers also liked(20)

Recently uploaded

Recently uploaded(20)

VHPC'12: Pre-Copy and Post-Copy VM Live Migration for Memory Intensive Applications