Yabusame:  Postcopy Live migration for QEmu/KVM* Isaku Yamahata, VALinux Systems Japan K.K. <yamahata@private.email.ne.jp>...
Agenda    ●        Demo    ●        Precopy vs Postcopy    ●        Implementation    ●        Evaluation    ●        Futu...
Demo
Precopy vs Postcopy
Precopy live migration1.Enable dirty page tracking2.Copy all memory pages to destination3.Copy memory pages dirtied during...
Postcopy live migration1.Stop VM2.Copy non-memory VM states  to destination3.Resume VM at destination4.Copy memory pages o...
Asynchronous Page Fault(APF)                                                            4. switch AFP can be utilized for ...
Total migration/down time           Copy VM memory before switching the execution host                                    ...
Precopy vs Postcopy                         precopy                        postcopyTotal migration time     (RAM size / li...
Precopy characteristic●    Total migration time and downtime depend    on memory dirtying speed    ●        Especially the...
Postcopy characteristic●    network bandwidth friendly    ●        Postcopy transfer a page only once●    reliability    ●...
Postcopy is applicable for●   Planned maintenance    ●   Predictable total migration time is important●   Dynamic consolid...
Implementation
Hooking guest RAM access  ●      Design choice      ●          Insert hooks all accesses to guest RAM      ●          Char...
Implementation              4. page contents is sent back              Connection for live-migration              is reuse...
Evaluation
This evaluation is done with the experimental implementation
Dynamic consolidationThis evaluation is done with the experimental implementation
This evaluation is done with the experimental implementation
Memory scanning with postcopy●   6GB memory Guest RAM●   4thread●   Per-thread    ●   1GB    ●   Each thread accessing all...
Memory scan time(real)
Total CPU time allocated to guest VM VCPU execution efficiency is improved cpu-time/real-time APF enabled: 0.7 APF disable...
Analyze with SystemTapvCPU is executing during page is being served                                                Serving...
Siege benchmark with Apache●   Host        Core2 quad CPU Q9400 4core                                                     ...
Precopy                 PostcopyPrecopy                Postcopy w/o background transferMigrate set_speed=1G   Prefault for...
Future work●   Upstream merge    ●   QEmu/KVM Live-migration code needs more love        –   Code clean up, Feature negoti...
Thank you●   Questions?●   Resources    ●   Project page        –   http://grivon.apgrid.org/quick-kvm-migration        – ...
Backup slides
Real-Time Issues in Live Migration of Virtual Machineshttp://retis.sssup.it/~tommaso/publications/VHPC09.pdf
Upcoming SlideShare
Loading in …5
×

Yabusame: postcopy live migration for qemu/kvm

2,259 views

Published on

Published in: Technology, Business
  • Be the first to comment

Yabusame: postcopy live migration for qemu/kvm

  1. 1. Yabusame: Postcopy Live migration for QEmu/KVM* Isaku Yamahata, VALinux Systems Japan K.K. <yamahata@private.email.ne.jp> Takahiro Hirofuchi, AIST <t.hirofuchi@aist.go.jp> LinuxConJapan June 7th, 2012
  2. 2. Agenda ● Demo ● Precopy vs Postcopy ● Implementation ● Evaluation ● Future work ● Summary From wikipediaYabusame is a joint project with Takahiro Hirofuchi, AIST and Satoshi Itoh, AIST.This work is partly supported by JST/CREST ULP and KAKENHI (23700048).The development of Yabusame was partly funded by METI (Minister of Economy,Trade and Industry) and supported by NTT Communications Corporation.
  3. 3. Demo
  4. 4. Precopy vs Postcopy
  5. 5. Precopy live migration1.Enable dirty page tracking2.Copy all memory pages to destination3.Copy memory pages dirtied during the Repeat this stepprevious copy again4.Repeat the 3rd step until the rest ofmemory pages are enough small.5.Stop VM6.Copy the rest of memory pages andnon-memory VM states7.Resume VM at destination
  6. 6. Postcopy live migration1.Stop VM2.Copy non-memory VM states to destination3.Resume VM at destination4.Copy memory pages on- demand/backgroundly • Async PF can be utilized Copy memory pages ● On-demand(network fault) ● background(precache)
  7. 7. Asynchronous Page Fault(APF) 4. switch AFP can be utilized for postcopy1.Guest RAM as host page can be Thread A Thread B swapped out(or on the src machine 8.unblock in case of postcopy)2.When the page is faulted, worker 3 notify 7. notify 1. Page fault threads starts IO APF to guest IO completion3.Notify it to (PV) guest Guest RAM4.Guest blocks the faulting thread and switches to another thread5.When IO is completed, it is KVM notified to guest6.Unblock the previously blocked thread 2. pass page Request to Work queue work queue 5.Start IO 6. IO completion
  8. 8. Total migration/down time Copy VM memory before switching the execution host Round 2 Round N stop resume ... restart Precopy Remaining prepara VM statePrecopy tion Round 1 Dirty page Performance degradation Down time Due to dirty page tracking Total migration timetime stop resume restart Postcopy prepara VM statePostcopy tion Demand/pre paging(with async PF) Down time Performance degradation Due to network fault Total migration time Copy VM memory after switching the execution host
  9. 9. Precopy vs Postcopy precopy postcopyTotal migration time (RAM size / link speed) + (RAM size / link speed) + overhead + ovedrhead Non-deterministic Depending on precopy roundsWorst downtime (VMState time) + VMState time (RAM size/link speed) Followed by postcopy phase The number of dirtied pages Postcopy time is limited by can be optimized by ordering (RAM size/link speed) pages sent in precopy phase Alleviated by Prepaging + AsyncPF
  10. 10. Precopy characteristic● Total migration time and downtime depend on memory dirtying speed ● Especially the number of dirty pages doesnt converge when dirtying speed > link speed Downtime Analysis for Pro-Active Virtual Machine Live Migration : Felix Salfner http://www.tele-task.de/archive/video/flash/13549/
  11. 11. Postcopy characteristic● network bandwidth friendly ● Postcopy transfer a page only once● reliability ● VM can be lost if network failure occurs during migration
  12. 12. Postcopy is applicable for● Planned maintenance ● Predictable total migration time is important● Dynamic consolidation ● In cloud use case, usually resources are over-committed ● If machine load becomes high, evacuate the VM to other machine promptly – Precopy optimization (= CPU cycle) may make things worse● Wide area migration ● Inter-Datacenter live-migration – L2 connectivity among datacenters with L2 over L3 has becoming common – VM migration over DCs as Disaster Recovery● LAN case ● Not all network bandwidth can be used for migration ● network bandwidth might be reserved by QoS
  13. 13. Implementation
  14. 14. Hooking guest RAM access ● Design choice ● Insert hooks all accesses to guest RAM ● Character device driver (umem char device) – Async page fault support ● Backing store(block device or file) ● Swap device Pros ConsModify VMM portability impracticalBacking store No new device driver Difficult future improvement Some kvm host features wouldnt workCharacter Device Straight forward Need to fix kvm host features Future improvementSwap device Everything is normal after Administration migration Difficult future improvement
  15. 15. Implementation 4. page contents is sent back Connection for live-migration is reused qemu-kvm daemon qemu-kvm 5. page contents 0. mmap() 3. Request for page is passed down to 1. access to the driver guest RAM 6. resolve page fault guest RAM character guest RAM device vma::fault 2. page fault is hooked byHost kernel Host kernel Character device.source destination
  16. 16. Evaluation
  17. 17. This evaluation is done with the experimental implementation
  18. 18. Dynamic consolidationThis evaluation is done with the experimental implementation
  19. 19. This evaluation is done with the experimental implementation
  20. 20. Memory scanning with postcopy● 6GB memory Guest RAM● 4thread● Per-thread ● 1GB ● Each thread accessing all pages Guest VM after switching execution ● Time from the first page access to Thread 1 ... Thread 4 the last page access● Start each thread right after Memory scan starting post-copy migration ...● Background transfer is disabled 1GB 1GB Host OS(src) Host OS(dst) 1GB ether
  21. 21. Memory scan time(real)
  22. 22. Total CPU time allocated to guest VM VCPU execution efficiency is improved cpu-time/real-time APF enabled: 0.7 APF disabled: 0.39
  23. 23. Analyze with SystemTapvCPU is executing during page is being served Serving page fault vCPU execution Serving page fault vCPU execution vCPU cant be executed while page is being served
  24. 24. Siege benchmark with Apache● Host Core2 quad CPU Q9400 4core GB ● ● Memory 16GB● Qemu/kvm ● Vcpu = 4, kernel_irqchip=on migration ● Memory about 6G(-m 6000) ● virtio Guest VM● Apache Apache ● Prefork 128 process fixed Prefork 128 process ● Data: 100K * 10000files = about 1GB ● Warm up by siege before migration Host(src) Host(dst) – So all data are cached on memory● Siege https ● http load testing and benchmarking utility ● http://www.joedog.org/siege-home/ GB ● 64 parallel (-c 64) ● Random URL to 10000 URLs siege Client machine
  25. 25. Precopy PostcopyPrecopy Postcopy w/o background transferMigrate set_speed=1G Prefault forward=100 migrate -p -n tcp:10.0.0.18:4444 100 0
  26. 26. Future work● Upstream merge ● QEmu/KVM Live-migration code needs more love – Code clean up, Feature negotiation... ● Investigate for fuse version of umem device and evaluation – See if its possible and its performance is acceptable ● Evaluation/benchmark● KSM and THP● Threading ● Page compression(XBRZLE) is coming. Non-blocking read + checking if all data is ready is impractical for compressed page any more.● Mix precopy/postcopy● Avoid memory copy● Not to fetch pages when writing/clearing whole page ● cleancache/frontswap might be good candidate ● Free page arent necessary to be transferred – Self-ballooning? ● Hint via PV interface?● Libvirt support?(and Openstack?)● Cooperate with Kemari
  27. 27. Thank you● Questions?● Resources ● Project page – http://grivon.apgrid.org/quick-kvm-migration – http://sites.google.com/site/grivonhome/quick-kvm-migration ● Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension: proof-of-concept, ad-hoc prototype. not a new design – http://grivon.googlecode.com/svn/pub/docs/ccgrid2010-hirofuchi-paper.pdf – http://grivon.googlecode.com/svn/pub/docs/ccgrid2010-hirofuchi-talk.pdf ● Reactive consolidation of virtual machines enabled by postcopy live migration: advantage for VM consolidation – http://portal.acm.org/citation.cfm?id=1996125 – http://www.emn.fr/x-info/ascola/lib/exe/fetch.php?media=internet:vtdc-postcopy.pdf ● Qemu Wiki – http://wiki.qemu.org/Features/PostCopyLiveMigration ● Demo video – http://www.youtube.com/watch?v=lo2JJ2KWrlA
  28. 28. Backup slides
  29. 29. Real-Time Issues in Live Migration of Virtual Machineshttp://retis.sssup.it/~tommaso/publications/VHPC09.pdf

×