Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Evaluation and Enhancement to Memory             Sharing and Swapping in Xen 4.1                Xiaowei Yang, Chuan Ye, Qi...
Agenda• Background• Memory Overcommit Features / Policy• Evaluation
Background• Challenges  • Memory becomes bottleneck as VM# grows       • E.g.: VM# > 100 in VDI scenario on 2-socket serve...
Goals• Evaluate current memory overcommit features in Xen4.1, particularly memory sharing / swapping• Make enhancements to...
Agenda• Background• Memory Overcommit Features / Policy• Evaluation
Blktap2 Sharing                                          How it works                                          Rely on lin...
Blktap2 Sharing (2)• Pros  • Overhead is lower than content-based page sharing       • No need to calculate/compare page c...
0-page Sharing                                             How it works                                             Share ...
0-page Sharing (2)• 0-page sharing is the most valuable part in content-based page sharing per our evaluation• 0-page shar...
Host Swap                                          How it works                                          Swap out         ...
Host Swap (2)• Pros  • The only memory overcommit feature that guarantees    overcommit ratio• Cons  • Page selection is h...
Host Swap Policies• Basics  • Random  • Sequential• Improvements  • Skip low memory: which is used for BIOS, kernel image ...
Memory Overcommit Policy• Metrics  •   Host Free Memory #  •   VM Free Memory #  •   VM Maximum Memory  •   VM Reserved Me...
Memory Overcommit Policy (2)• No memory pressure  • Turn off memory overcommit                         used% < 80%• Host m...
Memory Overcommit Support Matrix                            Balloon                Sharing     Host SwapQEMU              ...
Agenda• Background• Memory Overcommit Features / Policy• Evaluation
Experimental Environment                                     Host Configure   Processor                    2x Intel X5670 ...
Blktap2 / 0-page Sharing # -- VM startup                                 Unshared #• 0-page sharing # is dominant  • Xen s...
Blktap2 / 0-page Sharing # -- VDI workload                    VM’s Sharing # diff w/ VDI workload                         ...
0-page Sharing # -- Windows v.s. Linux• On startup almost all 0-pages are from `free memory’• Windows: Free memory is 0-pa...
Performance Impact of blktap2 Sharing• Performance impact of blktap2 is negligible• Scalability is very good• In theory bl...
Performance Impact of 0-page Sharing• Few impacts on the benchmarks’ scores• Impacts of different scan rates are almost th...
Host Swap Policies• Different policies result in different performances• Swap->sharing policy brings the best performance ...
Host Swap v.s. Balloon• Balloon usually performs better than swap  • Balloon transfers the memory pressure from host to gu...
VM Density -- VDI Workload                                                                 Projected                      ...
Takeaways• 0-page sharing complements blktap2 sharing. Thecombination of both is competitive• The performance impact and o...
Thank youwww.huawei.com
Upcoming SlideShare
Loading in …5
×

Evaluation and Enhancement to Memory Sharing and Swapping in Xen 4.1

4,336 views

Published on

Published in: Technology, Art & Photos
  • Be the first to comment

Evaluation and Enhancement to Memory Sharing and Swapping in Xen 4.1

  1. 1. Evaluation and Enhancement to Memory Sharing and Swapping in Xen 4.1 Xiaowei Yang, Chuan Ye, Qiangmin Lin {xiaowei.yang, yechuan, linqiangmin}@huawei.com HUAWEISponsored by: &
  2. 2. Agenda• Background• Memory Overcommit Features / Policy• Evaluation
  3. 3. Background• Challenges • Memory becomes bottleneck as VM# grows • E.g.: VM# > 100 in VDI scenario on 2-socket server • What’s the proper vMem size? • vMem# too small: bad performance • vMem# too big: low utilization• Memory overcommit is a key factor to high VM density /memory utilization• ESX / KVM have rich memory overcommit features / policies • ESX: Balloon, TPS, Host swap, compression • KVM: Balloon, KSM, Host swap• Xen adds sharing / swapping since 4.0, but • Untested in production • Need improvements in terms of efficiency, performance • No policies HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  4. 4. Goals• Evaluate current memory overcommit features in Xen4.1, particularly memory sharing / swapping• Make enhancements to current memory sharing /swapping features• Design a memory overcommit policy to reach higherVM density w/o sacrificing performance HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  5. 5. Agenda• Background• Memory Overcommit Features / Policy• Evaluation
  6. 6. Blktap2 Sharing How it works Rely on link-clone of parent image Share 1st read: • tapdisk2 read from the parent image on disk, and records the read / page (p1) Later read: • Tapdisk2 finds the record, notifies Xen HV to share with p1 Unshare • Write to the RO sharing page triggers unshareHUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  7. 7. Blktap2 Sharing (2)• Pros • Overhead is lower than content-based page sharing • No need to calculate/compare page contents in the background • Relief VM bootup storm issue • Later read from memory cache• Cons • Sharing% is much lower comparing to content-based sharing • Only VMs with the same parent image can share pages • Rely on PV driver • No share before PV driver loaded – startup • Page sharing # is limited by shm # • Between tapdisk2 processes to store global hash table HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  8. 8. 0-page Sharing How it works Share • zerosharing triggers Xen HV to scan VM’s page contents periodically • If Xen HV finds the page’s content is all 0, free the original page, and points VM’s corresponding p2m entry to a special RO 0-page Unshare • Write to the RO 0-page triggers unshare* Red words are our enhancements HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  9. 9. 0-page Sharing (2)• 0-page sharing is the most valuable part in content-based page sharing per our evaluation• 0-page sharing is more useful to Windows VM than toLinux VM • All free mem are 0-page in Windows – it scrubs before free• A proper scan rate of 0-page sharing is important • Slow: can’t relief memory pressure in time • Fast: high CPU util%• Actually 0-page sweep is used in POD • Usage limited - only when POD cache under the pressure HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  10. 10. Host Swap How it works Swap out • xenpaging selects guest pages thru policy • Xenpaing saves their contents to disk, then notifies Xen HV to free them Swap in • Guest access to the swapped-out page triggers violation • Xen HV notifies xenpaging to read back the contents to a new allocated pageHUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  11. 11. Host Swap (2)• Pros • The only memory overcommit feature that guarantees overcommit ratio• Cons • Page selection is hard • Which page is proper to be swapped out? • Inefficient • Disk access latency is much higher • Double swap scenario HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  12. 12. Host Swap Policies• Basics • Random • Sequential• Improvements • Skip low memory: which is used for BIOS, kernel image • MRU: prevent recent swapped-in pages from being swapped out • Aggressive MRU: prevent X continuous pages adjacent to each MRU page from being swapped out • Swap->sharing: If the elected page is 0-page, share it instead of swap• Advanced • Based on statistic of Guest OS page usage in HV • Based on Guest OS MM knowledge – enlightened HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  13. 13. Memory Overcommit Policy• Metrics • Host Free Memory # • VM Free Memory # • VM Maximum Memory • VM Reserved Memory • VM Current Zero Page # • VM Current Balloon Page # • VM Current Swap Page # • VM Current Sharing Page #• Configure options • Sharing threshold (default: 20%) • Balloon threshold (default: 10%) • Swap threshold (default: 5%) • 0-page sharing scan rate (frequency, page #) • … HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  14. 14. Memory Overcommit Policy (2)• No memory pressure • Turn off memory overcommit used% < 80%• Host memory pressure is moderate • Turn on blktap2 / 0-page sharing used% > 80% • Set 0-page sharing scan rate• Host memory pressure is severe • Adjust 0-page sharing scan rate used% > 90% • Start balloon; balloon # by VM metrics• Host memory pressure is critical • Balloon can’t afford memory consumption used% > 95% • Start swap• When the pressure goes down, return memory to VM HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  15. 15. Memory Overcommit Support Matrix Balloon Sharing Host SwapQEMU OK Sharing Trigger swap in breaksPV driver OK Sharing Trigger swap in breaksI/O device passthru OK Conflict! Conflict!VM live migration OK Sharing 1. Trigger swap in breaks 2. Swapfile accessible after L.M.VM save OK OK Triggers swap invMem snapshot OK OK Triggers swap inVM resume OK Sharing OK breaksVM hibernate 1. Balloon in OK Triggers swap in 2. Redirect to 0-pageHUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  16. 16. Agenda• Background• Memory Overcommit Features / Policy• Evaluation
  17. 17. Experimental Environment Host Configure Processor 2x Intel X5670 @ 2.93GHz, SMT enabled Memory 96GB DDR3 Storage Intel 160G X25-M SSD VM Workloads SPECjbb 1 vCPU, 4G vMem; SLES 11 Heap size: 2.5GB Kernel Build 1 vCPU, 512M vMem; SLES 11 Linux kernel: 2.6.32 Sysbench OLTP 1 vCPU, 1G vMem; SLES 11 Database: mysql VDI benchmark 1 vCPU, 1G vMem; Windows 7 Workload: Office, IE, PDF, JavaHUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  18. 18. Blktap2 / 0-page Sharing # -- VM startup Unshared #• 0-page sharing # is dominant • Xen scrubs pages (on host startup, domain destroy, …) • Lots of `free memory’ are 0-page, can be shared• Linux uses more memory to boot up • Less 0-page sharing # • More blktap2 sharing # HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  19. 19. Blktap2 / 0-page Sharing # -- VDI workload VM’s Sharing # diff w/ VDI workload Before (MB) After (MB) Diff 0-page sharing # 750 400 ↓53% Blktap2 sharing # 42 171 ↑409% Unshared # 199 418 ↑210%HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  20. 20. 0-page Sharing # -- Windows v.s. Linux• On startup almost all 0-pages are from `free memory’• Windows: Free memory is 0-page all the time • Windows scrubs page before free • More friendly to page sharing• Linux: Free memory is 0-page only on startup • Linux doesn’t scrub free page* Mem Hog test case consumes 500MB memory when running HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  21. 21. Performance Impact of blktap2 Sharing• Performance impact of blktap2 is negligible• Scalability is very good• In theory blktap2 sharing could benefit READ intensive workload • First time: from disk; afterwards: from cache • But the benefit is not observed in KB test HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  22. 22. Performance Impact of 0-page Sharing• Few impacts on the benchmarks’ scores• Impacts of different scan rates are almost the same • 5%-7% CPU overhead in dom0 • Few new 0-pages are generated during the test – scan finishes fast • A better benchmark? HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  23. 23. Host Swap Policies• Different policies result in different performances• Swap->sharing policy brings the best performance most of thetime• When the remaining vMem# < working set, the performancedrops dramatically HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  24. 24. Host Swap v.s. Balloon• Balloon usually performs better than swap • Balloon transfers the memory pressure from host to guest • Guest OS knows better about memory usage: which is free; which is not least/most used• Swap->sharing policy narrows the gap between swap / balloon HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  25. 25. VM Density -- VDI Workload Projected * Host memory = 96GB 1. W/ no memory overcommit: VM# = 85, memory is bottleneck 2. W/ balloon: VM# = 120, CPU/memory are both bottlenecks 3. W/ balloon+sharing: when VM# = 120, host free memory = 17GB, CPU is bottleneck 4. W/ balloon+sharing: w/o CPU bottleneck, same memory can host 145 VM (projected) * Test 1/2/3 are performed w/ QoE unchangedHUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  26. 26. Takeaways• 0-page sharing complements blktap2 sharing. Thecombination of both is competitive• The performance impact and overhead of blktap2 / 0-page sharing is small if used properly• Windows is more friendly to 0-page sharing than Linux• Host swap policy do matters. Swap->sharing policynarrows the gap between swap / balloon• In VDI scenario a good memory overcommit policy canincrease VM density by 70+% w/ QoE uncompromisedHUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential
  27. 27. Thank youwww.huawei.com

×