XPDS13: Increasing XenServer's VM density - Jonathan Davies, Citrix
Upcoming SlideShare
Loading in...5
×
 

XPDS13: Increasing XenServer's VM density - Jonathan Davies, Citrix

on

  • 1,216 views

As the number of CPU cores in server-class hardware increases, the demand on a virtualisation platform increases for greater VM density. Most commercial virtualisation platforms now support several ...

As the number of CPU cores in server-class hardware increases, the demand on a virtualisation platform increases for greater VM density. Most commercial virtualisation platforms now support several hundred VMs per host. This talk will describe the scalability challenges that were overcome in Citrix XenServer 6.2 to enable support for up to 500 fully virtualised or 650 paravirtualised VMs per host. These include limits with event channels, blktap, xenstored and consoled. It will also discuss how dom0 CPU utilisation was reduced in order to make a large number of VMs responsive and thus usable, and will present benchmark measurements quantifying these improvements.

Statistics

Views

Total Views
1,216
Slideshare-icon Views on SlideShare
1,209
Embed Views
7

Actions

Likes
0
Downloads
21
Comments
0

2 Embeds 7

http://www.xenproject.org 6
http://xenproject.org 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    XPDS13: Increasing XenServer's VM density - Jonathan Davies, Citrix XPDS13: Increasing XenServer's VM density - Jonathan Davies, Citrix Presentation Transcript

    • Increasing XenServer’s VM density Jonathan Davies, XenServer System Performance Lead XenServer Engineering, Citrix Cambridge, UK 24 Oct 2013 Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 1 / 34
    • Outline 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 2 / 34
    • Scalability expectations Outline 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 3 / 34
    • Scalability expectations Users expect VM density to scale with hardware Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 4 / 34
    • Scalability expectations Users expect VM density to scale with hardware Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 5 / 34
    • Scalability expectations Users get upset when. . . Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 6 / 34
    • Scalability expectations XenServer’s VM density scalability hard density limit :-( hardware's theoretical capacity XS 6.1 (and earlier) Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 7 / 34
    • Scalability expectations XenServer’s VM density scalability hard density limit :-( hardware's theoretical capacity XS 6.1 (and earlier) XS 6.2 practical density limit (depending on nature of VMs) Jonathan Davies (Citrix) Increasing XenServer’s VM density hard density limit :-) 24 Oct 2013 8 / 34
    • Hard limits Outline 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 9 / 34
    • Hard limits Enumerated causes of limitations Hard limit 1: dom0 event channels Cause of limitation XenServer uses a 32-bit dom0 This means 1,024 dom0 event channels #define MAX_EVTCHNS(d) (BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d)) Various VM functions use a dom0 event channel VM density hard limit 225 VMs per host (PV with 1 vCPU, 1 VIF, 1 VBD) 150 VMs per host (HVM with 1 vCPU, 1 VIF, 3 VBDs) Mitigation for XS 6.2 Hack for dom0 to enjoy 4,096 event channels → 800 VMs per host (PV with 1 vCPU, 1 VIF, 1 VBD) → 570 VMs per host (HVM with 1 vCPU, 1 VIF, 3 VBDs) Mitigation for future Change the ABI to provide unlimited event channels → this would remove the limit Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 10 / 34
    • Hard limits Enumerated causes of limitations Hard limit 2: blktap2 device minor numbers Cause of limitation blktap2 only supports up to 1,024 minor numbers (despite the kernel allowing up to 1,048,576) #define MAX_BLKTAP_DEVICE 1024 Each virtual block device requires one device VM density hard limit 341 VMs per host (with 3 disks per VM) Mitigation for XS 6.2 Double this constant to 2,048 → 682 VMs per host (with 3 disks per VM) Mitigation for future Move away from blktap2 altogether? → this would remove the limit Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 11 / 34
    • Hard limits Enumerated causes of limitations Hard limit 3: number of aio requests Cause of limitation Each blktap2 instance creates an asynchronous I/O context for receiving 402 events. Default system-wide number of aio requests was 444,416 in XS 6.1. VM density hard limit 368 VMs per host (with 3 disks per VM) Mitigation for XS 6.2 Set fs.aio-max-nr to 1,048,576 → 869 VMs per host (with 3 disks per VM) Mitigation for future Increase fs.aio-max-nr further or use storage driver domains → this would remove the limit Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 12 / 34
    • Hard limits Enumerated causes of limitations Hard limit 4: dom0 grant references Cause of limitation Windows VMs use receive-side copy (RSC) by default in XS 6.1. netback allocates (at least) 22 grant-table entries per virtual interface for RSC. dom0 had a total of 8,192 grant-table entries in XS 6.1. VM density hard limit 372 VMs per host (with 1 interface per VM) Mitigation for XS 6.2 Don’t use RSC in Windows VMs anyway → this removes the limit Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 13 / 34
    • Hard limits Enumerated causes of limitations Hard limit 5: connections to xenstored Cause of limitation xenstored uses select(2), which can only listen on 1,024 file descriptors. #define __FD_SETSIZE 1024 qemu opens 3 file descriptors to xenstored. VM density hard limit 333 VMs per host (HVM) Mitigation for XS 6.2 Make two qemu watches share a connection → 500 VMs per host (HVM) Mitigation for future Upstream qemu doesn’t connect to xenstored → this will remove the limit Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 14 / 34
    • Hard limits Enumerated causes of limitations Hard limit 6: connections to consoled Cause of limitation Similarly, consoled uses select(2) Each PV domain opens 3 fds to consoled VM density hard limit 341 VMs per host (PV) Mitigation for XS 6.2 Use poll(2) rather than select(2) in consoled → this removes the limit Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 15 / 34
    • Hard limits Enumerated causes of limitations Hard limit 7: dom0 low memory Cause of limitation Each running VM eats about 1MB of dom0 lowmem VM density hard limit around 650 VMs per host Mitigation for future Use a 64-bit dom0 → this will remove the limit Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 16 / 34
    • Hard limits Summary of hard limits Summary of hard limits Limits on number of HVM guests with 1 vCPU, 1 VBD, 1 VIF (with PV drivers) Limitation dom0 event channels blktap minor numbers aio requests dom0 grant references xenstored connections consoled connections dom0 low memory Overall limit Limited by Jonathan Davies (Citrix) XS 6.1 225 1024 1105 372 333 no limit 650 225 event channels XS 6.2 800 2048 2608 no limit 500 no limit 650 500 xenstored Increasing XenServer’s VM density Future no limit no limit no limit no limit no limit no limit no limit very high something else! 24 Oct 2013 17 / 34
    • Hard limits Summary of hard limits Summary of hard limits Limits on number of HVM guests with 1 vCPU, 3 VBDs, 1 VIF (with PV drivers) Limitation dom0 event channels blktap minor numbers aio requests dom0 grant references xenstored connections consoled connections dom0 low memory Overall limit Limited by Jonathan Davies (Citrix) XS 6.1 150 341 368 372 333 no limit 650 150 event channels XS 6.2 570 682 869 no limit 500 no limit 650 500 xenstored Increasing XenServer’s VM density Future no limit no limit no limit no limit no limit no limit no limit very high something else! 24 Oct 2013 18 / 34
    • Hard limits Summary of hard limits Summary of hard limits Limits on number of PV guests with 1 vCPU, 1 VBD, 1 VIF Limitation dom0 event channels blktap minor numbers aio requests dom0 grant references xenstored connections consoled connections dom0 low memory Overall limit Limited by Jonathan Davies (Citrix) XS 6.1 225 1024 368 no limit no limit 341 650 225 event channels XS 6.2 1000 2048 869 no limit no limit no limit 650 650 dom0 lowmem Increasing XenServer’s VM density Future no limit no limit no limit no limit no limit no limit no limit very high something else! 24 Oct 2013 19 / 34
    • Hard limits Summary of hard limits 500 Windows VMs on a host Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 20 / 34
    • Soft limits Outline 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 21 / 34
    • Soft limits xenstored High dom0 CPU utilisation by xenstored top - 16:29:33 up 36 min, 1 user, load average: 0.80, 0.56, 0.47 Tasks: 132 total, 1 running, 131 sleeping, 0 stopped, 0 zombie Cpu(s): 40.1%us, 40.0%sy, 0.0%ni, 17.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4186504k total, 443480k used, 3743024k free, 23696k buffers Swap: 524280k total, 0k used, 524280k free, 132504k cached PID 7339 6665 7225 7269 7413 7283 10938 16403 1 2 3 USER root root root root root root root root root root root PR 20 20 20 20 20 20 20 20 20 20 RT Jonathan Davies (Citrix) NI VIRT RES SHR S %CPU %MEM 0 6732 2240 840 S 80.2 0.1 0 4344 2636 584 S 0.4 0.1 0 48892 5356 1736 S 0.3 0.1 0 23704 3684 1308 S 0.3 0.1 0 195m 21m 8932 S 0.3 0.5 0 7492 4860 1200 S 0.3 0.1 0 29808 1856 956 S 0.3 0.0 0 2428 1104 824 R 0.3 0.0 0 2164 656 564 S 0.0 0.0 0 0 0 0 S 0.0 0.0 0 0 0 0 S 0.0 0.0 Increasing XenServer’s VM density TIME+ 0:10.22 0:04.03 0:03.35 0:03.47 0:10.28 0:08.65 0:00.40 0:02.31 0:00.83 0:00.00 0:00.01 COMMAND xenstored fe xcp-rrdd xcp-rrdd-iostat xapi xcp-rrdd-xenpm v6d top init kthreadd migration/0 24 Oct 2013 22 / 34
    • Soft limits xenstored High dom0 CPU utilisation by xenstored dom0 vCPUs domU vCPUs Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 23 / 34
    • Soft limits xenstored High dom0 CPU utilisation by xenstored xenstored's dom0 vCPU other dom0 vCPUs domU vCPUs Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 24 / 34
    • Soft limits xenstored High dom0 CPU utilisation by xenstored Cause of limitation xenstored CPU utilisation bottleneck Mitigation for XS 6.2 Reduce xenstore use by XenServer’s toolstack: remove some spurious writes replace polling with watching Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 25 / 34
    • Soft limits qemu High dom0 CPU utilisation due to qemu top - 16:40:27 up 2:07, 1 user, load average: 89.62, 87.22, 76.90 Tasks: 1015 total, 65 running, 950 sleeping, 0 stopped, 0 zombie Cpu(s): 23.4%us, 55.5%sy, 0.0%ni, 4.8%id, 0.0%wa, 0.0%hi, 15.4%si, 0.5%st Mem: 4180480k total, 1615840k used, 2564640k free, 3804k buffers Swap: 524280k total, 0k used, 524280k free, 122852k cached PID 7143 6653 16496 16970 997 3195 3497 3562 3993 7597 8150 8679 8974 11937 12545 14053 17752 USER root root tcpdump root 65583 65684 65656 65685 65546 65659 65550 65627 65661 root 65556 65598 65567 PR NI VIRT RES SHR S 20 0 0 0 0 R 10 -10 12264 7796 1152 R 20 0 5508 2132 1248 R 20 0 2952 1552 736 R 20 0 24696 4732 1572 S 20 0 24632 4736 1572 S 20 0 24760 4740 1576 R 20 0 24696 4732 1572 S 20 0 24888 4744 1580 S 20 0 24632 4736 1576 S 20 0 24760 4740 1580 R 20 0 24632 4740 1576 R 20 0 24568 4736 1572 S 20 0 0 0 0 S 20 0 24824 4748 1584 S 20 0 24760 4736 1576 S 20 0 24952 4740 1576 S Jonathan Davies (Citrix) %CPU %MEM 33.9 0.0 31.8 0.2 10.5 0.1 6.3 0.0 3.1 0.1 3.1 0.1 3.1 0.1 3.1 0.1 3.1 0.1 3.1 0.1 3.1 0.1 3.1 0.1 3.1 0.1 3.1 0.0 3.1 0.1 3.1 0.1 3.1 0.1 Increasing XenServer’s VM density TIME+ 17:21.63 36:14.34 5:35.12 0:00.11 0:56.30 0:27.34 0:28.65 0:26.97 0:53.19 0:28.86 0:51.71 0:31.18 0:27.97 1:12.92 0:58.46 0:31.33 0:56.82 COMMAND rpciod/0 ovs-vswitchd tcpdump top qemu-dm qemu-dm qemu-dm qemu-dm qemu-dm qemu-dm qemu-dm qemu-dm qemu-dm nfsiod qemu-dm qemu-dm qemu-dm 24 Oct 2013 26 / 34
    • Soft limits qemu qemu burning dom0 CPU 200 idle Windows guests, each qemu utilising 3% of a CPU means 6 dom0 vCPUs wasted! Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 27 / 34
    • Soft limits qemu What is qemu busy doing? Emulated device USB CD-ROM Buffered I/O & RTC timer Parallel port Serial port VNC qemu monitor qemu events per VM per second 221 38 13 1 1 1 1 Mitigation for XS 6.2 Use an event-channel for buffered I/O notifications Provide options to disable all emulated devices Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 28 / 34
    • Benchmarks Outline 1 Scalability expectations 2 Hard limits 3 Soft limits 4 Benchmarks Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 29 / 34
    • Benchmarks Bootstorm Booting 90 Win7 VMs Time to fully boot 90 VMs (25 at a time) 2500 Tampa (XS 6.1) Clearwater (XS 6.2) Elapsed time (s) 2000 1500 1000 XS 6.2 is 60% faster 500 0 0 25 50 75 100 125 150 175 200 VM index Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 6272 @ 2.1GHz) Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 30 / 34
    • Benchmarks Bootstorm Booting 120 Win7 VMs Time to fully boot 120 VMs (25 at a time) 2500 Tampa (XS 6.1) Clearwater (XS 6.2) Elapsed time (s) 2000 1500 1000 XS 6.2 is 75% faster 500 0 0 25 50 75 100 125 150 175 200 VM index Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 6272 @ 2.1GHz) Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 31 / 34
    • Benchmarks Bootstorm Booting 200 Win7 VMs Time to fully boot 200 VMs (25 at a time) 2500 Tampa (XS 6.1) Clearwater (XS 6.2) XS 6.1 can't even get 200 VMs running! Elapsed time (s) 2000 1500 It took XS 6.2 just 13 minutes to boot 200 VMs (on this hardware) 1000 500 0 0 25 50 75 100 125 150 175 200 VM index Machine used: Dell PowerEdge R815 (Quad 16-core AMD Opteron 6272 @ 2.1GHz) Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 32 / 34
    • Benchmarks LoginVSI number of VMs performing acceptably LoginVSI: number of usable Windows VMs XS 6.2 XS 6.1 number of VMs running Machine used: Quad 10-core Intel E7-4860 @ 2.27GHz Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 33 / 34
    • Questions Questions ? Jonathan Davies (Citrix) Increasing XenServer’s VM density 24 Oct 2013 34 / 34