Scaling Xen within Rackspace Cloud Servers

Scaling Xen withinRackspace Cloud Servers Chris Behrens chris.behrens@rackspace.comXenSummit 2012, San Diego, CA USAAugust 28, 2012 1

Agenda•  Intro to Rackspace Cloud Servers•  Why Xen?•  Xen to XenServer•  OpenStack Basics•  OpenStack at Rackspace•  Scaling OpenStack Deployment•  Future RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2

Intro to Cloud Servers RACKSPACE® HOSTING | WWW.RACKSPACE.COM 3

Intro to Cloud ServersOverview•  Public Cloud product under Rackspace’s Open Cloud portfolio•  First Gen –  Code originated from Slicehost acquisition –  Initially Xen 3.x, but migrated to XenServer•  Next Gen –  Public Launch 8/1/2012 –  Powered by OpenStack•  XenServer 6•  Linux, Windows, and FreeBSD VMs RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4

Intro to Cloud ServersStatistics•  180,000+ total Rackspace customers (not just Cloud Servers)•  Tens of thousands of hosts•  Hundreds of thousands of VMs•  Millions of snapshots RACKSPACE® HOSTING | WWW.RACKSPACE.COM 5

Why Xen? RACKSPACE® HOSTING | WWW.RACKSPACE.COM 6

Why Xen?•  Thin hypervisor layer•  Open source•  Excellent driver support due to Linux dom0•  Good API•  Good performance•  Along with Linux, Windows VMs are a must –  Same virtualization technology desired for Windows –  Microsoft will support –  Good PV driver support RACKSPACE® HOSTING | WWW.RACKSPACE.COM 7

Xen to XenServer RACKSPACE® HOSTING | WWW.RACKSPACE.COM 8

Xen to XenServerCloud Servers use of Xen•  LVM –  Logical volumes per VM•  Base images stored as tar files•  Kernel and ramdisk outside of images•  Shell out to run ‘xm’ commands•  Snapshots w/ LVM –  Mounted and contents rsynced to another host•  No agent –  Modifications inside images done via mounting powered off VM RACKSPACE® HOSTING | WWW.RACKSPACE.COM 9

Xen to XenServerXen to XenServer changes•  VHDs for virtual disk images –  Tarred and gzipped•  Kernel and ramdisk inside images•  Snapshots via XenAPI –  Whole VDI chain wrapped in OVF container for upload to glance•  Agent inside VM images –  Communication via Xenstore –  File injection, password resets, networking setup RACKSPACE® HOSTING | WWW.RACKSPACE.COM 10

Xen to XenServerMigrating•  Pick a XenServer host•  Create VDI•  Attach VDI to dom0•  Partition, mkfs, and mount the new virtual disk•  LVM Snapshot original Xen VM and mount it•  rsync snapshot contents to mounted VDI on new host•  Power off original Xen VM and mount its virtual disk•  rsync again to new host to catch changes since snapshotting•  Extract a tar file into mounted VDI containing kernel, ramdisk, agent•  Unmount and detach VDI from dom0•  Create new VM record and attach VDI•  Start the new VM RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11

OpenStack The Basics RACKSPACE® HOSTING | WWW.RACKSPACE.COM 12

OpenStackComponents•  Nova –  Compute controller with various hypervisor drivers•  Glance –  VM Image registry•  Swift –  Object storage•  Quantum/Melange –  Networking•  Cinder –  Volumes/Block storage•  Keystone –  Authentication –  Authorization –  API Endpoint discovery RACKSPACE® HOSTING | WWW.RACKSPACE.COM 13

OpenStackTypical OpenStack Deployment keystone nova glance nova-api glance-api glance-registry RabbitMQ MySQL nova-scheduler MySQL nova-network nova-compute swift RACKSPACE® HOSTING | WWW.RACKSPACE.COM 14

OpenStacknova-compute w/ XenAPI•  Runs in a utility domU on each host•  Images downloaded in dom0 Physical host running XenServer" –  glance XenAPI plugin•  Images attached to nova-compute dom0" nova utility domU" domU –  Partitioning changes glance plugin nova-compute –  file-system resizing agent plugin•  Monitors VMs’ power_state•  Snapshots/backups xenstore plugin nova instance•  VM resize/migration domUs –  rsync to new host•  Communicates w/ agent inside VMs –  Via xenstore RACKSPACE® HOSTING | WWW.RACKSPACE.COM 15

OpenStackXenAPI VM Calls•  VM.start / VM.start_on •  VM.add_to_VCPUs_params•  VM.destroy •  VM.get_VBDs•  VM.clean_reboot •  VM.get_record•  VM.hard_reboot •  VM.get_by_uuid•  VM.clean_shutdown •  VM.get_by_name_label•  VM.hard_shutdown •  VM.set_name_label•  VM.pause •  VM.add_to_xenstore_data•  VM.unpause •  VM.remove_from_xenstore_data•  VM.suspend •  VM.assert_can_migrate•  VM.resume •  VM.migrate_send•  VM.snapshot •  VM.pool_migrate RACKSPACE® HOSTING | WWW.RACKSPACE.COM 16

OpenStackRackspace Specifics RACKSPACE® HOSTING | WWW.RACKSPACE.COM 17

OpenStack Rackspace SpecificsSource Code•  Rackspace tracks trunk –  Deployed code generally less than 2 weeks behind trunk•  Some custom patches on top of trunk –  Features specific to Rackspace –  Custom scheduling to meet Rackspace needs –  Scaling with OpenStack Compute Cells RACKSPACE® HOSTING | WWW.RACKSPACE.COM 18

OpenStack Rackspace SpecificsOpenStack Compute Cells•  Scaling beyond small nova deployments –  Effectively ties multiple nova deployments together•  Developed by Rackspace•  Targeted for Grizzly OpenStack release•  Hierarchical tree –  Multiple parents supported•  API cell(s) at top of the tree•  Compute cell(s) underneath•  DB and RabbitMQ per cell•  Scalability –  Splits connections to DB and RabbitMQ –  Less work to schedule new VM placement RACKSPACE® HOSTING | WWW.RACKSPACE.COM 19

OpenStack Rackspace Specifics OpenStack Nova Multi-cell Diagram API Cell RabbitMQ MySQL nova-api nova-cellsCompute Cell 1" Compute Cell 2" nova-cells nova-cells RabbitMQ RabbitMQ MySQL MySQL nova-scheduler nova-scheduler nova-network nova-network nova-compute nova-compute RACKSPACE® HOSTING | WWW.RACKSPACE.COM 20

Deploying OpenStack RACKSPACE® HOSTING | WWW.RACKSPACE.COM 21

Deploying OpenStack‘Inception’•  Private internal cloud –  Based on Openstack, called iNova –  OpenStack services for public cloud run virtualized under iNova•  Easy to spin up additional services for load balancing –  Enables quick reaction to load spikes•  Easier deployments of new compute cells•  Automated reaction to downed hosts•  Enables new strategies for deployments –  Replace vs Upgrade RACKSPACE® HOSTING | WWW.RACKSPACE.COM 22

Deploying OpenStackiNova Diagram iNova" OpenStack Control Public Cloud Infrastructure for XenServer Public Cloud" hosts XenServer hosts" domUs XenServer host w/ Public Cloud OpenStack Control Infrastructure for iNova" XenServer hosts RACKSPACE® HOSTING | WWW.RACKSPACE.COM 23

Deploying OpenStackContinuous Deployment•  Trunk and custom branches merged multiple times daily –  Unit Tested and Packaged•  Configuration managed with puppet –  Follows same QE and Continuous Deployment rules as code•  QE Environment –  OpenStack control infrastructure deployed on iNova –  nova-compute upgraded on 200 hosts –  Smoke tests•  Staging Deploy –  Uses the same packages and puppet manifests RACKSPACE® HOSTING | WWW.RACKSPACE.COM 24

Future RACKSPACE® HOSTING | WWW.RACKSPACE.COM 25

FutureImprovements desired from Xen•  Better VM resource isolation –  I/O scheduling weights per virtual disk•  Built in live migration without shared storage•  More information about VM resource usage –  Disk usage –  Help us improve OpenStack code –  Help customers react to bad situations•  XenAPI call to write to live xenstore•  Improve XenServer upgrade experience RACKSPACE® HOSTING | WWW.RACKSPACE.COM 26

FutureRackspace’s Other Interests•  Better VDI introduction for new VMs –  Look at VDI.introduce vs mv’ing VDIs and doing an sr-scan•  Incremental backups –  Delta VHDs instead of all VHDs•  Updated plugins using python >= 2.6 –  Cleans up workarounds for no built-in ‘uuid’ module –  Allows use of newer python features like context managers•  Better continuous deployment and testing –  Virtualize all of OpenStack •  Requires spinning XenServer on XenServer •  Wipe out compute nodes for QE environment •  Enable virtual labs for OpenStack developers –  Puppet used to configure XenServer RACKSPACE® HOSTING | WWW.RACKSPACE.COM 27

FutureiNova with Dev/QE Environments OpenStack QE iNova" Environment" OpenStack XenServer Control ‘hosts’ Infrastructure XenServer hosts" OpenStack Dev Environment" domUs OpenStack XenServer Control ‘hosts’ Infrastructure XenServer host w/ OpenStack Control Infrastructure for iNova" RACKSPACE® HOSTING | WWW.RACKSPACE.COM 28

" Chris Behrens" chris.behrens@rackspace.com" RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM 29

http://www.xen.org	97153
http://xen.org	21530
http://xen.xensource.com	297
http://translate.googleusercontent.com	292
http://www.xenproject.org	148
http://staging.xen.org	111
http://xenproject.org	34
http://xenorg.cloudaccess.net	28
http://131.253.14.66	8
http://www.xen.org.	6
http://webcache.googleusercontent.com	4
http://irq.tumblr.com	4
http://xen.org.	3
http://50.57.168.107	3
http://xen.org.sixxs.org	3
http://mail.xen.org	2
http://safe.tumblr.com	2
http://xen	2
http://www.xen.org&_=1352285117656 HTTP	1
https://www.google.com&_=1352570042024 HTTP	1
http://www.xen.org&_=1352783478143 HTTP	1
http://www.xen.org&_=1352783455005 HTTP	1
http://www.xen.org&_=1352285259296 HTTP	1
https://translate.googleusercontent.com	1
http://www.xen.org.sixxs.org	1
http://www.google.com&_=1349622827067 HTTP	1
http://xen.xensource.com&_=1349886236015 HTTP	1
http://www.xen.org&_=1348163584571 HTTP	1
http://74.6.238.254	1
http://xen.org&_=1348016971135 HTTP	1
http://131.253.14.98	1
http://www.xenproject.org.	1

Scaling Xen within Rackspace Cloud Servers

by XenProject.org on Aug 30, 2012

Accessibility

Categories

Upload Details

Usage Rights

32 Embeds 119K

Statistics

Scaling Xen within Rackspace Cloud Servers Presentation Transcript