Scaling Xen withinRackspace Cloud Servers                            Chris Behrens                    chris.behrens@racksp...
Agenda•  Intro to Rackspace Cloud Servers•  Why Xen?•  Xen to XenServer•  OpenStack Basics•  OpenStack at Rackspace•  Scal...
Intro to Cloud Servers                    RACKSPACE® HOSTING   |   WWW.RACKSPACE.COM                                      ...
Intro to Cloud ServersOverview•  Public Cloud product under Rackspace’s Open Cloud portfolio•  First Gen –  Code originate...
Intro to Cloud ServersStatistics•  180,000+ total Rackspace customers (not just Cloud Servers)•  Tens of thousands of host...
Why Xen?           RACKSPACE® HOSTING   |   WWW.RACKSPACE.COM                                                        6
Why Xen?•  Thin hypervisor layer•  Open source•  Excellent driver support due to Linux dom0•  Good API•  Good performance•...
Xen to XenServer                   RACKSPACE® HOSTING   |   WWW.RACKSPACE.COM                                             ...
Xen to XenServerCloud Servers use of Xen•  LVM –  Logical volumes per VM•  Base images stored as tar files•  Kernel and ra...
Xen to XenServerXen to XenServer changes•  VHDs for virtual disk images –  Tarred and gzipped•  Kernel and ramdisk inside ...
Xen to XenServerMigrating•  Pick a XenServer host•  Create VDI•  Attach VDI to dom0•  Partition, mkfs, and mount the new v...
OpenStack The Basics              RACKSPACE® HOSTING   |   WWW.RACKSPACE.COM                                              ...
OpenStackComponents•  Nova  –  Compute controller with various hypervisor drivers•  Glance  –  VM Image registry•  Swift  ...
OpenStackTypical OpenStack Deployment                                   keystone                  nova                    ...
OpenStacknova-compute w/ XenAPI•  Runs in a utility domU on each host•  Images downloaded in dom0              Physical ho...
OpenStackXenAPI VM Calls•  VM.start / VM.start_on   •  VM.add_to_VCPUs_params•  VM.destroy               •  VM.get_VBDs•  ...
OpenStackRackspace Specifics                      RACKSPACE® HOSTING   |   WWW.RACKSPACE.COM                              ...
OpenStack Rackspace SpecificsSource Code•  Rackspace tracks trunk –  Deployed code generally less than 2 weeks behind trun...
OpenStack Rackspace SpecificsOpenStack Compute Cells•  Scaling beyond small nova deployments  –  Effectively ties multiple...
OpenStack Rackspace Specifics   OpenStack Nova Multi-cell Diagram            API Cell                          RabbitMQ   ...
Deploying OpenStack                 RACKSPACE® HOSTING   |   WWW.RACKSPACE.COM                                            ...
Deploying OpenStack‘Inception’•  Private internal cloud  –  Based on Openstack, called iNova  –  OpenStack services for pu...
Deploying OpenStackiNova Diagram                   iNova"                                   OpenStack Control             ...
Deploying OpenStackContinuous Deployment•  Trunk and custom branches merged multiple times daily –  Unit Tested and Packag...
Future         RACKSPACE® HOSTING   |   WWW.RACKSPACE.COM                                                      25
FutureImprovements desired from Xen•  Better VM resource isolation  –  I/O scheduling weights per virtual disk•  Built in ...
FutureRackspace’s Other Interests•  Better VDI introduction for new VMs –  Look at VDI.introduce vs mv’ing VDIs and doing ...
FutureiNova with Dev/QE Environments                                          OpenStack QE                iNova"          ...
"                                                                        Chris Behrens"                                   ...
Upcoming SlideShare
Loading in...5
×

Scaling Xen within Rackspace Cloud Servers

129,866

Published on

Rackspace has years of experience with running Xen at scale, starting with Xen and migrating to XenServer. We will share why we use Xen/XenServer along with some of the issues that we've experienced. We will touch on our experience with migrating from Xen to XenServer and the challenges there. We will share information about Rackspace Cloud Servers architecture, and touch briefly on OpenStack when doing so. We will explain how we use Xen to quickly deploy new Openstack services with what we call Nova on Nova. And finally, we will discuss what additional features and improvements are needed and why.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
129,866
On Slideshare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
129
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Scaling Xen within Rackspace Cloud Servers

  1. 1. Scaling Xen withinRackspace Cloud Servers Chris Behrens chris.behrens@rackspace.comXenSummit 2012, San Diego, CA USAAugust 28, 2012 1
  2. 2. Agenda•  Intro to Rackspace Cloud Servers•  Why Xen?•  Xen to XenServer•  OpenStack Basics•  OpenStack at Rackspace•  Scaling OpenStack Deployment•  Future RACKSPACE® HOSTING | WWW.RACKSPACE.COM 2
  3. 3. Intro to Cloud Servers RACKSPACE® HOSTING | WWW.RACKSPACE.COM 3
  4. 4. Intro to Cloud ServersOverview•  Public Cloud product under Rackspace’s Open Cloud portfolio•  First Gen –  Code originated from Slicehost acquisition –  Initially Xen 3.x, but migrated to XenServer•  Next Gen –  Public Launch 8/1/2012 –  Powered by OpenStack•  XenServer 6•  Linux, Windows, and FreeBSD VMs RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4
  5. 5. Intro to Cloud ServersStatistics•  180,000+ total Rackspace customers (not just Cloud Servers)•  Tens of thousands of hosts•  Hundreds of thousands of VMs•  Millions of snapshots RACKSPACE® HOSTING | WWW.RACKSPACE.COM 5
  6. 6. Why Xen? RACKSPACE® HOSTING | WWW.RACKSPACE.COM 6
  7. 7. Why Xen?•  Thin hypervisor layer•  Open source•  Excellent driver support due to Linux dom0•  Good API•  Good performance•  Along with Linux, Windows VMs are a must –  Same virtualization technology desired for Windows –  Microsoft will support –  Good PV driver support RACKSPACE® HOSTING | WWW.RACKSPACE.COM 7
  8. 8. Xen to XenServer RACKSPACE® HOSTING | WWW.RACKSPACE.COM 8
  9. 9. Xen to XenServerCloud Servers use of Xen•  LVM –  Logical volumes per VM•  Base images stored as tar files•  Kernel and ramdisk outside of images•  Shell out to run ‘xm’ commands•  Snapshots w/ LVM –  Mounted and contents rsynced to another host•  No agent –  Modifications inside images done via mounting powered off VM RACKSPACE® HOSTING | WWW.RACKSPACE.COM 9
  10. 10. Xen to XenServerXen to XenServer changes•  VHDs for virtual disk images –  Tarred and gzipped•  Kernel and ramdisk inside images•  Snapshots via XenAPI –  Whole VDI chain wrapped in OVF container for upload to glance•  Agent inside VM images –  Communication via Xenstore –  File injection, password resets, networking setup RACKSPACE® HOSTING | WWW.RACKSPACE.COM 10
  11. 11. Xen to XenServerMigrating•  Pick a XenServer host•  Create VDI•  Attach VDI to dom0•  Partition, mkfs, and mount the new virtual disk•  LVM Snapshot original Xen VM and mount it•  rsync snapshot contents to mounted VDI on new host•  Power off original Xen VM and mount its virtual disk•  rsync again to new host to catch changes since snapshotting•  Extract a tar file into mounted VDI containing kernel, ramdisk, agent•  Unmount and detach VDI from dom0•  Create new VM record and attach VDI•  Start the new VM RACKSPACE® HOSTING | WWW.RACKSPACE.COM 11
  12. 12. OpenStack The Basics RACKSPACE® HOSTING | WWW.RACKSPACE.COM 12
  13. 13. OpenStackComponents•  Nova –  Compute controller with various hypervisor drivers•  Glance –  VM Image registry•  Swift –  Object storage•  Quantum/Melange –  Networking•  Cinder –  Volumes/Block storage•  Keystone –  Authentication –  Authorization –  API Endpoint discovery RACKSPACE® HOSTING | WWW.RACKSPACE.COM 13
  14. 14. OpenStackTypical OpenStack Deployment keystone nova glance nova-api glance-api glance-registry RabbitMQ MySQL nova-scheduler MySQL nova-network nova-compute swift RACKSPACE® HOSTING | WWW.RACKSPACE.COM 14
  15. 15. OpenStacknova-compute w/ XenAPI•  Runs in a utility domU on each host•  Images downloaded in dom0 Physical host running XenServer" –  glance XenAPI plugin•  Images attached to nova-compute dom0" nova utility domU" domU –  Partitioning changes glance plugin nova-compute –  file-system resizing agent plugin•  Monitors VMs’ power_state•  Snapshots/backups xenstore plugin nova instance•  VM resize/migration domUs –  rsync to new host•  Communicates w/ agent inside VMs –  Via xenstore RACKSPACE® HOSTING | WWW.RACKSPACE.COM 15
  16. 16. OpenStackXenAPI VM Calls•  VM.start / VM.start_on •  VM.add_to_VCPUs_params•  VM.destroy •  VM.get_VBDs•  VM.clean_reboot •  VM.get_record•  VM.hard_reboot •  VM.get_by_uuid•  VM.clean_shutdown •  VM.get_by_name_label•  VM.hard_shutdown •  VM.set_name_label•  VM.pause •  VM.add_to_xenstore_data•  VM.unpause •  VM.remove_from_xenstore_data•  VM.suspend •  VM.assert_can_migrate•  VM.resume •  VM.migrate_send•  VM.snapshot •  VM.pool_migrate RACKSPACE® HOSTING | WWW.RACKSPACE.COM 16
  17. 17. OpenStackRackspace Specifics RACKSPACE® HOSTING | WWW.RACKSPACE.COM 17
  18. 18. OpenStack Rackspace SpecificsSource Code•  Rackspace tracks trunk –  Deployed code generally less than 2 weeks behind trunk•  Some custom patches on top of trunk –  Features specific to Rackspace –  Custom scheduling to meet Rackspace needs –  Scaling with OpenStack Compute Cells RACKSPACE® HOSTING | WWW.RACKSPACE.COM 18
  19. 19. OpenStack Rackspace SpecificsOpenStack Compute Cells•  Scaling beyond small nova deployments –  Effectively ties multiple nova deployments together•  Developed by Rackspace•  Targeted for Grizzly OpenStack release•  Hierarchical tree –  Multiple parents supported•  API cell(s) at top of the tree•  Compute cell(s) underneath•  DB and RabbitMQ per cell•  Scalability –  Splits connections to DB and RabbitMQ –  Less work to schedule new VM placement RACKSPACE® HOSTING | WWW.RACKSPACE.COM 19
  20. 20. OpenStack Rackspace Specifics OpenStack Nova Multi-cell Diagram API Cell RabbitMQ MySQL nova-api nova-cellsCompute Cell 1" Compute Cell 2" nova-cells nova-cells RabbitMQ RabbitMQ MySQL MySQL nova-scheduler nova-scheduler nova-network nova-network nova-compute nova-compute RACKSPACE® HOSTING | WWW.RACKSPACE.COM 20
  21. 21. Deploying OpenStack RACKSPACE® HOSTING | WWW.RACKSPACE.COM 21
  22. 22. Deploying OpenStack‘Inception’•  Private internal cloud –  Based on Openstack, called iNova –  OpenStack services for public cloud run virtualized under iNova•  Easy to spin up additional services for load balancing –  Enables quick reaction to load spikes•  Easier deployments of new compute cells•  Automated reaction to downed hosts•  Enables new strategies for deployments –  Replace vs Upgrade RACKSPACE® HOSTING | WWW.RACKSPACE.COM 22
  23. 23. Deploying OpenStackiNova Diagram iNova" OpenStack Control Public Cloud Infrastructure for XenServer Public Cloud" hosts XenServer hosts" domUs XenServer host w/ Public Cloud OpenStack Control Infrastructure for iNova" XenServer hosts RACKSPACE® HOSTING | WWW.RACKSPACE.COM 23
  24. 24. Deploying OpenStackContinuous Deployment•  Trunk and custom branches merged multiple times daily –  Unit Tested and Packaged•  Configuration managed with puppet –  Follows same QE and Continuous Deployment rules as code•  QE Environment –  OpenStack control infrastructure deployed on iNova –  nova-compute upgraded on 200 hosts –  Smoke tests•  Staging Deploy –  Uses the same packages and puppet manifests RACKSPACE® HOSTING | WWW.RACKSPACE.COM 24
  25. 25. Future RACKSPACE® HOSTING | WWW.RACKSPACE.COM 25
  26. 26. FutureImprovements desired from Xen•  Better VM resource isolation –  I/O scheduling weights per virtual disk•  Built in live migration without shared storage•  More information about VM resource usage –  Disk usage –  Help us improve OpenStack code –  Help customers react to bad situations•  XenAPI call to write to live xenstore•  Improve XenServer upgrade experience RACKSPACE® HOSTING | WWW.RACKSPACE.COM 26
  27. 27. FutureRackspace’s Other Interests•  Better VDI introduction for new VMs –  Look at VDI.introduce vs mv’ing VDIs and doing an sr-scan•  Incremental backups –  Delta VHDs instead of all VHDs•  Updated plugins using python >= 2.6 –  Cleans up workarounds for no built-in ‘uuid’ module –  Allows use of newer python features like context managers•  Better continuous deployment and testing –  Virtualize all of OpenStack •  Requires spinning XenServer on XenServer •  Wipe out compute nodes for QE environment •  Enable virtual labs for OpenStack developers –  Puppet used to configure XenServer RACKSPACE® HOSTING | WWW.RACKSPACE.COM 27
  28. 28. FutureiNova with Dev/QE Environments OpenStack QE iNova" Environment" OpenStack XenServer Control ‘hosts’ Infrastructure XenServer hosts" OpenStack Dev Environment" domUs OpenStack XenServer Control ‘hosts’ Infrastructure XenServer host w/ OpenStack Control Infrastructure for iNova" RACKSPACE® HOSTING | WWW.RACKSPACE.COM 28
  29. 29. " Chris Behrens" chris.behrens@rackspace.com" RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218 US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM 29
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×