Learning to Scale 
OpenStack 
Build Release Deploy System Evolution in 
Juno 
Created by: Rainya Mosher 
Modified Date: 05 NOV 2014 
Classification: Public 
Twitter: @rainyamosher | Freenode: rainya
In preparing for battle, I have always found that 
plans are useless, but planning is indispensable. 
~ Dwight D. Eisenhower, 34th President of the United States 
www.rackspace.com 2
www.rackspace.com 3 Space Unicorn, 2012, Gripp & Drouhard 
http://youtu.be/17o1OlroNSE
www.rackspace.com 4 Miltonious Blog Unicorn of Technical 
Difficulities, Source Unknown
www.rackspace.com 5 
5 time zones 
6 production regions 
6+ lower level environments 
(CI, Test, PreProd) 
20,000+ hypervisors 
20,000+ computes 
2,000+ control plane nodes
www.rackspace.com 6 
User Request Nova, Glance, Neutron 
Services 
OpenStack 
Magic 
XenServer, OVS 
e.g., Build 
Instance 
User Access to 
Cloud Instance 
Build Release 
Deploy System 
Upgrade 
OpenStack 
Services 
CONTROL PLANE (CP) 
DATA PLANE (CP) 
Customer 
VM
To date, the control plane has been 
upgraded 11 times in 2014. 
www.rackspace.com 7
2014 Control Plane Upgrades 
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 
Iteration 9 
2 deploys 
Migrate to Neutron 
Iteration 10 
6 deploys 
Boot from Volume 
Iteration 11 
3 deploys (planned) 
Expose Public Neutron API 
Iteration 8 
3 deploys 
Expose Public Glance API
www.rackspace.com 9
www.rackspace.com 10 
Upstream Nova, Glance, and 
Neutron project code
www.rackspace.com 11 
PLY 
Git-Based Patch Management 
https://github.com/rconradharris/ply
www.rackspace.com 12 
Transitioning from Puppet to Ansible 
on a project-by-project basis
www.rackspace.com 13 
Upstream, launchpad.net 
Internally, mixed use of: 
• Redmine 
http://www.redmine.org 
• Jira 
https://jira.atlassian.com
www.rackspace.com 14 
On Github Pull Request: 
• runs unit tests 
• validates issue # in commit 
• updates issue tracker link
www.rackspace.com 15 
Internal GitHub provides 
• secure place for sensitive settings 
• sandbox to move product 
development forward
www.rackspace.com 16 
Package artifact 
• Tarball of project code virtual 
environment + configs 
• .tar artifact uploaded to package 
server 
• distributed via torrent
www.rackspace.com 17 
Ansible playbooks triggered by 
Jenkins with python goodness 
• Upgrade control plane 
• Perform database changes 
• Upgrade computes
www.rackspace.com 18
www.rackspace.com 19
Iteration Change and Release Management 
www.rackspace.com 20 
Iteration CI/CD 
Iteration Cycle
www.rackspace.com 21
www.rackspace.com 22
www.rackspace.com 23
www.rackspace.com 24 
• Not production scale 
• Shared train leads to blocking issues 
• Multiple trains = coordination overhead 
• Production deploys stop the flow
The term "Microservice Architecture" has sprung up 
over the last few years to describe a particular way of 
designing software applications as suites of 
independently deployable services [with] certain 
common characteristics […] around business capability, 
automated deployment, intelligence in the endpoints, 
and decentralized control of languages and data. 
Martin Fowler 
http://martinfowler.com 
www.rackspace.com 25
•The Road to Minimally Impacting 
Live Upgrades of the Rackspace 
Public Cloud 
–TODAY from11:50 – 12:30 
– Room 252AB 
• Building the RackStack: Packaging 
from Upstream OpenStack 
–TODAY from 15:30 – 16:10 
– Amphitheatre Bleu 
26 
Companion Sessions 
www.rackspace.com
Our highest priority is to satisfy the customer 
through early and continuous delivery of 
valuable software. 
~ Principles Behind the Agile Manifesto 
http://www.agilemanifesto.org/principles.html 
www.rackspace.com 27
THANK YOU 
Twitter: @rainyamosher | Freenode: rainya

Learning to Scale OpenStack: Juno Update from the Rackspace Public Cloud

  • 1.
    Learning to Scale OpenStack Build Release Deploy System Evolution in Juno Created by: Rainya Mosher Modified Date: 05 NOV 2014 Classification: Public Twitter: @rainyamosher | Freenode: rainya
  • 2.
    In preparing forbattle, I have always found that plans are useless, but planning is indispensable. ~ Dwight D. Eisenhower, 34th President of the United States www.rackspace.com 2
  • 3.
    www.rackspace.com 3 SpaceUnicorn, 2012, Gripp & Drouhard http://youtu.be/17o1OlroNSE
  • 4.
    www.rackspace.com 4 MiltoniousBlog Unicorn of Technical Difficulities, Source Unknown
  • 5.
    www.rackspace.com 5 5time zones 6 production regions 6+ lower level environments (CI, Test, PreProd) 20,000+ hypervisors 20,000+ computes 2,000+ control plane nodes
  • 6.
    www.rackspace.com 6 UserRequest Nova, Glance, Neutron Services OpenStack Magic XenServer, OVS e.g., Build Instance User Access to Cloud Instance Build Release Deploy System Upgrade OpenStack Services CONTROL PLANE (CP) DATA PLANE (CP) Customer VM
  • 7.
    To date, thecontrol plane has been upgraded 11 times in 2014. www.rackspace.com 7
  • 8.
    2014 Control PlaneUpgrades Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Iteration 9 2 deploys Migrate to Neutron Iteration 10 6 deploys Boot from Volume Iteration 11 3 deploys (planned) Expose Public Neutron API Iteration 8 3 deploys Expose Public Glance API
  • 9.
  • 10.
    www.rackspace.com 10 UpstreamNova, Glance, and Neutron project code
  • 11.
    www.rackspace.com 11 PLY Git-Based Patch Management https://github.com/rconradharris/ply
  • 12.
    www.rackspace.com 12 Transitioningfrom Puppet to Ansible on a project-by-project basis
  • 13.
    www.rackspace.com 13 Upstream,launchpad.net Internally, mixed use of: • Redmine http://www.redmine.org • Jira https://jira.atlassian.com
  • 14.
    www.rackspace.com 14 OnGithub Pull Request: • runs unit tests • validates issue # in commit • updates issue tracker link
  • 15.
    www.rackspace.com 15 InternalGitHub provides • secure place for sensitive settings • sandbox to move product development forward
  • 16.
    www.rackspace.com 16 Packageartifact • Tarball of project code virtual environment + configs • .tar artifact uploaded to package server • distributed via torrent
  • 17.
    www.rackspace.com 17 Ansibleplaybooks triggered by Jenkins with python goodness • Upgrade control plane • Perform database changes • Upgrade computes
  • 18.
  • 19.
  • 20.
    Iteration Change andRelease Management www.rackspace.com 20 Iteration CI/CD Iteration Cycle
  • 21.
  • 22.
  • 23.
  • 24.
    www.rackspace.com 24 •Not production scale • Shared train leads to blocking issues • Multiple trains = coordination overhead • Production deploys stop the flow
  • 25.
    The term "MicroserviceArchitecture" has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services [with] certain common characteristics […] around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data. Martin Fowler http://martinfowler.com www.rackspace.com 25
  • 26.
    •The Road toMinimally Impacting Live Upgrades of the Rackspace Public Cloud –TODAY from11:50 – 12:30 – Room 252AB • Building the RackStack: Packaging from Upstream OpenStack –TODAY from 15:30 – 16:10 – Amphitheatre Bleu 26 Companion Sessions www.rackspace.com
  • 27.
    Our highest priorityis to satisfy the customer through early and continuous delivery of valuable software. ~ Principles Behind the Agile Manifesto http://www.agilemanifesto.org/principles.html www.rackspace.com 27
  • 28.
    THANK YOU Twitter:@rainyamosher | Freenode: rainya

Editor's Notes

  • #3 This is one of my favorite quotes. Each six month cycle of OpenStack is a bit like a battle, where we vie with ourselves and the Community to deliver something exceptional. I have a whole bunch of letters I can put after my name to certify I know how to plan projects and implement effective software development processes, but I know from time in the real world as a software developer that a plan becomes useless as soon as it is created. The real value comes from the process of planning, the conversations you have with the important people involved in whatever you are trying to accomplish. The whole OpenStack Design Summit is a testament to how important planning is, even though after another six months of internal and external collaboration and conflict, what we deliver as a community next Spring may only loosely resemble what we plan this week.
  • #4 Intent of Talk: to bring everyone up to speed on the last six months at Rackspace and provide a framework to leverage and adapt should you find yourself needing to upgrade a production cloud at scale yourself. Space unicorn Soaring through the stars Delivering the rainbows All around the world Source: http://www.kotzendes-einhorn.de/blog/2012-12/das-space-unicorn-liefert-immernoch-weltweit-regenbogen-aus/
  • #5 When I’m honest with myself, the best laid plans of mice and men, women, and unicorns generally end up looking a bit like this. We start with unicorns and rainbows and “this is going to be great!” at the start with a tangled mess of “didn’t expect that to happen” somewhere in the middle before we get to the end state. For the OpenStack community – operators, users, developers – I get the sense that we’re in the tangle right now after four years of unicorns and rainbows.
  • #27 We were fairly successful in collaborating on our solutions. My coworker, Chris Blumentritt, will be diving in to the work we’ve done in conjunction with the Cloud Servers development team to implement nova-conductor and graceful compute shutdowns on Wednesday at 11:50. At 3:30 on Wednesday, teammates from the build release deploy systems team will breakdown how we handle the packaging from upstream OpenStack.