Rackspace uses Chef and other operational tools to automate the deployment of OpenStack. They developed their own cookbooks to model real-world deployments across multiple operating systems and to handle updates and new OpenStack services. Over time, they added capabilities like high availability and vendor integration. To simplify deployment and reduce operator overhead, they created OpenCenter, which automates tasks through a solver and provides an API. OpenCenter lowers the OpenStack knowledge needed by operators.

1. 1
April 26, 2013
Deploying OpenStack with Chef and Operational Tooling
#chefconf 2013

2. RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Who Are We?
Justin Shepherd
Principal Architect
Rackspace Private Cloud
github.com/galstrom21
Joseph Breu
Deployment Engineer
Rackspace Private Cloud
github.com/rackerjoe
@rackerjoe

3. 3RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Only consume Open Source projects
• All changes go upstream
• CHEF-3889 - tunables for postgres
• CHEF-3849 - redirect login bug
• Everything must be automated through the deployment and
management stacks.
• Everything we build is Open Souce
• https:/github.com/rcbops
• github.com/rcbops-cookbooks
Rackspace Private Cloud Philosophy

4. 4RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Jenkins is used for testing and gating of our cookbooks
• Syntax Verification (pep8, foodcritic)
• Unit Tests for OpenCenter
• Full deployment of OpenStack utilizing the cookbooks with
the proposed changeset applied
• Functional Tests of OpenStack
• OpenStack API Testing
• OpenStack CLI Testing
• Upstream opscode cookbooks are tested before inclusion
Testing

5. 5RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How Did We Get Here?
5

6. 6RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Installing Bexar and Cactus by hand
• Problems
• Deployed from trunk
• No packages available for distributions
• Not easily repeatable
• Not easily supported
In The Beginning
Early 2011

7. 7RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Defining Our Mission
7

8. 8RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We need to make it easy to install
• We need to make the installation repeatable
• We need to model “Real World Deployments”
The Mission
Early 2011

9. 9RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Automating OpenStack Deployment
9

10. 10RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Deploying Cactus and Diablo
• Puppet
• Inherited from OpenStack Team
• djeep based
• Very simplistic, not flexible
• Represented the developer case - not real world deployments
10
Automate!
Late 2011

11. 11RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Mission Status
We need to make it easy to install FAIL
We need to make the installation repeatable PASS
We need to model “Real World Deployments” FAIL

12. 12RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Crowbar
• Dell platform for deployment
• Chef based
• Very heavyweight
• ISO installation
• Boot shim for system discovery
• Used for a few deployments and contributed code upstream
• Upgrades were painful or not possible at all
12
Automate!
Late 2011 - Diablo

13. 13RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Mission Status
We need to make it easy to install Almost Passed
We need to make the installation repeatable PASS
We need to model “Real World Deployments” Almost Passed

14. 14RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Feature Creep
14

15. 15RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We need to make it easy to install
• We need to make the installation repeatable
• We need to mode “Real World Deployments”
• We need to handle updates
• We need to quickly consume new OpenStack services
• We need to plug into existing infrastructure (PXE, DNS, LDAP, etc)
15
We now need to…
March 2012

16. 16RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did Puppet do?
We need to make it easy to install FAIL
We need to make the installation repeatable PASS
We need to model “Real World Deployments” FAIL
We need to handle updates FAIL
We need to quickly consume new OpenStack Services FAIL
We need to plug into existing infrastructure FAIL

17. 17RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did Crowbar do?
We need to make it easy to install Almost Passed
We need to make the installation repeatable PASS
We need to model “Real World Deployments” Almost Passed
We need to handle updates FAIL
We need to quickly consume new OpenStack Services FAIL
We need to plug into existing infrastructure FAIL

18. 18RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Next Steps in Automation
18

19. 19RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Deploying Essex
• Started to build our own cookbooks
• Heavy utilization of Chef search
• Modeled real world deployments
• Support for multiple Operating Systems
Next Steps in Automation
March 2012

20. 20RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Deploying Folsom
• Creating LWRPs for REST based Services
• Rewrites of many cookbooks to support multiple Operating Systems
• Debian and RedHat based
• Search wrapper functions for ease of use
Next Steps in Automation
Late 2012

21. 21RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did we do?
We need to make it easy to install PASSED
We need to make the installation repeatable PASSED
We need to model “Real World Deployments” PASSED
We need to handle updates PASSED
We need to quickly consume new OpenStack Services PASSED
We need to plug into existing infrastructure Almost PASSED

22. 22RACKSPACE® HOSTING | WWW.RACKSPACE.COM
More Feature Creep
22

23. 23RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We need to make it easy to install
• We need to make the installation repeatable
• We need to mode “Real World Deployments”
• We need to handle updates
• We need to quickly consume new OpenStack services
• We need to plug into existing infrastructure (PXE, DNS, LDAP, etc)
• We need provide Highly Available OpenStack Services
• We need to provide support for Vendor Integration for Block Storage
• We need to support Software Defined Networks
23
We now need to…
December 2012

24. 24RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• keepalived for failover IP
• haproxy for load balancing REST based services
• Multi-master MySQL replication and keepalived
• Active/Passive RabbitmQ and keepalived
• This is sketchy in Folsom - better in Grizzly
Creating Highly Available Services
Early 2013

25. 25RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Support for multiple Cinder Backends
• Currently NetApp and EMC
• Support for SDN
• Currently only OVS
• LDAP Support
• Greenfield LDAP deployments only
• No AD support... yet...
Other Services
Early 2013

26. 26RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did we do?
We need to make it easy to install FAILED!
We need to make the installation repeatable PASSED
We need to model “Real World Deployments” PASSED
We need to handle updates PASSED
We need to quickly consume new OpenStack Services PASSED
We need to plug into existing infrastructure PASSED
We need to provide Highly Available OpenStack
services
PASSED
We need to provide support for Vendor Integration for
Block Storage
PASSED
We need to Support Software Defined Networks PASSED

27. 27RACKSPACE® HOSTING | WWW.RACKSPACE.COM
The Tipping Point
27

28. 28RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• We had a system that can do real interesting things
• You now need to have staff with strong Chef knowledge to use it
• And strong OpenStack knowledge...
• OpenSource Chef is great for some things – but not everything
• We need to orchestrate changes across a cluster
• We need to do it in a defined order
The Tipping Point
Early 2013

29. 29RACKSPACE® HOSTING | WWW.RACKSPACE.COM
OpenCenter
29

30. 30RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Simplifies, speeds up deployment
• Lowers Operator overhead
• Minimizes the internal OpenStack knowledge
• Provides resiliency in the Cloud
• API for command line access
What is OpenCenter?

31. 31RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is OpenCenter?

32. 32RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is OpenCenter

33. 33RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Lowers Operator Overhead

34. 34RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Resiliency for Private Cloud

35. 35RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Solver – Automates Operator Tasks

36. 36RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Extensible
Write your own Python/Bash scripts to create
workflows that extend OpenCenter

37. 37RACKSPACE® HOSTING | WWW.RACKSPACE.COM
How did we do?
We need to make it easy to install PASSED
We need to make the installation repeatable PASSED
We need to model “Real World Deployments” PASSED
We need to handle updates PASSED
We need to quickly consume new OpenStack Services PASSED
We need to plug into existing infrastructure PASSED
We need to provide Highly Available OpenStack
services
PASSED
We need to provide support for Vendor Integration for
Block Storage
PASSED
We need to Support Software Defined Networks PASSED

38. 38RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Errata
38

39. 39RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Partial Searches and Templates
• Remove dependencies on monitoring and logging
• Make our cookbooks more easily consumable
• Upstream our cookbooks to the OpsCode OpenStack repos
• Utilize OpenStack CI testing and gating
Looking forward

40. 4040
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM