The document outlines a CI/CD pipeline for deploying and maintaining an OpenStack IaaS cloud, detailing the technologies and processes involved. It emphasizes the importance of continuous integration and delivery for ensuring reliability and improved quality in deployment cycles. Key components include development practices, testing methodologies, and the use of tools like Gerrit and Jenkins for validation and automation.

1. CI/CD Pipeline to
Deploy and Maintain an
OpenStack IaaS Cloud
Simon McCartney, Site Reliability Engineering
OpenStack Summit 2014 - Paris
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

2. Background
● This project predates HP Helion OpenStack
● We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

3. Why?
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

4. Why Continuous Integration & Delivery?
• Repeatable, Reliable & Incrementally Improving
• Constant flow of changes to benefit users
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

5. Why Continuous Integration & Delivery?
● Infrastructure as Code > Infrastructure as Art
● Frequent Small Batches > Infrequent Big Batches
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

6. Why Continuous Integration & Delivery?
● Time Vampires
● Manual Builds
● Manual Testing
● Improved consistency & quality
● Faster Cycle Time
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

7. Why Continuous Integration & Delivery
● Unit Tests - salt formulae
● Integration Tests - do these salt formulae work together?
● End-to-End Tests - does the OpenStack Cloud we built work?
● Performance Test - did we break/de-tune something?
● Deployment Tests - do we have everything?
● Configuration Tests - can we test different environments before
hitting the real environment?
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

8. Our Pipeline
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

9. Our Tech Stack
● Vendor packages for OpenStack (Ubuntu)
● Configuration management (SaltStack)
● Software Engineering
● git, gerrit, gitshelf & jenkins
● test-kitchen (kitchen-salt, serverspec)
● Infrastructure Engineering
● Vagrant/VirtualBox & Public Cloud
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

10. Our Pipeline
1. Individual Salt Formula development
2. Personal multi-node, package based OpenStack environment on your
workstation for dev & validation
3. Go public – push to gerrit for review & automated testing
4. Pull in to the deploy-kit
5. deploy-kit tooling kicks in & builds deploy artefacts
6. Auto deploy to ephemeral public cloud test environment
7. Deploy to physical staging environment
8. Ready for production
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

11. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

12. Validating & Approving a change
• gerrit - defacto web based open source code review system
• submit changes (a patch set) for review by your peers..
• ..and for testing by Jenkins
• Jenkins & Gerrit Trigger
• Gerrit plugins allow certain jobs to be triggered on patch-set events (create,
merge etc)
• results of these jobs are then recorded in gerrit
• test-kitchen
• test harness to execute your configuration management code in isolation (i.e.
fresh VMs)
• providers (vagrant, LXC, openstack, gce, aws etc)
• provisioners (Chef, Salt, Puppet)
• testing frameworks (Bats, RSpec, serverspec etc)
• can be used interactively in your dev environment as well as in validation jobs
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

13. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

14. In-cloud validation
1. contractor creates instances &
networks
2. bootstrap.sh used prepare vanilla
image for use with salt
3. Jenkins uploads release artifacts and
deploys salt states
4. Performa validation tests
"roles": {
...
"compute": {
"image": "261844b3-479c-...",
"flavor": "101",
"keypair": "jenkins",
"instances": {
"ae1": [{
"az": "az3",
"number": 1,
"nics": [{
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
"network": "public-net",
"fixed_ip": "172.16.0.101",
"floating_ip": "15.126.241.109"
},
{
"network": "private-net",
"fixed_ip": "10.0.0.101"
}]

15. Deployment Automation
• Production Deploys
• Move slowly when required
• Confirm host is still “good” using existing monitoring framework
• Caution over nova-* service restarts
• Next
• rundeck
• chatops
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

16. Links
● http://devops.com/blogs/continuous-delivery-pipeline/
● http://martinfowler.com/bliki/DeploymentPipeline.html
● https://github.com/secure-pipeline/jenkins-example
● https://speakerdeck.com/garethr/continuous-integration-for-infrastructure
● https://github.com/gitshelf/gitshelf
● https://github.com/moniker-dns/contractor
● https://github.com/macgreagoir/gobstack
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

17. Questions?
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

18. Thank you!
Simon McCartney
@simonmcc
&
Mick Gregg
mickgregg.com
© Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.