Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud

2,250 views

Published on

Our pipeline that builds on individual configuration management modules to layer up to a full multi-node dev environment and on to production deploys

Published in: Technology
  • Be the first to comment

CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud

  1. 1. CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud Simon McCartney, Site Reliability Engineering OpenStack Summit 2014 - Paris © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  2. 2. Background ● This project predates HP Helion OpenStack ● We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  3. 3. Why? © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  4. 4. Why Continuous Integration & Delivery? • Repeatable, Reliable & Incrementally Improving • Constant flow of changes to benefit users © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  5. 5. Why Continuous Integration & Delivery? ● Infrastructure as Code > Infrastructure as Art ● Frequent Small Batches > Infrequent Big Batches © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  6. 6. Why Continuous Integration & Delivery? ● Time Vampires ● Manual Builds ● Manual Testing ● Improved consistency & quality ● Faster Cycle Time © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  7. 7. Why Continuous Integration & Delivery ● Unit Tests - salt formulae ● Integration Tests - do these salt formulae work together? ● End-to-End Tests - does the OpenStack Cloud we built work? ● Performance Test - did we break/de-tune something? ● Deployment Tests - do we have everything? ● Configuration Tests - can we test different environments before hitting the real environment? © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  8. 8. Our Pipeline © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  9. 9. Our Tech Stack ● Vendor packages for OpenStack (Ubuntu) ● Configuration management (SaltStack) ● Software Engineering ● git, gerrit, gitshelf & jenkins ● test-kitchen (kitchen-salt, serverspec) ● Infrastructure Engineering ● Vagrant/VirtualBox & Public Cloud © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  10. 10. Our Pipeline 1. Individual Salt Formula development 2. Personal multi-node, package based OpenStack environment on your workstation for dev & validation 3. Go public – push to gerrit for review & automated testing 4. Pull in to the deploy-kit 5. deploy-kit tooling kicks in & builds deploy artefacts 6. Auto deploy to ephemeral public cloud test environment 7. Deploy to physical staging environment 8. Ready for production © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  11. 11. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  12. 12. Validating & Approving a change • gerrit - defacto web based open source code review system • submit changes (a patch set) for review by your peers.. • ..and for testing by Jenkins • Jenkins & Gerrit Trigger • Gerrit plugins allow certain jobs to be triggered on patch-set events (create, merge etc) • results of these jobs are then recorded in gerrit • test-kitchen • test harness to execute your configuration management code in isolation (i.e. fresh VMs) • providers (vagrant, LXC, openstack, gce, aws etc) • provisioners (Chef, Salt, Puppet) • testing frameworks (Bats, RSpec, serverspec etc) • can be used interactively in your dev environment as well as in validation jobs © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  13. 13. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  14. 14. In-cloud validation 1. contractor creates instances & networks 2. bootstrap.sh used prepare vanilla image for use with salt 3. Jenkins uploads release artifacts and deploys salt states 4. Performa validation tests "roles": { ... "compute": { "image": "261844b3-479c-...", "flavor": "101", "keypair": "jenkins", "instances": { "ae1": [{ "az": "az3", "number": 1, "nics": [{ © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. "network": "public-net", "fixed_ip": "172.16.0.101", "floating_ip": "15.126.241.109" }, { "network": "private-net", "fixed_ip": "10.0.0.101" }]
  15. 15. Deployment Automation • Production Deploys • Move slowly when required • Confirm host is still “good” using existing monitoring framework • Caution over nova-* service restarts • Next • rundeck • chatops © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  16. 16. Links ● http://devops.com/blogs/continuous-delivery-pipeline/ ● http://martinfowler.com/bliki/DeploymentPipeline.html ● https://github.com/secure-pipeline/jenkins-example ● https://speakerdeck.com/garethr/continuous-integration-for-infrastructure ● https://github.com/gitshelf/gitshelf ● https://github.com/moniker-dns/contractor ● https://github.com/macgreagoir/gobstack © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  17. 17. Questions? © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  18. 18. Thank you! Simon McCartney @simonmcc & Mick Gregg mickgregg.com © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

×