Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A CI/CD Pipeline to Deploy and Maintain OpenStack - cfgmgmtcamp2015

1,848 views

Published on

An intro into the pipeline & related tools we built to build a CI/CD pipeline for building and maintaining a package based OpenStack installation, with realistic, portable multi-machine development environments.

Published in: Internet
  • Be the first to comment

A CI/CD Pipeline to Deploy and Maintain OpenStack - cfgmgmtcamp2015

  1. 1. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Simon McCartney, Site Reliability Engineering cfgmgmtcamp 2015 CI/CD Pipeline to Deploy and Maintain an OpenStack IaaS Cloud
  2. 2. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Who is this guy anyway? ● Dev, then Ops, then Management, back to WebOps ● Solaris & Makefiles ● CentOS w/ kickstart & hairy bash/php → puppet ● Chef ● Salt ● Likes to take the best tools to the next job (kitchen-salt)
  3. 3. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Background ● This project predates HP Helion OpenStack ● We’re using Ubuntu 12.04, OpenStack Grizzly & SaltStack 0.16.2
  4. 4. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why?
  5. 5. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? • Repeatable, Reliable & Incrementally Improving • Constant flow of changes to benefit users
  6. 6. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? ● Infrastructure as Code > Infrastructure as Art ● Frequent Small Batches > Infrequent Big Batches
  7. 7. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery? ● Time Vampires ● Manual Builds ● Manual Testing ● Improved consistency & quality ● Faster Cycle Time
  8. 8. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Why Continuous Integration & Delivery ● Unit Tests - salt formulae ● Integration Tests - do these salt formulae work together? ● End-to-End Tests - does the OpenStack Cloud we built work? ● Performance Test - did we break/de-tune something? ● Deployment Tests - do we have everything? ● Configuration Tests - can we test different environments before hitting the real environment?
  9. 9. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Pipeline
  10. 10. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Tech Stack ● Vendor packages for OpenStack (Ubuntu) ● Configuration management (SaltStack) ● Software Engineering ● git, gerrit, gitshelf & jenkins ● test-kitchen (kitchen-salt, serverspec) ● Infrastructure Engineering ● Vagrant/VirtualBox & Public Cloud
  11. 11. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Our Pipeline 1. Individual Salt Formula development 2. Personal multi-node, package based OpenStack environment on your workstation for dev & validation 3. Go public – push to gerrit for review & automated testing 4. Pull in to the deploy-kit 5. deploy-kit tooling kicks in & builds deploy artefacts 6. Auto deploy to ephemeral public cloud test environment 7. Deploy to physical staging environment 8. Ready for production
  12. 12. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  13. 13. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Validating & Approving a change • gerrit - defacto web based open source code review system • submit changes (a patch set) for review by your peers.. • ..and for testing by Jenkins • Jenkins, Jenkins Job Builder & Gerrit Trigger • Gerrit plugins allow certain jobs to be triggered on patch-set events (create, merge etc) • results of these jobs are then recorded in gerrit • test-kitchen • test harness to execute your configuration management code in isolation (i.e. fresh VMs) • providers (vagrant, LXC, openstack, gce, aws etc) • provisioners (Chef, Salt, Puppet) • testing frameworks (Bats, RSpec, serverspec etc) • can be used interactively in your dev environment as well as in validation jobs
  14. 14. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Jenkins Job Builder ● Because nobody like XML ● Generate Jenkins Jobs from re-useable & composable YAML fragments
  15. 15. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  16. 16. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  17. 17. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. gitshelf ● Accidentally re-invented the wheel ● BUT, it's still very useful ● ~= Librarian Puppet or Berkshelf or dependencies.txt ● Specify specific versions of the contents of a git repo at specific location (tag, branch, SHA1 etc) ● Also supports variable expansion & symlink management ● You should probably use repo from Google's AOSP
  18. 18. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.
  19. 19. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. In-cloud validation 1. contractor creates instances & networks 2. bootstrap.sh used prepare vanilla image for use with salt 3. Jenkins uploads release artifacts and deploys salt states 4. Performa validation tests "roles": { ... "compute": { "image": "261844b3-479c-...", "flavor": "101", "keypair": "jenkins", "instances": { "ae1": [{ "az": "az3", "number": 1, "nics": [{ "network": "public-net", "fixed_ip": "172.16.0.101", "floating_ip": "15.126.241.109" }, { "network": "private-net", "fixed_ip": "10.0.0.101" }]
  20. 20. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Deployment Automation • Production Deploys • Move slowly when required • Confirm host is still “good” using existing monitoring framework • Caution over nova-* service restarts • Next • rundeck • chatops
  21. 21. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Links ● http://devops.com/blogs/continuous-delivery-pipeline/ ● http://martinfowler.com/bliki/DeploymentPipeline.html ● https://github.com/secure-pipeline/jenkins-example ● https://speakerdeck.com/garethr/continuous-integration-for-infrastructure ● https://github.com/gitshelf/gitshelf ● https://github.com/moniker-dns/contractor ● https://github.com/macgreagoir/gobstack
  22. 22. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Questions?
  23. 23. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice. Thank you! Simon McCartney @simonmcc
  24. 24. © Copyright 2013 Hewlett-Packrd Development Company, L.P. The information contained herein is subject to change without notice.

×