Puppet Release Workflows at Jive Software

2,064
-1

Published on

"Puppet Release Workflows at Jive Software" by Devon Peters at Puppet Camp Portland 2014.

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,064
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
58
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Puppet Release Workflows at Jive Software

  1. 1. Puppet Release Workflows at Jive Devon Peters
  2. 2. What is a Release? 2 © Jive confidential
  3. 3. Push to production, always, immediately 3 © Jive confidential
  4. 4. Releases… •  Software Release Cycle –  “A software release life cycle is the sum of the stages of development and maturity for a piece of computer software: ranging from its initial development to its eventual release…” – Wikipedia •  Release Management –  “the process of managing software releases from development stage to software release” – Wikipedia •  Release Engineering –  “Automation of release management throughout all stages of release” – Me 5 © Jive confidential
  5. 5. More Dev in our Ops RY ATO LIG OB PS EVO D AGE IM 6 © Jive confidential
  6. 6. We have formal release management processes for all of our puppet Domains 7 © Jive confidential
  7. 7. Different Releases for Different Domains •  Core puppet –  All infrastructure –  Many shared services •  Hosted puppet –  Hosted Jive installations –  Puppet is tied closely to Administration tooling •  Continuous Deployment –  New-style services –  Some infrastructure 8 © Jive confidential
  8. 8. Releasing Core Puppet 9 © Jive confidential
  9. 9. Core Puppet •  All core infrastructure systems •  Most shared services •  Puppet+hiera does everything on these systems (very few exceptions) •  Simple deployment environment breakdown –  Dev, QA, Production (multiple production environments) •  Dynamic puppet environments –  ‘production’ puppet environment is default –  ad-hoc environments are used for testing/staging •  Use git flow branching strategy 10 © Jive confidential
  10. 10. Git Details •  Wrappers to simplify and standardize common tasks –  j-tech (j-new, j-hotfix, j-commit, many more) –  Tied into Jira and Crucible –  Feature branches are named after Jira tickets •  Code Deployments –  Dev/QA: develop is deployed to ‘production’ puppet environment –  Prod: master is deployed to ‘production’ puppet environment 12 © Jive confidential
  11. 11. Testing Changes •  Automated –  Pre-commit: puppet/erb/ruby/yaml syntax, puppetlint, hiera-gpg •  Local –  Vagrant VM –  Push branch out to dev puppet master •  Environment Specific –  Push branch to env-specific puppet master 13 © Jive confidential
  12. 12. Merging Changes •  j-review –  Submit a code review of your branch •  j-commit –  Merges feature branch to develop –  Merges hotfix branch to master and develop •  develop branch is deployed to Dev on commit –  jenkins •  develop branch is deployed to QA Mon-Fri @ 10am –  jenkins •  If it’s not a hotfix it won’t go to production yet… –  Technically, this because the change isn’t in the ‘master’ branch yet, but there’s more to it than that 14 © Jive confidential
  13. 13. Change Control for Production •  Bi-weekly CC meetings –  Monday and Thursday •  Puppet changes go through CC process –  Hotfixes can be promoted outside of CC process –  Weekly change windows for high-impact changes •  If it’s a puppet change, it’s done as a hotfix •  Puppet release started every Thursday @ 4pm –  j-release -S: starts a release branch –  Jenkins runs this, and generates a CCR ticket with all commits –  Changes are reviewed in Monday CC meeting •  Puppet release finished every Tuesday –  j-release -F: finishes a release branch (manual) –  Jenkins code deployment jobs are triggered manually 15 © Jive confidential
  14. 14. Core Release Overview 16 © Jive confidential
  15. 15. Releasing Hosted Puppet 17 © Jive confidential
  16. 16. Hosted Puppet •  Nodes that run hosted customer installations •  Very homogenous •  Relatively simple puppet code –  Puppet mostly supplements an in house administration tool (JCA) •  Deployment environment breakdown –  Dev, QA, Prod •  Uses static puppet environments –  ENC dictates the environment for a given node –  Jenkins jobs deploy from git to appropriate puppet environment •  Uses the git flow branching strategy –  All the same j-tech tools 18 © Jive confidential
  17. 17. Testing Changes •  Automated –  Pre-commit: puppet/erb/ruby/yaml syntax, puppetlint, hiera-gpg •  Get your own installation setup in Dev –  Commit changes, and don’t walk away till you know they work •  It’s somewhat acceptable to break dev, but try not to –  Most OS related changes are just plucked from Core, and were likely tested more thoroughly there 19 © Jive confidential
  18. 18. Merging Changes •  j-review –  Submit a code review of your branch •  j-commit –  Merges feature branch to develop –  Merges hotfix branch to master and develop •  develop branch is deployed to Dev on commit –  jenkins –  make sure you don’t break it •  develop branch is deployed to QA ad-hoc –  QA changes are tied to JCA application release QA cycles –  jenkins 20 © Jive confidential
  19. 19. Creating Releases •  Puppet releases are typically tied to JCA application releases –  The app releases every 2 weeks –  A puppet release could be a part of it •  Doing a release –  j-release -A: create, and finish a release branch 21 © Jive confidential
  20. 20. Staged Production Deployments •  UAT –  UAT nodes use the same puppet infrastructure as production –  Jenkins deploys master to the ‘hosted_uat’ puppet environment •  Production –  Jenkins deploys master to the ‘hosted’ puppet environment 22 © Jive confidential
  21. 21. Hosted Release Overview 23 © Jive confidential
  22. 22. Continuous Deployment 24 © Jive confidential
  23. 23. CD Overview •  Deployable –  Framework for deploying java (and other) services •  Many java services (over 70) •  Data Infrastructure –  Kafka, Hadoop, HBase, SenseiDB, Elasticsearch •  Other Infrastructure –  Puppet, Nginx, Sensu, OpenTSDB •  Gerrit –  Code reviews are mandatory –  Branch strategy is still git flow style •  Complex Puppet Code •  Complex Deployment Pipeline 25 © Jive confidential
  24. 24. Environment Overview •  We call them clusters –  virtual, dev, integ, test, release, preprod, prod •  We also have geo-specific datacenters –  local, intinteg, inttest, intrelease, phxpreprod, phxprod, amsprod •  Hiera hierarchy includes all of these –  These exist in hiera for other Puppet domains as well •  Deployable configuration hierarchy includes all of these 26 © Jive confidential
  25. 25. Puppet Details 27 © Jive confidential
  26. 26. Puppet Overview •  Dedicated puppet master(s) per cluster •  Puppet agent is run on-demand by the deployment process •  CD pipeline determines if puppet code can be promoted to the next cluster •  “Special” module and hiera trees –  We don’t want to duplicate everything in Core –  Developers need the ability to change puppet code or hiera data –  We setup something we call Puppet for Projects •  Combines Core puppet code with Project puppet code 28 © Jive confidential
  27. 27. Puppet for Projects - Layout •  Core puppet –  Basic layout is: •  hiera/ •  manifests/ •  modules/ –  manifests/site.pp is basically: hiera_include(‘classes’) –  Every commit triggers an artifact build job (jenkins) •  Artifacts are uploaded to a Nexus repo, as puppet-0.0.1-<count>-<committish> •  Project puppet –  A project is basically a repo –  The project repo contains the following directories: •  puppet/hiera •  puppet/modules –  Contains maven configuration for Core puppet artifact and Combined puppet artifact 29 © Jive confidential
  28. 28. Puppet for Projects – Configuration •  puppet.conf –  modulepath = /path/modules:/path/project/modules •  hiera.yaml –  project/%{some-hierarchy} –  %{some-hierarchy} 30 © Jive confidential
  29. 29. Puppet for Projects - Artifact •  Combined artifact –  A commit to the puppet code in the project triggers a combined artifact build –  Artifact contains: •  •  •  •  •  hiera/ hiera/project manifests/ modules/ project/modules (from Core artifact) (from puppet/hiera in the project repo) (from Core artifact) (from Core artifact) (from puppet/modules in the project repo) –  Module Collisions •  If a module with the same name exists in both, the project always wins and the Core module is excluded from the final artifact 31 © Jive confidential
  30. 30. Deployable 32 © Jive confidential
  31. 31. Deployable Framework •  Provides standardized… –  Configuration •  j-config – CLI or service •  Hierarchical JSON –  Logging •  log-publisher service, writes to Logstash –  Metrics •  metric-publisher service, writes to OpenTSDB –  Monitoring •  Autogenerated sensu checks –  Deployment •  Supports multiple run phases –  Service Management •  j-status, j-start, j-stop 33 © Jive confidential
  32. 32. Puppet as a Deployable •  puppetmaster-deployable –  Target: puppet master –  puppet tree is packaged into a <release-ver> artifact –  Artifact is deployed to /etc/puppet/environments/jive_<release-ver> on the puppet master(s) •  puppet-deployable –  Target: all systems –  j-start executes: puppet agent --environment jive_<release-ver> –  If puppet fails, the deploy fails and stops •  <release-ver> is always the version of the artifact being built/deployed –  0.0.1-<count>-<committish> –  The deployment process converts the string to a puppet safe string •  0_0_1_<count>_<committish> 34 © Jive confidential
  33. 33. Deployment Run Phases •  ops-tools –  Deploy j-tech first so everything else will work •  puppetmaster –  Get out teh codes •  puppet –  Run puppet –  Includes Hadoop, HBase, Zookeeper, Elasticsearch •  pre –  Deploy core/base services –  Includes Kafka, and SenseiDB •  Main –  Everything else 35 © Jive confidential
  34. 34. Making a Puppet Change 36 © Jive confidential
  35. 35. Making a Puppet Change – Virtual •  Vagrant VM –  It’s big •  minimum 4CPU, 8GB – for the VM alone –  Full stack gets deployed and run •  Hadoop, Kafka, etc, etc, etc – even all 70+ services if you want –  j-vm -r •  Fetches proper Core puppet artifact •  Builds and deploys a puppet tree for vagrant, based on your local git repo •  Executes vagrant puppet provisioner –  Once it works on the vm, submit a review to Gerrit 37 © Jive confidential
  36. 36. Making a Puppet Change – Integ •  Once the review is submitted, jenkins will: –  Build Artifacts –  Launch a VM-test job •  Validating that what you did works on a vagrant VM there –  Launch an integ deployment •  Validating that things work in a multi-node non-vagrant environment –  Comment on your review with Validated +1, or -1 (depending on results) –  Once other reviewers give you +2 Gerrit will merge your commit –  Once it’s merged… 38 © Jive confidential
  37. 37. Making a Puppet Change – Test •  Once the commit is merged, jenkins will: –  Build artifacts –  Launch a Test deployment –  Run more extensive tests –  If the deployment and all tests pass, it’s ready for the next step 39 © Jive confidential
  38. 38. Making a Puppet Change – Release •  Daily at 8am all commits that have passed Test are merged from develop to master •  Once this happens, jenkins will: –  Build artifacts –  Trigger a Release deployment –  Rerun all of the tests –  If all tests pass… 40 © Jive confidential
  39. 39. Making a Puppet Change – Preprod •  If all tests pass, jenkins will: –  Promote artifacts to preprod Nexus repo –  Trigger a Preprod deploy –  At this point, everything should be stable 41 © Jive confidential
  40. 40. Making a Puppet Change – Prod •  During the next scheduled production release, someone will: –  Trigger a Prod deploy –  Currently done manually 42 © Jive confidential
  41. 41. That’s about it… 44 © Jive confidential
  42. 42. Review •  Core –  Complex puppet code to manage everything –  Releases tied to Change Control –  ~1000 nodes •  Hosted –  Relatively simple puppet code –  Releases tied to Administration tool’s application releases –  ~14000 nodes •  Continuous Deployment –  Complex puppet code –  Fully Automated Release and Deployment –  <200 nodes (but growing) 45 © Jive confidential
  43. 43. Thank You! © Jive confidential 46
  44. 44. Questions? © Jive confidential 47
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×