Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Experiences from Running Masterless Puppet - PuppetConf 2014


Published on

Experiences from Running Masterless Puppet - Erik Dalén, Spotify

Published in: Technology

Experiences from Running Masterless Puppet - PuppetConf 2014

  1. 1. September 24, 2014 Experiences from running masterless Puppet Erik Dalén
  2. 2. whoami • System engineer at Spotify since almost 3 years. • Community contributor to Puppet. • Author of Puppet Explorer web UI. • Author of the puppetdbquery, dnsquery and puppet ls modules. 2
  3. 3. Agenda • Background about our puppet setup • How we implemented masterless • Why we did the switch to masterless • New workflows and future improvements 3
  4. 4. Background • Puppet users since 4 years • Two large Puppet installations, 20+ Puppet masters. • A single repository with all modules • Git branches mapped to Puppet environments • Code review in Gerrit to merge into production branch • Tests using AWS virtual machines on every code review 4
  5. 5. Section name Implementation Started with just switching to the same workflow but masterless. Using a custom Ruby “wrapper” around puppet apply. 5 Gerrit SITE X SITE X git mirror git mirror SITE Y git mirror git mirror PuppetDB Node Node Node ● ServerDB ● LDAP Gerrit replication Gerrit replication git pull git pull failover if pod mirrors are unavailable Queries * Queries * Store facts * Store catalog * Store report hiera secrets hiera secrets HTTPS GET auth by cert Puppet CA request certificate
  6. 6. Benefits of this setup • Debug logging of compilation and hiera lookups • Ability to get traceback of custom functions directly on the node • Deprecation warnings from compilation visible on the node $ sppuppet hiera resolvconf::servers 6
  7. 7. Drawbacks • Requires more configuration, and can’t use puppet to manage it. • Quite specialised. 7
  8. 8. Secret management • Small web service where nodes authenticates using their SSL cert • Calculates the facts for the node from the certificate • Sends the secret hiera data that applies to that particular node 8
  9. 9. PuppetDB access control • Very binary per default, allow everyone or a whitelist 9 Node Apache PuppetDB mod_ext_filter jq
  10. 10. Why switch to masterless? • Scaling the Puppet masters not an issue • But scaling the workflow is an issue 10
  11. 11. Growing amount of committers 11 Number of monthly committers 220 165 110 55 0 2 years ago 1 year ago now
  12. 12. Growing amount of modules 12 Number of modules 600 450 300 150 0 2 years ago 1 year ago now
  13. 13. More complex dependencies • 126 modules using the apache module • 91 modules using the nginx module • 92 modules using the postgresl module Backwards incompatible changes to shared modules almost impossible to do. 13
  14. 14. Flexible workflow needed • Allow one set of modules per Puppet run determined by hiera • Easier ways to do continuous delivery of both application and configuration • Gradual rollout of new module versions 14
  15. 15. Why not r10k or librarian-puppet? Both great utilities that allow building a environment dynamically. But it is still a fixed environment for many nodes. production_new_apache production_new_apache_new_postgres production_old_apache_new_postgres production_new_apache_old_postgres 15
  16. 16. Our solution to the problem At each puppet run we look in hiera data which modules to install. 16 hiera/common.yaml: modules: spotify-hostbase: latest hiera/role/spotify_web.yaml modules: spotify-apache: latest hiera/role/puppetexplorer.yaml modules: puppetlabs-apache: latest
  17. 17. Internal forge • A simple forge implementation using the new V3 forge API • Mirrors the upstream forge • Will be used for distributing internal modules as well 17
  18. 18. Questions?