Running at Scale: Practical Performance Tuning with Puppet - PuppetConf 2013

4,080 views

Published on

"Running at Scale: Practical Performance Tuning with Puppet" by Sam Kottler Engineer, Red Hat.

Presentation Overview: This session will talk about some production issues I've seen running Puppet in large environments. From how to manage a single master with hundreds of hosts to real-life patterns for building high availability clusters that scale to 10's of thousands of agents. Another important topic that will be covered is how to deploy networked filesystems that perform well under high load and streaming files to many hosts simultaneously.

Speaker Bio: Sam Kottler is a software engineer in the Virtualization R&D group at Red Hat. He's helped build infrastructure for leading startups, including Digg.com, Acquia, and Venmo and is a contributor to Puppet, the Fedora Project, Drupal, and the Rubygems.org. Sam speaks around the world on the topics of internet security, systems automation, and software architecture.

  • Be the first to comment

Running at Scale: Practical Performance Tuning with Puppet - PuppetConf 2013

  1. 1. Sam Kottler | Puppet at Scale1 Puppet at Scale Sam Kottler @samkottler
  2. 2. Sam Kottler | Puppet at Scale2 About me ● Worked on large-scale infra for the web @ Venmo, Acquia, and Digg ● Rubygems.org infrastructure ● Bundler core ● Fedora developer ● Core committer on the Foreman
  3. 3. Sam Kottler | Puppet at Scale3 What we'll cover 1. Some basics 2. Master vs. masterless deployment 3. CA management 4. Clustering 5. Node management 6. Development + deployment practices
  4. 4. Sam Kottler | Puppet at Scale4 Why we care ● Hyperscale computing ● Massive, multi-DC infrastructure ● Dynamic environments ● The Cloud ™
  5. 5. Sam Kottler | Puppet at Scale5 Master vs. masterless
  6. 6. Sam Kottler | Puppet at Scale6 Provisioning nodes with a master 1. New node comes online 2. A script is run to install packages and configure /etc/hosts 3. The agent gets run, generates a CSR, and sends it to the master 4. The cert gets signed based on an autosign rule or `puppet cert --sign <nodename>` 5. Puppet runs
  7. 7. Sam Kottler | Puppet at Scale7 Provisioning nodes without a master 1. New node comes online knowing its role 2. A script runs to install packages and retrieve package/tarball 3. puppet apply
  8. 8. Sam Kottler | Puppet at Scale8 Certificate authority ● Use by Puppet to authenticate agents ● CSR generated and signed by the CA ● Shared CRL across all CA machines
  9. 9. Sam Kottler | Puppet at Scale9 Clustering patterns ● CA has lots of state ● Masters should be stateless ● Reduce the number of file shares
  10. 10. Sam Kottler | Puppet at Scale10 DNS-based clustering
  11. 11. Sam Kottler | Puppet at Scale11 Load balanced clustering
  12. 12. Sam Kottler | Puppet at Scale12 Masters across data-centers ● Shared CA vs. per-region ● Deploy in stages across data-centers
  13. 13. Sam Kottler | Puppet at Scale13 Multi-cluster
  14. 14. Sam Kottler | Puppet at Scale14 Node classification
  15. 15. Sam Kottler | Puppet at Scale15
  16. 16. Sam Kottler | Puppet at Scale16 External node classifiers ● Output YAML based on external data ● The Foreman, Puppet Enterprise, Puppet Dashboard ● Your own custom data source ● Key integration source with your own CMDB
  17. 17. Sam Kottler | Puppet at Scale17
  18. 18. Sam Kottler | Puppet at Scale18 Packaging for masterless https://github.com/skottler/librarian-masterless-packaging ● Use /etc/puppet/modules (or modulepath) ● Build RPM's/deb's for distribution ● Publish packages to a repo ● Install/update packages on all machines
  19. 19. Sam Kottler | Puppet at Scale19 Distributed runs ● Run puppet based on changes in your code ● Mcollective/SSH/cron
  20. 20. Sam Kottler | Puppet at Scale20 Deployment ● Masters are just another deployment target! ● Build CI pipelines ● One-click deployments to masters ● Lint and test your modules
  21. 21. Sam Kottler | Puppet at Scale21 cap puppetmaster deploy DC=london
  22. 22. Sam Kottler | Puppet at Scale22 Controlled releases ● Separate hosts into groups to do red/black releases ● Build smaller sub-groups of canary hosts ● Monitor your puppet runs
  23. 23. Sam Kottler | Puppet at Scale23 Thanks! @samkottler shk@redhat.com https://github.com/skottler

×