Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own Mess – Josh Snyder, Yelp


Published on

Here are the slides from Josh Snyder's presentation called Deconfiguration Management: Making Puppet Clean Up Its Own Mess. Watch the videos at

Published in: Technology
  • Be the first to comment

PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own Mess – Josh Snyder, Yelp

  2. 2. WHAT'S THE PROBLEM? We use Puppet We change things
  3. 3. I have been known to deploy 20 times daily And Puppet runs every 30 minutes
  4. 4. That thing you deployed in a hurry isn't always in the place it should be long term. Or...we deploy a thing, and decide next week (or tomorrow) that it isn't right for us Or we just straight-up make mistakes Or our coworkers make mistakes git revertcan't save me
  5. 5. My colleague sends me a PR to deploy memcached config files under a new path. Should I comment on their failure to clean up the old ones? YES!
  6. 6. But they're busy But I'm busy And I really want to get home to make dinner
  8. 8. IGNORE THE PROBLEM Ignore it! It probably won't hurt anything Except it will confuse people. It WILL confuse you, at two AM JVM versions will exist on some older dev machines, but not newer ones
  9. 9. WRITE CODE TO CLEAN UP AFTER OTHER CODE file { '/etc/memcached.conf': ensure => absent, }
  10. 10. ENSEMBLES OF RESOURCES GET COMPLEX if $ensure == 'present' { file { '/etc/init/memcached.conf': ensure => 'file', ... } } else { file { '/etc/init/memcached.conf': ensure => 'absent', ... } }
  11. 11. NO FUN AT ALL more code to write more code to review my cleanup code might be broken someone has to clean up the cleanup code
  12. 12. WRITE MORE CODE (IN A DIFFERENT LANGUAGE) Use Ansible clean up a er Puppet?
  13. 13. REBUILD MACHINES Good! Cattle, not pets I'm deploying new changes every 30 minutes We only clean things up every day? 30 days? 60 days? Five years? ∃ an impedance and tooling mismatch
  14. 14. IMMUTABLE IMAGES A more aggressive version of rebuilding machines Use if it suits you
  15. 15. THE SOLUTION (TO ALL YOUR PROBLEMS) 1. Use Puppet to specify what should be deployed 2. Allow Puppet to remove anything it doesn't know about
  16. 16. WHAT'S TO COME? A few basic examples (directory purging, cronjobs, hosts) A bit of Puppet internals Use Puppet internals to achieve more purging
  18. 18. DIRECTORY PURGING file { '/etc/cassandra': ensure => 'directory', recurse => true, purge => true, force => true, ... }
  19. 19. WHAT WILL THIS DO? Use puppet agent --noop Or add  noop => true Look at the system: $ ls /etc/cassandra ... $ dpkg -S /etc/cassandra ...
  20. 20. EXAMPLE 1: PARTIAL MANAGEMENT Situation: We want to purge /etc/cassandra, but we need to generate the list of seeds outside of Puppet file { '/etc/cassandra': ensure => directory, recurse => true, purge => true, force => true, ... } file { '/etc/cassandra/seeds': ensure => file, replace => false, ... }
  21. 21. EXAMPLE 2: CRONJOBS Lots of cronjobs Lots of cronjobs! Using Yelp's One file per job in /etc/cron.d puppet-cron
  22. 22. PURGING CRONJOBS Problem: purge => truewould wipe out crons provided by Debian packages.
  23. 23. Option ImplicationsOption Implications Recompile cron to read from a supplemental directory Anyone else using would have to use our patched cron Create Fileresources for each file we expect a from a deb. Whenever someone installs a package with a new cronjob in it, they'd get a nasty surprise Find some way to identify those cronjobs that were originally created by Puppet Good puppet-cron
  24. 24. What this solution ends up looking like: (ish) file { '/nail/etc/cron.d': ensure => directory, purge => true, force => true, recurse => true, } file { '/nail/etc/cron.d/myjob': ensure => file, ... } -> file { '/etc/cron.d/myjob': ensure => link, target => '/nail/etc/cron.d/myjob', }
  25. 25. EXAMPLE 3: /ETC/HOSTS Puppet agent has a RAL (resource abstraction layer) RAL is responsible for representing resources on the system as Puppet Resource objects $ puppet resource host host { 'ip6-allnodes': ensure => 'present', ip => 'ff02::1', target => '/etc/hosts', } host { 'ip6-allrouters': ensure => 'present', ip => 'ff02::2', target => '/etc/hosts', } host { 'localhost': ensure => 'present',
  26. 26. Puppet diffs resources in the catalog against the RAL it constructs Could we ask it remove resources present in the RAL but not in the catalog? YES!
  27. 27. IT'S THIS SIMPLE resources { 'host': purge => true, } Use noop => trueto try before you buy
  28. 28. HOW IT WORKS All on the agent, a er catalog compilation Iterate over resources, calling the generateor eval_generate method on each. Each resource has the opportunity to add more resources to the Puppet run.
  29. 29. Walkthrough: fetching files from a fileserver file { '/etc/cassandra': ensure => directory, source => 'puppet:///modules/cassandra/config_dir', recurse => true, purge => true, force => true, } 1. Get catalog with this resource declared 2. Puppet agent calls eval_generateon this resource 3. eval_generateexamines the disk, compares it with the Puppet fileserver 4. Generates more resources to represent the files beneath this directory
  30. 30. HOW THE RESOURCESTYPE WORKS resources { 'host': purge => true, } 1. Puppet calls generate 2. Generate finds all resources of type Hostin the catalog 3. Asks providers of Hostfor their instances 4. Compare the two 5. Emit new resources: host { 'ip6-allnodes': ensure => absent, }
  32. 32. PURGING UNDESIRED DEBIAN PACKAGES Let's say I do: resources { 'package': purge => true, } This happens: Notice: /Stage[main]/Main/Package[libxtst6]/ensure: current_value Notice: /Stage[main]/Main/Package[libxcb-dri3-0]/ensure: current_va Notice: /Stage[main]/Main/Package[powermgmt-base]/ensure: current_v Notice: /Stage[main]/Main/Package[python3-py]/ensure: current_value Notice: /Stage[main]/Main/Package[libtk8.6]/ensure: current_value Notice: /Stage[main]/Main/Package[node-ansi-color-table]/ensure: Notice: /Stage[main]/Main/Package[libxpp3-java]/ensure: current_val Notice: /Stage[main]/Main/Package[python3-newt]/ensure: current_val Notice: /Stage[main]/Main/Package[bsdmainutils]/ensure: current_val Notice: /Stage[main]/Main/Package[libpulse0]/ensure: current_value Notice: /Stage[main]/Main/Package[liblvm2app2.2]/ensure: current_va
  33. 33. NO BUENO Why doesn't puppet understand that it should remove all packages that: aren't in the catalog no other package depends on
  34. 34. We need to be dependency-aware What removes packages and is dependency aware? apt-get autoremove
  35. 35. APT-GET AUTOREMOVE Divides packages into: manually installed (we're sure we want this) auto installed (a dependency)
  36. 36. THIS MAPS WELL TO PUPPET puppet state ⇒ autoremover state puppet state ⇒ autoremover state in catalog manually installed not in catalog automatically installed
  37. 37. AN IMPLEMENTATION COMES TOGETHER 1. Synchronize the autoremover database with the Puppet catalog 2. Run apt-get autoremove 3. Problem?
  38. 38. AN IMPLEMENTATION COMES TOGETHER (PARTE DUEX) 1. Synchronize the autoremover database with the Puppet catalog 2. Run apt-get -s autoremove 3. Read the output and create Puppet package resources 4. Much rejoicing!
  39. 39. GENERAL PURPOSE SOLUTIONS? Could there be a jack-of-all-trades solution to purging? What if we could do: purge { 'user': unless => [ 'uid', '<=', '500' ], } It exists:
  41. 41. NOT MY FAVORITE DEFAULT Q: What does this do? package { 'mysql-server-5.7': } package { 'bash': } A: Creates version dri
  42. 42. UPGRADE (SOME OF THE) THINGS package { 'mysql-server-5.7': ensure => $my_favorite_mysql_version } package { 'bash': } aptly_purgecan set all versioned packages as held by dpkg aptly_purge { 'packages': hold => true, } Upshot: apt-get dist-upgradeand unattended-upgrades will only touch packages without a specific Puppet version specified.
  43. 43. END MATTER Please do tell me your stories of Puppet resource purging. Walk up and say hi right now. works in Puppet Community Slack@josnyder