Successfully reported this slideshow.
Your SlideShare is downloading. ×

Enjoying the Journey from Puppet 3.x to Puppet 4.x (PuppetConf 2016)

Upcoming SlideShare
Power of Puppet 4
Power of Puppet 4
Loading in …3

Check these out next

1 of 31 Ad

Enjoying the Journey from Puppet 3.x to Puppet 4.x (PuppetConf 2016)

Download to read offline

Let's describe the process for upgrading from Puppet 3 to 4, list some common pitfalls and how to avoid them, and be sure to enjoy ourselves in the process!

Let's describe the process for upgrading from Puppet 3 to 4, list some common pitfalls and how to avoid them, and be sure to enjoy ourselves in the process!


More Related Content

Slideshows for you (20)


Similar to Enjoying the Journey from Puppet 3.x to Puppet 4.x (PuppetConf 2016) (20)

Recently uploaded (20)


Enjoying the Journey from Puppet 3.x to Puppet 4.x (PuppetConf 2016)

  1. 1. Enjoying the Journey from Puppet 3.x to Puppet 4.x Rob Nelson
  2. 2. Who Am I? Puppet user since 2014 (3.3 era) Vox Pupuli, puppet-lint contributor @rnelson0,
  3. 3. Agenda • Why upgrade? • Refactor our codebase for Puppet 4 • Upgrade our Puppet master(s) and agents to Puppet 4.x • Refactor our codebase to remove Puppet 2- and 3-isms • Tips, tricks, and tools • Enjoying ourselves
  4. 4. Why? • Puppet 4 is old! First released March, 2015 • Puppet 3 is really old! End Of Support on December 31, 2016 • Puppet 4 only modules • Puppet 4 language improvements • Application Orchestration • PE first, FOSS eventually; some free implementations (such as choria) appearing • AIO Puppet and Puppetserver • Better performance, security; same agent/puppetserver between FOSS and PE • Puppet 5 is coming!
  5. 5. Who does this apply to? • Puppet Enterprise users • Puppet Opensource users • Foreman (1.13+) users • Master and Masterless
  6. 6. Blueprint • Start with Puppet 3.x • Read the release notes • Plan the roadmap • Validate / create tests • Refactor until the new version passes all tests • Upgrade/Replace the master(s) • Upgrade the agents • Repeat the Refactor / Upgrade steps until you get to 4.latest
  7. 7. Release Notes • All of them - not just the latest version • Identify potential issues, deprecated features, etc • Determine the minimum version required to upgrade to target version • Stay up to date
  8. 8. Define the Roadmap • Determine the current version • In-place upgrades or new infrastructure? • Identify intermediate version steps • Enable Future Parser [and Strict Variables] before you hit 4.x • PE: Requires intermediate upgrades or fresh installs (check KB) • FOSS: Go straight to 4.latest • Determine how upgrades and interruptions affect ecosystem products – PE Console/puppetboard, SEIMs, monitoring, etc.
  9. 9. FOSS Example Roadmap • 3.6.0 -> 3.8.7 • 3.8.7 w/Future Parser [and Strict Variables] • 3.8.7 -> 4.7.0 • Today’s example roadmap
  10. 10. PE Example Roadmap • 3.7.2 -> 3.8.6 • 3.8.6 w/Future Parser [and Strict Variables] • 3.8.6 -> 2015.3.3 • 2015.3.3 -> 2016.2.1
  11. 11. Validate/Create Tests • Tests assure (mostly) predictable behavior • Determine what kinds of tests you need - unit, acceptance, integration, other? • Good testing setup in puppet-module-skeleton • puppet-retrospec generates naive tests that need tuned • Existing tests must pass before modifying code • Turn on Future Parser [and Strict Variables] only at 3.8.x • Use puppet-lint community plugins, esp for v4 transition • Beyond tests: catalog diffs, personalized tests
  12. 12. Rspec Tests $ cat spec/classes/apache_spec.rb require 'spec_helper' describe 'profile::apache', :type => :class do let :facts do { facts_hash } end context 'with defaults for all parameters' do it { create_class('profile::apache') } it { contain_package('httpd') } it { contain_user("apache") } end end
  13. 13. Rspec Run [rnelson0@build03 profile:production]$ bundle exec rspec spec/classes/apache_spec.rb profile::apache with defaults for all parameters should contain Class[profile::apache] should contain Package[httpd] should contain User[apache] Finished in 7.82 seconds (files took 2.49 seconds to load) 3 examples, 0 failures
  14. 14. Refactor • Create a new branch for the target version, e.g. 3.8.7 • Test against current and target versions, e.g. ~>3.6.0 and ~>3.8.0, with and without Future Parser/Strict Variables • Identify failing tests, refactor to fix • Upgrade modules as early as possible. Be aware of the required Puppet version for a module version, and look out for defunct or migrated modules, such as those transferred to Vox Pupuli • Move forward when tests are green for the next version – previous may be red
  15. 15. Testing with particular Puppet versions $ grep PUPPET Gemfile gem "puppet", ENV['PUPPET_GEM_VERSION'] || '~> 4.0' [rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='~>3.8.0' [rnelson0@build controlrepo]$ bundle update Installing puppet 3.8.7 (was 4.6.0) [rnelson0@build controlrepo]$ bundle exec puppet --version 3.8.7 [rnelson0@build controlrepo]$ export PUPPET_GEM_VERSION='3.8.1' [rnelson0@build controlrepo]$ bundle update Installing puppet 3.8.1 (was 3.8.7) [rnelson0@build controlrepo]$ bundle exec puppet --version 3.8.1
  16. 16. High level Master(s) upgrade process • Prep for new master/in-place upgrade • Deploy new/upgrade in testing • Revert • Deploy new/upgrade in production • Start with Master of Masters or other “parent” nodes first • Update separate PuppetDB node, puppetdb-termini on masters
  17. 17. Replace the Master • Prepare a new operational environment • Do not serve bad/incorrect catalogs to existing nodes • Deploy a new master on the target puppet version • Bootstrap configuration/code • Test the master against itself, puppet agent -t • Deploy and test canary nodes in the same operational environment
  18. 18. In-place Master upgrade • Snapshot (or equivalent) the master(s) and canary nodes • Restrict access to the master: • Control access with firewall/load balancer • Disable puppet agent on nodes with orchestration • Revoke certificates for non-canary nodes • Revoke the CA, generate a new CA and new agent certs for canary nodes only • Upgrade the master • Test the master then canary nodes with puppet agent -t
  19. 19. Troubleshooting • Collect logs from the master and canaries • Look for changed resources, number of resources in catalog • Revert production environment • Analyze cause(s) • Refactor code and data to address issues • Try again • Learn from failures, prevent them in the future
  20. 20. Upgrade the Agents • Can often skip on PATCH versions and some MINOR versions (see rel notes) • puppetlabs/puppet_agent (requirements) updates agents on next check-in • Orchestration • Replace nodes with new instances running the new agent • By hand
  21. 21. Repeat • Relax, enjoy the success of an upgrade! • Start working on the next version/feature flags • Repeat the Refactor / Upgrade steps
  22. 22. Keeping Up
  23. 23. Keeping Up Refactor to take advantage of Puppet 4 language improvements, new tools (ex: r10k -> PE Code Manager), new file locations, etc. • PE has quarterly upgrades, FOSS more frequent • The less frequently you do something, the more painful it is. “Upgrade early and upgrade often!” • Try not to get more than 2 MINORs behind • Test against puppet version ~>4.0 (latest v4) and run bundle update before manual tests
  24. 24. Puppet 4 Language Improvements • Replace create_resources() with iteration • Replace validate_*() with data types (including a Sensitive type) • There is a validate_legacy() helper function available in puppetlabs/stdlib to assist with replacing validate_*() functions (blog) • Simplified resource wrappers with * and + operators • Improved default attributes are per-expression • New template type EPP is available • Puppet Lookup, Data In Modules, and other hiera improvements • Use $facts[] instead of global variables to tidy up the namespace and remove ambiguity
  25. 25. Tips & Tricks – Puppet Enterprise • PE includes support, use it for planning/errors • Puppet Enterprise Upgrade Service to engage Pro Services • PE Classifier changes over time. Review Preconfigured Node Groups documentation • pe_puppetserver_gem is out, puppetserver_gem is in • Do not use PE’s bundled Ruby for other Ruby tasks, conflicts between bundled/downloaded gems. Recommend rbenv/rvm or SCL-equiv instead • Do not ever do this on your master. EVER!
  26. 26. Tips & Tricks - Strings Understand how string conversion works in puppet, hiera, rspec-puppet, and how it has changed: • rspec-puppet: 'undef' represents an undefined value • Puppet DSL: it is the string undef! Try :undef, without quotes, instead • If you have a file resource with a title or path of ${undefvar}/${populatedvar}, rspec will start failing because file { 'undef/etc/app.conf' :} is not valid • Similar issue with 'true' vs true and 'false' vs false • Other common issues: input from hiera/ENC, quoted numbers as strings, stringify vs structured facts, unquoted strings in case selectors, etc • May require acceptance tests/canary nodes to become apparent
  27. 27. Tips & Tricks - Hiera • Hiera eyaml gem is lost during the upgrade to the 4.x puppetserver • Enable the yaml backend and ensure that the master does not rely on eyaml data • Run the agent on the master to redeploy the gem (with puppet/hiera or similar) before agents check in • %{}: used to prevent variable interpolation, as in %%{}{environment} to generate the string %{environment}. In 3.x and in 4.5 resolves to an empty string, in 4.0-4.4 it returned the scope, giving strings like %<#Hiera:7329A802#>{environment}. Use %{::} instead, as in %%{::}{environment}. Affects PE < 2016.2.0 • datadir: some versions expect :: prepends to variables and others do not. Change %{environment} to %{::environment}. Likely PE < 2016.2.0 as well
  28. 28. Tips & Tricks - Other • Review modules and their supported versions. May be incorrect or weak assumptions (>= 3 but should also include < 4 – check tests) • Upgrades across major versions mean additional troubleshooting • Upgrade early – but with caveats • Many tools to assist with automating version upgrades in your Puppetfile • ERB scope: prepend most variables with @ (<%= var %> to <%= @var %>) • Script to detect usage of hardcoded /etc/puppet paths, no longer correct in v4 • External fact weighting bug: FACT-1413 • Minimize coupled/entangled changes • Ask for help! Colleagues, social media, etc.
  29. 29. Tools • Puppet Community Slack / IRC and Mailing Lists • puppet-ghostbuster helps you find "dead code" that you may want to prune before you start on your refactoring journey. • rspec-puppet, puppetlabs_spec_helper, and puppet-lint are improving their Puppet 4 support • A number of catalog diff tools exist (diff generators and a viewer) to inspect the actual catalog differences from active nodes across different versions of Puppet.
  30. 30. Links Additional information on Puppet 4 and Migrations • Official Puppet Upgrade Docs • Whirlwind Tour of Puppet 4 by R.I. Pienaar • The Power of Puppet 4 by Martin Alfke • Puppet - our journey from Puppet 3.8 to Puppet 4 by Jonas Genannt
  31. 31. Summary • Plan the upgrade roadmap • Have working tests before upgrading • Step through the new versions / feature flags • Refactor code to take advantage of the language and tool improvements • Keep mowing • Enjoy the journey!

Editor's Notes

  • Tons of talks on Puppet 4 language here at PuppetConf, check them out!

    Puppetserver is the future and Ruby moves fast, take advantage of the AIO builds when possible – improved support and security, too

    Other reasons: better performance, packaging and security updates, same agent between FOSS and PE.

  • If you are on Puppet 2.x, you need to get to 3.x first! But that is not this talk. There were some good talks about this at PuppetConf 2015. If you are actually starting new - start with Puppet 4!
  • When you upgrade, you could affect data feeds into other systems, which could generate extra tickets or result in some data loss, at least during the upgrade. Add it to your external test regimen – is the SEIM still collecting the logs you would expect, is dynamic monitoring showing the new devices, etc?
  • Testing talks: "Turning Pain Into Gain" today at 2:30PM and "The Future of Testing Puppet Code" tomorrow at 3:45PM.

    Tests do not guarantee success, but can identify many regressions and determine when failure is guaranteed.

    Future parser existed in earlier versions but had some bugs, wait till you hit 3.8.

    Catalog diffs can be useful for many people, but not all – see the talk "Getting to the Latest Puppet" tomorrow at 1PM for more information. Don’t go overboard, just find what works and improve it over time.
  • This is a test on a profile module, not a component module, but this is absolutely needed! This is a majority of the code you are going to write and it helps with your design phase if you write them first.
  • There’s a lot of different ways to upgrade – a new master with the same IP/DNS as the original, an in-place upgrade, a brand new master with a new CA structure. You’ll have to determine which is best for your architecture. As usual, there is no right answer, only what works for you.

    Remember that --noop and canary tests will send reports to puppetdb. If your puppetdb and puppet master services are on different nodes, work on them together.
  • 'Environment' is such an overloaded term. What I mean by 'operational environment' is the set of proper network, host, etc. where the new master will reside. It should NOT be in the same operational environment as the existing master, so that there is no way existing agents could check in to the new master. Maybe use DNS to ensure the default server name puppet uses doesn’t get confused between the two.

    Of course, it is fairly privileged to assume you can do this. If you do not have that luxury, do your best to ensure old and new systems do not mix.

    There are many ways to bootstrap your master. If you do not have a process to bootstrap your master, we will talk about in-place upgrades shortly. You will eventually want the ability to bootstrap, though.
  • There is no right way, just a way that works for you!
  • Recently, Puppet changed their GPG signing key. Puppet has details to fix that at
  • EPP = Embedded Puppet Template. In addition to writing templates in the puppet DSL that you already know, it addresses some of the scoping issues that were common problems in the ruby-based ERB. You can use both template types just fine, however.
  • Now that the generalities are out of the way, we can talk specifics and lessons learned.

    I know many of us do not like to engage support, but if you do not use it for updates, when will you use it? And if you need someone to do the upgrade, not just assistance, that is an option now as well.

    Using the PE/AIO ruby can cause unexpected conflicts between bundled gems and downloaded gems. See PUP-6106 for an example.
  • Truthiness is a difficult concept sometimes, it’s worth gaining a deeper understanding of it. One suggestion is to look for active resources and resource counts on reports to indicate where these may be issues with its interpretation.

    Stringify vs structured facts: Foreman only received the ability to natively process structured facts in its latest release, 1.13.
  • Anecdata suggests that sometimes eyaml gem is lost on 4.x upgrades, though docs say it should not

    Be sure the hiera.yaml file gets put in the right location, $confdir or $codedir, depending on your puppet version – modules handle this for you
  • "Trust but verify" support levels.

    You should try and upgrade modules ASAP, but there are good reasons not to. Newer versions may require Puppet 4 before you get there, or a newer OS, or deploy a newer version of an application you are not ready for. This is a VERY rough guideline.

    If you use external facts in /etc/facter/facts.d, you may expect them to override calculated facts. The facter that addresses the issue has not shipped as of puppet-agent 1.7.0.

    You are going to be making a lot of changes at once, but minimize coupled or entangled changes. A week of small changes beats trying to do it all at once, whcih will likely be longer when including troubleshooting time.
  • Slack and IRC are great ways to engage the vendor as well as other community members – ask difficult questions or just get another pair of eyes on something before you commit it.

    Tools are constantly improving, check back frequently to see if your concerns are addressed.