Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Knee deep in the undef - Tales from refactoring old Puppet codebases

310 views

Published on

As Puppet pushes into it’s second decade of reign, there are several organisations out there that have been using Puppet for a long time. Sometimes, even since the beginning!

With the EOL announcement Puppet 3.X release, we’ve had a number of customers approach us to help with their upgrade. Normally the upgrade itself is fairly, it’s the code base that gives the biggest challenge, especially those with over 3 years of organic growth.

So let’s spread the word of common anti-patterns and issues that can come back to bite you

We’ll be talking about how Hiera is both the best and worst thing to happen to Puppet, marvel at how people were happily running 0.2 Puppet in production and what hacky solutions that seemed good at the time will come back to bite you!

By the end of this, you’ll hopefully have learnt how to make sure that your Puppet code is defensively coded to to make sure your Puppet code base is healthy for the next decade!

Published in: Technology
  • Be the first to comment

Knee deep in the undef - Tales from refactoring old Puppet codebases

  1. 1. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Knee deep in the undef: Tales from refactoring old Puppet codebases Peter Souter Senior Professional Services Engineer | Puppet @petersouter
  2. 2. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 2 Who am I? @petersouter Senior Professional Services Engineer 5 years using Puppet 2 years @ Puppet Inc Help customers deploy Puppet Teach Puppet classes Contribute to the community and open-source petems IRC/Slack/GitHub
  3. 3. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Warning: I speak quickly And I have a different accent... 3
  4. 4. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter “Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.” - http://www.retrospectives.com/pages/retroPrimeDirective.html 4
  5. 5. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Show of hands in the room What version of Puppet are you on now? 5
  6. 6. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Show of hands in the room How old is your Puppet codebase? When was your first commit? 6
  7. 7. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 7 The King is Dead… Long live the king! Puppet 3.X is now EOL
  8. 8. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Some customers we hadn’t talked to in a while... This is a good time for an upgrade or refactor of code 8
  9. 9. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Noticed a few repeating anti-patterns and issues with a lot of these customers And similar issues with people asking for community help in Slack and IRC 9
  10. 10. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 10 commit bbdecbeb1c199b7132f94c33ee44a66e265fa456 Author: Jane Doe <jane.doe@example.com> Date: Tue Jun 4 10:46:13 2013 +0100 Initial commit
  11. 11. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 11 commit ac4b7c3312d75c3584ef069c272f7ad4ff610eb4 Author: John Doe <john.doe@example.com> Date: Thu Mar 3 15:46:59 2011 +0000 Import git-svn-id: https://svn.example.com/svn/puppet/trunk@1 3d7401f0-959d-0410-a61e-fbe730d1da08
  12. 12. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter I’m not going to talk about the Puppet 4 upgrade stuff Lots of talks on that… but it’s a common time for people to look over their codebases when upgrading 12
  13. 13. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 13 1. Hiera overload/Too much data 2. Lack of validation/CI 3. Reinventing the wheel 4. Lack of VCS best practises 5. Make the newbie experience better 5 Key Areas that came up over again
  14. 14. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Hiera overload/Too much data Hiera is simultaneously the best and worst thing to happen to Puppet 14
  15. 15. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter YAML management is the biggest pain point in the DevOps world... Not just limited to Puppet... 15
  16. 16. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Why is this a problem? Two main issues... 16
  17. 17. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter First: Context switching is the mind killer Humans suck at multitasking 17
  18. 18. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 18 “The trick here is that when you manage programmers, specifically, task switches take a really, really, really long time. That's because programming is the kind of task where you have to keep a lot of things in your head at once. The more things you remember at once, the more productive you are at programming. A programmer coding at full throttle is keeping zillions of things in their head at once: everything from names of variables, data structures, important APIs, the names of utility functions that they wrote and call a lot, even the name of the subdirectory where they store their source code.” Human Task Switches Considered Harmful - Joel Spolsky http://www.joelonsoftware.com/articles/fog0000000022.html
  19. 19. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 19 ● Get the people who are maintaining the code to look at what's written ● How long does it take them to find out how something’s done in your code? ● Ask them if they can easily understand what’s happening and why To fix it: Get an outsider's perspective
  20. 20. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Second: Maintaining this data is costly Both technically and mentally 20
  21. 21. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 21 $ find . -type f | wc -l 587 Hiera can become bloated over time
  22. 22. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 22 ● Minimise node specific data: it’s the hardest to maintain and easiest to get stale ● Abstract information into a hierarchy to DRY it up ● Purge irrelevant data ● Remember, it’ll be in the git history still! Clean up your hiera!
  23. 23. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 23 PR I’ve been meaning to resurrect: maybe at the contributor summit? - https://github.com/voxpupuli/puppet-syntax/pull/57
  24. 24. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Lack of validation/CI The earlier you can catch errors, the cheaper it is to fix them. 24
  25. 25. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 25 Gareth’s talk about the Future of Puppet testing https://speakerdeck.com/garethr/the-future-o f-testing-puppet-code
  26. 26. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 26 ● Puppet-syntax checks puppet, yaml, erb and epp files ● When you get that under control, use puppet-lint for style checks ● Then rspec-puppet for unit tests ● Then beaker/testkitchen for acceptance tests ● If you don’t have CI, fix that! Or at least use git pre/post commit hooks as a basic gate At its most basic, a syntax error should not be deployable
  27. 27. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 27 Testing with containers makes tests fast and repeatable http://cfgmgmtcamp.eu/schedule/testing/andy-henriod.html
  28. 28. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Lack of VCS best practises Monolithic repos make it hard to upgrade modules and for multiple people to work on the module at one time 28
  29. 29. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Make the newbie experience better People change teams, new people join. Make sure they can contribute as soon as possible 29
  30. 30. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter ● If you want people to follow specific workflows or requirements make it easy as possible ● Prevent risky newbie mistakes with CI and tooling (puppet-lint, rubocop, rake tasks etc) ● Pair-programming is not just about the code: it’s about learning the processes and improving the experience for both new and old users 30 The bar should be low enough to trip over
  31. 31. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 313131 ● Comments can be a crutch ● The best comment is a good name for a method or variable ● Sometimes your hand is forced: make sure you comment clearly and concisely ● Give the full context in the commit message Code should be self-documenting
  32. 32. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 323232 https://blog.codinghorror.com/code-smells/ “There's a fine line between comments that illuminate and comments that obscure. Are the comments necessary? Do they explain ‘why’ and not ‘what’? Can you refactor the code so the comments aren't required? And remember, you're writing comments for people, not machines.”
  33. 33. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 333333 ● Separate subject from body with a blank line ● Limit the subject line to 50 characters ● Capitalize the subject line ● Do not end the subject line with a period ● Use the imperative mood in the subject line ● Wrap the body at 72 characters ● Use the body to explain what and why vs. how Git Commit best practices
  34. 34. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 343434 http://chris.beams.io/posts/git-commit/ “Re-establishing the context of a piece of code is wasteful. We can't avoid it completely, so our efforts should go to reducing it [as much] as possible. Commit messages can do exactly that and as a result, a commit message shows whether a developer is a good collaborator.”
  35. 35. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 35 Commit Often, Perfect Later, Publish Once Learn how rebasing and amending works
  36. 36. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 363636 An example of good git commits at work http://www.philandstuff.com/2014/02/09/git-pickaxe.html
  37. 37. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 373737 The Git Pickaxe The reason I care about commit messages is because I'm an avid user of the git pickaxe. If I'm ever confused about a line of code, and I want to know what was going through the mind of the developer when they were writing it, the pickaxe is the first tool I'll reach for. For example, let's say I was looking at this line from our puppet-graphite module: exec <%= @root_dir %>/bin/carbon-cache.py --debug start That --debug option looks suspect. I might think to myself: "Why are we running carbon-cache in --debug mode? Isn't that wasteful? Do we capture the output? Why was it added in the first place?" In order to answer these questions, I'd like to find the commit that added the switch.
  38. 38. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 383838 commit 5288d5804a3fc20dae4f3b2deeaa7f687595aff1 Author: Philip Potter <philip.g.potter@gmail.com> Date: Tue Dec 17 09:33:59 2013 +0000 Re-add --debug option (reverts #11) The --debug option is somewhat badly named -- it *both* adds debug output, *and* causes carbon-cache to run in the foreground. Removing the option in #11 caused the upstart script to lose track of the process as carbon-cache started unexpectedly daemonizing. Ideally we want to have a way of running through upstart without the debug output, but this will fix the immediate problem. Ta-dah, mystery solved!
  39. 39. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Reinventing the wheel There are existing modules, use them! 39
  40. 40. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 404040 The more existing boilerplate and modules you can leverage: ● The less work you have to do (Yay for you!) ● The more maintainable it will be (Yay for your team!) ● The more supportable it will be (Yay for everyone!) Start off on the right foot
  41. 41. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 4141414141 Start with the puppet control repo 41https://github.com/puppetlabs/control-repo
  42. 42. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 424242424242 Use as supported modules as your number 1 choice https://forge.puppet.com/modules?endorsements=supported
  43. 43. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 4343 Use approved modules as your number 2 choice 43https://forge.puppet.com/modules?endorsements=approved
  44. 44. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 4444 If there’s no supported or approved, use your best judgement... 44https://forge.puppet.com/modules?utf-8=%E2%9C%93&sort=downloads&q=limits
  45. 45. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter When writing new modules, use a custom module skeleton https://github.com/petems/petems-puppet-module-skeleton 45
  46. 46. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 46 Write as little as possible Code is like a puppy: anything you write you are responsible for
  47. 47. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 47 Summary What have we learnt?
  48. 48. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 48 Get your hiera under control Clean up old files, validate and try and abstract
  49. 49. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 49 Get validation and CI on your codebase syntax, lint, spec, acceptance
  50. 50. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 50 Don’t reinvent the wheel Try and reuse as much as possible,
  51. 51. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 51 Make sure you’re VCS is top notch Git is the primary place to show your intentions and the history of the code
  52. 52. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter 52 Ensure your newbie experience is great Make the bar low enough to fall over
  53. 53. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter ● PuppetConf 2016: Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T 71 views http://www.slideshare.net/PuppetLabs/puppetconf-2016-enjoying-the-journey- from-puppet-3x-to-4x-rob-nelson-att ● Does Your Configuration Code Smell? - Tushar Sharma, Marios Fragkoulis and Diomidis Spinellis - Dept of Management Science and Technology http://www.tusharma.in/wp-content/uploads/2016/03/ConfigurationSmells_pre print.pdf ● Roles and profiles: A complete example https://docs.puppet.com/pe/2016.5/r_n_p_full_example.html ● Puppet Design Patterns - David Danzilio, Kovarus http://www.slideshare.net/DavidDanzilio/puppet-design-patterns-puppetconf 53 Want to know more?
  54. 54. Knee deep in the undef Tales from refactoring old Puppet codebases @petersouter Q&A 54

×