Knee deep in the undef - Tales from refactoring old Puppet codebases

Knee deep in the undef
Tales from refactoring old Puppet codebases
@petersouter
Knee deep in the undef:
Tales from refactoring old
Puppet codebases
Peter Souter
Senior Professional Services Engineer | Puppet
@petersouter

@petersouter
2
Who
am I?
@petersouter
Senior Professional
Services Engineer
5 years using Puppet
2 years @ Puppet Inc
Help customers deploy
Puppet
Teach Puppet classes
Contribute to the
community and
open-source
petems
IRC/Slack/GitHub

@petersouter
Warning: I speak quickly
And I have a different accent...
3

@petersouter
“Regardless of what we discover, we understand and truly
believe that everyone did the best job they could, given
what they knew at the time, their skills and abilities, the
resources available, and the situation at hand.”
- http://www.retrospectives.com/pages/retroPrimeDirective.html
4

@petersouter
Show of hands in the room
What version of Puppet are you on now?
5

@petersouter
Show of hands in the room
How old is your Puppet codebase? When was your first
commit?
6

@petersouter
7
The King is Dead… Long live the king!
Puppet 3.X is now EOL

@petersouter
Some customers we hadn’t talked to in a
while...
This is a good time for an upgrade or refactor of code
8

@petersouter
Noticed a few repeating anti-patterns
and issues with a lot of these customers
And similar issues with people asking for community help
in Slack and IRC
9

@petersouter
10
commit bbdecbeb1c199b7132f94c33ee44a66e265fa456
Author: Jane Doe <jane.doe@example.com>
Date: Tue Jun 4 10:46:13 2013 +0100
Initial commit

@petersouter
11
commit ac4b7c3312d75c3584ef069c272f7ad4ff610eb4
Author: John Doe <john.doe@example.com>
Date: Thu Mar 3 15:46:59 2011 +0000
Import
git-svn-id: https://svn.example.com/svn/puppet/trunk@1
3d7401f0-959d-0410-a61e-fbe730d1da08

@petersouter
I’m not going to talk about the Puppet 4
upgrade stuff
Lots of talks on that… but it’s a common time for people to
look over their codebases when upgrading
12

@petersouter
13
1. Hiera overload/Too much data
2. Lack of validation/CI
3. Reinventing the wheel
4. Lack of VCS best practises
5. Make the newbie experience better
5 Key Areas that came up over again

@petersouter
Hiera overload/Too much data
Hiera is simultaneously the best and worst thing to happen
to Puppet
14

@petersouter
YAML management is the biggest pain
point in the DevOps world...
Not just limited to Puppet...
15

@petersouter
Why is this a problem?
Two main issues...
16

@petersouter
First: Context switching is the mind killer
Humans suck at multitasking
17

@petersouter
18
“The trick here is that when you manage programmers, specifically,
task switches take a really, really, really long time. That's because
programming is the kind of task where you have to keep a lot of things
in your head at once. The more things you remember at once, the
more productive you are at programming. A programmer coding at
full throttle is keeping zillions of things in their head at once: everything
from names of variables, data structures, important APIs, the names of
utility functions that they wrote and call a lot, even the name of the
subdirectory where they store their source code.”
Human Task Switches Considered Harmful - Joel Spolsky
http://www.joelonsoftware.com/articles/fog0000000022.html

@petersouter
19
● Get the people who are maintaining the code to look at
what's written
● How long does it take them to find out how something’s
done in your code?
● Ask them if they can easily understand what’s
happening and why
To fix it: Get an outsider's perspective

@petersouter
Second: Maintaining this data is costly
Both technically and mentally
20

@petersouter
21
$ find . -type f | wc -l
587
Hiera can become bloated over time

@petersouter
22
● Minimise node specific data: it’s the hardest to maintain
and easiest to get stale
● Abstract information into a hierarchy to DRY it up
● Purge irrelevant data
● Remember, it’ll be in the git history still!
Clean up your hiera!

@petersouter
23
PR I’ve been meaning to resurrect:
maybe at the contributor summit?
- https://github.com/voxpupuli/puppet-syntax/pull/57

@petersouter
Lack of validation/CI
The earlier you can catch errors, the cheaper it is to fix
them.
24

@petersouter
25
Gareth’s talk about the Future of Puppet testing
https://speakerdeck.com/garethr/the-future-o
f-testing-puppet-code

@petersouter
26
● Puppet-syntax checks puppet, yaml, erb and epp files
● When you get that under control, use puppet-lint for
style checks
● Then rspec-puppet for unit tests
● Then beaker/testkitchen for acceptance tests
● If you don’t have CI, fix that! Or at least use git pre/post
commit hooks as a basic gate
At its most basic, a syntax error should not be
deployable

@petersouter
27
Testing with containers makes tests fast and
repeatable
http://cfgmgmtcamp.eu/schedule/testing/andy-henriod.html

@petersouter
Lack of VCS best practises
Monolithic repos make it hard to upgrade modules and for
multiple people to work on the module at one time
28

@petersouter
Make the newbie experience better
People change teams, new people join. Make sure they
can contribute as soon as possible
29

@petersouter
● If you want people to follow specific workflows or
requirements make it easy as possible
● Prevent risky newbie mistakes with CI and tooling
(puppet-lint, rubocop, rake tasks etc)
● Pair-programming is not just about the code: it’s about
learning the processes and improving the experience
for both new and old users
30
The bar should be low enough to trip over

@petersouter
313131
● Comments can be a crutch
● The best comment is a good name for a method or
variable
● Sometimes your hand is forced: make sure you
comment clearly and concisely
● Give the full context in the commit message
Code should be self-documenting

@petersouter
323232
https://blog.codinghorror.com/code-smells/
“There's a fine line between comments that illuminate and comments
that obscure. Are the comments necessary? Do they explain ‘why’ and
not ‘what’? Can you refactor the code so the comments aren't required?
And remember, you're writing comments for people, not
machines.”

@petersouter
333333
● Separate subject from body with a blank line
● Limit the subject line to 50 characters
● Capitalize the subject line
● Do not end the subject line with a period
● Use the imperative mood in the subject line
● Wrap the body at 72 characters
● Use the body to explain what and why vs. how
Git Commit best practices

@petersouter
343434
http://chris.beams.io/posts/git-commit/
“Re-establishing the context of a piece of code is wasteful. We can't
avoid it completely, so our efforts should go to reducing it [as much] as
possible. Commit messages can do exactly that and as a result, a
commit message shows whether a developer is a good
collaborator.”

@petersouter
35
Commit Often, Perfect Later, Publish Once
Learn how rebasing and amending works

@petersouter
363636
An example of good git commits at work
http://www.philandstuff.com/2014/02/09/git-pickaxe.html

@petersouter
373737
The Git Pickaxe
The reason I care about commit messages is because I'm an avid user of the git
pickaxe. If I'm ever confused about a line of code, and I want to know what
was going through the mind of the developer when they were writing it, the
pickaxe is the first tool I'll reach for. For example, let's say I was looking at this line
from our puppet-graphite module:
exec <%= @root_dir %>/bin/carbon-cache.py --debug start
That --debug option looks suspect. I might think to myself: "Why are we running
carbon-cache in --debug mode? Isn't that wasteful? Do we capture the output?
Why was it added in the first place?" In order to answer these questions, I'd like to
find the commit that added the switch.

@petersouter
383838
commit 5288d5804a3fc20dae4f3b2deeaa7f687595aff1
Author: Philip Potter <philip.g.potter@gmail.com>
Date: Tue Dec 17 09:33:59 2013 +0000
Re-add --debug option (reverts #11)
The --debug option is somewhat badly named -- it *both* adds debug
output, *and* causes carbon-cache to run in the foreground. Removing the
option in #11 caused the upstart script to lose track of the process as
carbon-cache started unexpectedly daemonizing.
Ideally we want to have a way of running through upstart without the
debug output, but this will fix the immediate problem.
Ta-dah, mystery solved!

@petersouter
Reinventing the wheel
There are existing modules, use them!
39

@petersouter
404040
The more existing boilerplate and modules you can
leverage:
● The less work you have to do (Yay for you!)
● The more maintainable it will be (Yay for your team!)
● The more supportable it will be (Yay for everyone!)
Start off on the right foot

@petersouter
4141414141
Start with the puppet control repo
41https://github.com/puppetlabs/control-repo

@petersouter
424242424242
Use as supported modules as your number 1 choice
https://forge.puppet.com/modules?endorsements=supported

@petersouter
4343
Use approved modules as your number 2 choice
43https://forge.puppet.com/modules?endorsements=approved

@petersouter
4444
If there’s no supported or approved, use your best judgement...
44https://forge.puppet.com/modules?utf-8=%E2%9C%93&sort=downloads&q=limits

@petersouter
When writing new modules, use a custom module skeleton
https://github.com/petems/petems-puppet-module-skeleton 45

@petersouter
46
Write as little as possible
Code is like a puppy: anything you write you are responsible
for

@petersouter
47
Summary
What have we learnt?

@petersouter
48
Get your hiera under control
Clean up old files, validate and try and abstract

@petersouter
49
Get validation and CI on your codebase
syntax, lint, spec, acceptance

@petersouter
50
Don’t reinvent the wheel
Try and reuse as much as possible,

@petersouter
51
Make sure you’re VCS is top notch
Git is the primary place to show your intentions and the
history of the code

@petersouter
52
Ensure your newbie experience is great
Make the bar low enough to fall over

@petersouter
● PuppetConf 2016: Enjoying the Journey from Puppet 3.x to 4.x – Rob
Nelson, AT&T 71 views
http://www.slideshare.net/PuppetLabs/puppetconf-2016-enjoying-the-journey-
from-puppet-3x-to-4x-rob-nelson-att
● Does Your Configuration Code Smell? - Tushar Sharma, Marios
Fragkoulis and Diomidis Spinellis - Dept of Management Science and
Technology
http://www.tusharma.in/wp-content/uploads/2016/03/ConfigurationSmells_pre
print.pdf
● Roles and profiles: A complete example
https://docs.puppet.com/pe/2016.5/r_n_p_full_example.html
● Puppet Design Patterns - David Danzilio, Kovarus
http://www.slideshare.net/DavidDanzilio/puppet-design-patterns-puppetconf
53
Want to know more?

@petersouter
Q&A
54

Knee deep in the undef - Tales from refactoring old Puppet codebases

More Related Content

What's hot

Viewers also liked

Similar to Knee deep in the undef - Tales from refactoring old Puppet codebases

More from Peter Souter

Recently uploaded

Knee deep in the undef - Tales from refactoring old Puppet codebases