The talk provides an overview of implementing Puppet in a large enterprise at Constant Contact, detailing the challenges, successes, and lessons learned from this journey. Key issues include a complex and tightly coupled codebase, lack of testing and documentation, and the need to modernize from an outdated version of Puppet. The future steps focus on improving interface design, testing culture, and addressing user needs for effective deployment.

1. 2 0 0 , 0 0 0 L I N E S L AT E R
O U R J O U R N E Y T O M A N A G E A B L E P U P P E T C O D E
D AV I D D A N Z I L I O
C O N S TA N T C O N TA C T

2. W H O A M I ?
• Puppet Evangelist at
Constant Contact
• Joined Constant Contact
in 2014
• SysAdmin background but
more of a developer these
days
• Second time speaking at
PuppetConf

4. W H AT I S C O N S TA N T C O N TA C T ?
B A C K G R O U N D

5. • Founded in 1995
• Headquarters in Waltham, MA
• Offices in San Francisco, NYC, London, among others
• ~ 1,500 Employees, ~ 350 developers, ~ 70 operations
• 160+ apps (SOA - microservices for hipsters)
• We’re hiring! (See me after for details)

6. W H AT ’ S T H I S TA L K A B O U T ?
• An overview of our environment and our challenges
• Our successes, failures, and lessons from trying to
transform Puppet in a large enterprise
• Plans for achieving the Puppet singularity
• My quest to understand and affect change in a
massively complex organization

7. R U N N I N G P U P P E T AT
S C A L E I S H A R D
T L ; D R

8. A L O N G T I M E A G O
I N A G A L A X Y FA R FA R A WAY

9. E A R LY A D O P T E R S Y N D R O M E
• Started using Puppet in 2009 with version 0.24.8
• You name the bug, we’ve probably seen it
• Forge didn’t exist so we had to write everything
• Puppet talent was (still is) hard to come by
• Writing Puppet was a little harder back then…

10. Feature 0.23.x 0.24.x 0.25.x 2.6.x 2.7.x 3.x 3.2.x 3.4.x
Dynamic Scope X X X X X
Appending to attributes in class inheritance (+>) X X X X X X X X
Multi-line C-style comments X X X X X X X
Arrays of resource references allowed in relationships X X X X X X X
Overrides in class inheritance X X X X X X X
Appending to variables in child scopes (+=) X X X X X X X
Class names starting with 0-9 X X X
Regular expressions in node definitions X X X X X X
Assigning expressions to variables X X X X X X
Regular expressions in conditionals/expresions X X X X X X
elsif in if statements X X X X X
Chaining Resources X X X X X
Hashes X X X X X
Class Parameters X X X X X
Run Stages X X X X X
The “in” operator X X X X X
$title, $name, and $module_name available in parameter lists X X X X X
Optional trailing comma in parameter lists X X X X
Hyphens/dashes allowed in variable names * X
Automatic class parameter lookup via data bindings X X X
“Unless” conditionals X X X
Iteration over arrays and hashes X X
The modulo (%) operator X X
$trusted hash X

11. • Lots of the code we wrote back then is still around…
• The company was growing so fast, nobody had time to
focus on Puppet
• Ops mandated that all configuration and app
deployment had to be done with Puppet
• Lots of push back
• Puppet became a dirty word

12. T H E G O O D
• Everything is in Puppet
• Everything is in Git
• Our infrastructure is built to be ephemeral
• We can provision physical nodes about as fast as we
can spin up a virtual machine
• Moving an app to OpenStack is just a matter of running
Puppet

13. T H E B A D
• Stuck on Puppet 2.7.3; lots of the code was written
before parameterized classes
• Most of our code lives in a single Git repo
• Testing is up to the developer
• No design beyond the initial implementation; most of
the code just evolved organically
• Propensity for home-grown solutions, and a habit of
thinking only in terms of the current use case

14. T H E U G LY
• Massive codebase (more on this in a second)
• Frightening complexity
• Tightly coupled codebase (massive understatement)
• Very few tests (effectively none)
• Very little documentation
• Too easy to get into production
• No dependency management

16. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E
8 , 2 6 7 P U P P E T 4 3 , 0 5 9 2 1 , 9 3 3 1 9 8 , 9 0 1
1 , 0 7 6 E R B 2 5 , 1 9 8 6 6 1 4 3 , 7 2 4
3 9 1 B O U R N E S H E L L 6 , 7 6 1 6 , 5 3 1 5 5 , 1 6 2
1 9 3 X M L 2 , 7 1 4 5 , 1 1 1 3 7 , 3 6 0
5 8 P E R L 4 , 1 9 9 6 , 5 2 0 2 3 , 8 6 6
5 4 2 R U B Y 4 , 9 2 2 1 , 7 2 0 2 2 , 6 6 8
5 1 P Y T H O N 2 , 9 5 4 1 , 6 2 2 1 1 , 2 6 8
6 7 S Q L 4 5 2 1 9 7 9 , 9 9 9
4 6 6 YA M L 6 2 3 8 4 2 8 , 7 9 9
1 0 0 O T H E R 2 , 5 3 9 6 , 0 3 4 8 , 3 2 8
1 1 , 2 1 1 T O TA L 9 3 , 4 2 1 5 0 , 5 7 6 5 2 0 , 0 7 5

17. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E
2 , 3 7 6 R U B Y 5 2 , 3 5 2 3 0 , 4 6 3 2 2 9 , 6 5 4
4 0 E R B 6 1 7 1 1 2 , 2 7 1
7 X M L 5 0 2 1 , 5 5 4
5 6 YA M L 1 6 4 1 , 5 1 1
3 2 B O U R N E S H E L L 4 0 2 4 7 9 1 , 6 3 8
1 6 J S O N 7 5 0 1 , 1 7 5
8 2 P U P P E T 1 5 9 4 6 6 9 4
2 3 S M A R T Y 3 8 0 3 6 9
2 L I S P 4 4 9 6 3 0 0
1 4 O T H E R 1 3 3 1 1 3 6 2 9
2 , 6 4 8 T O TA L 5 3 , 8 8 6 3 1 , 2 1 4 2 3 9 , 7 9 5

18. T H E J O U R N E Y B E G I N S

19. U N D E R S TA N D

20. I D E N T I F Y O U R U S E R S
• Consumers use Puppet to deploy their apps. They just
want Puppet to work.
• Developers may write Puppet modules or profiles.
They may consume Forge modules.
• Architects help shape the strategic direction of Puppet
in the organization.

21. • Consumers have a very different use case than
Architects or Developers.
• Most of our users are Consumers, so we should focus
on them.
• Consumers aren’t necessarily interested in learning
Puppet, but they need to use it to deploy their apps.
• Interfaces are very important to these people.

22. T E A C H

23. T E A C H I N G P U P P E T
• Most people working with Puppet didn’t really know the
language well
• But they had to get shit done
• Rampant propagation of bad patterns
• Wrote a 4-hour intro class and started teaching
• Get everybody on a common baseline, a good
foundation to build good habits
• Offer more advanced training to people who want it

24. U P G R A D E

25. A M O D E R N V E R S I O N O F P U P P E T
• Puppet 2.7.3 is buggy and slow
• Couldn’t use any modules from the Forge out of the
box because of trailing commas
• Puppet 2.7 was holding us back
• Offload file serving to Puppet 3 masters, cheap
performance win: bit.ly/1VfSwYM
• Problem: code isn’t compatible with Puppet 3

26. class mysql_site {
# Set some defaults
$mysql_log_rotate_enable=0
$mysql_log_rotation='daily'
$mysql_log_retention='14'
$mysql_slowlog_rotate_enable=0
$mysql_slowlog_rotation='weekly'
$mysql_slowlog_retention='4'
$mysql_errlog_rotate_enable=0
$mysql_errlog_rotation='monthly'
$mysql_errlog_retention='6'
$mysql_genlog_rotate_enable=0
$mysql_genlog_rotation='daily'
$mysql_genlog_retention='14'
...AND SO ON…
}
class mysql_site::dev inherits mysql_site {
# Overrides for Dev environment
$mysql_log_rotate_enable=1
$mysql_slowlog_rotate_enable=1
$mysql_errlog_rotate_enable=1
$mysql_genlog_rotate_enable=1
}
class mysql_site::dev::app
inherits mysql_site::dev {
# Overrides for app in dev environment
$mysql_slowlog_rotation='monthly'
$mysql_errlog_rotation='daily'
include mysql
}

27. M I N I M U M
V I A B L E R E FA C T O R

28. • We knew the code was bad, but we didn’t have the
resources to focus on a redesign
• Refactor just enough to get to Puppet 3
• Fix templates
• Eliminate dynamic scope
• Hiera played a big role in this
• Move hosts to Puppet 3 as code becomes compatible

29. TA K I N G T H E P L U N G E

30. • We built out a Puppet 3 infrastructure alongside the
legacy infrastructure
• Shared CA, so the clients could talk to both sets of
masters
• Move hosts over as the code becomes compatible
• Turns out this is really hard when there’s no standard
way to classify hosts

31. • We started with the easiest stuff: our own hosts
• Moved on to our CD apps
• In process of moving legacy apps now

32. M E A N W H I L E …

33. • We couldn’t stop time and do all this work in a vacuum,
life was still happening around us
• New modules are put into their own repos and
deployed with r10k
• We’ve kept Puppet 3 environment up to date. Started
on 3.4.3, now on 3.8.3
• New modules must have tests and documentation
• We’re using roles and profiles for new things

34. • CI workflow and infrastructure
• Testing environments, harnesses, and helpers
• Deployment tooling
• Knowledge of testing frameworks
• No official images

35. N E X T S T E P S

36. T H I N G S W E ’ R E TA C K L I N G N E X T
• Node classification
• Roles and Profiles
• Decoupling our code
• Desire for bespoke everything
• Writing modules with a single entry point
• Dependency management
• The Forge test

37. T H E F O R G E T E S T / P U P P E T A P P R O V E D
T E S T / D E F I N I T I O N O F D O N E
• Can this module be deployed to the Forge?
• Is there any proprietary data embedded?
• Is it modular enough to meet unforeseen use cases?
• Does it have a robust suite of tests?
• Does it conform to community style expectations?
• Is it fully documented?

38. I T ’ S A L L A B O U T T H AT I N T E R FA C E
• Make it easier to use our modules
• Our consumers care about the interface, not the
implementation
• Design better and fewer interfaces
• Modules with well defined interfaces for composability
and better dependency management
• Documentation and tests as part of the definition of
done

39. T E S T I N G
• Make testing easier
• Make testing automatic
• Make testing a part of the culture
• Teach people about BDD
• Show them how much easier and more reliable an
automated test can be than a manual one
• Eventually, make testing required

40. L E S S O N S L E A R N E D

41. • Know your audience
• Don’t try and please everybody
• It’s always going to take longer than you think
• Don’t forget about the supporting infrastructure
• People, process, and technology. In that order
• It’s never going to move as fast as you want it to. Walk,
don’t run

42. • Don’t let yourself become paralyzed by complexity
• Take notes, write down things you need to work on,
you’ll get to them eventually
• Always take the easy wins
• Focus on high impact improvements, help as many
people as you can
• Small improvements add up

43. There’s nothing more successful than success

44. L E S S O N S T O B E L E A R N E D

45. • How do we get people to feel invested in the design?
• How can we reenforce good behaviors?
• How can we make sure our massive codebase is being
regularly evaluated?
• How can (should?) we hold people accountable for
their Puppet code?
• How can we get operations folks to realize they’re
software developers?

46. T H E E N D

47. H O W T O F I N D M E
• @djdanzilio on Twitter
• danzilio on Freenode (you can usually find me in
#puppet-community, #puppet-dev, and #puppet)
• danzilio on GitHub and The Forge
• david (at) danzil (dot) io
• blog.danzilio.net

48. I M A G E C R E D I T S
• The space photos are from the NASA Image Galleries
and are either in the public domain or licensed under
CC-BY 2.0
• All other photos are mine or my employer’s