Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
2 0 0 , 0 0 0 L I N E S L AT E R
O U R J O U R N E Y T O M A N A G E A B L E P U P P E T C O D E
D AV I D D A N Z I L I O
...
W H O A M I ?
• Puppet Evangelist at
Constant Contact
• Joined Constant Contact
in 2014
• SysAdmin background but
more of ...
W H AT I S C O N S TA N T C O N TA C T ?
B A C K G R O U N D
• Founded in 1995
• Headquarters in Waltham, MA
• Offices in San Francisco, NYC, London, among others
• ~ 1,500 Employees,...
W H AT ’ S T H I S TA L K A B O U T ?
• An overview of our environment and our challenges
• Our successes, failures, and l...
R U N N I N G P U P P E T AT
S C A L E I S H A R D
T L ; D R
A L O N G T I M E A G O
I N A G A L A X Y FA R FA R A WAY
E A R LY A D O P T E R S Y N D R O M E
• Started using Puppet in 2009 with version 0.24.8
• You name the bug, we’ve probab...
Feature 0.23.x 0.24.x 0.25.x 2.6.x 2.7.x 3.x 3.2.x 3.4.x
Dynamic Scope X X X X X      
Appending to attributes in class in...
• Lots of the code we wrote back then is still around…
• The company was growing so fast, nobody had time to
focus on Pupp...
T H E G O O D
• Everything is in Puppet
• Everything is in Git
• Our infrastructure is built to be ephemeral
• We can prov...
T H E B A D
• Stuck on Puppet 2.7.3; lots of the code was written
before parameterized classes
• Most of our code lives in...
T H E U G LY
• Massive codebase (more on this in a second)
• Frightening complexity
• Tightly coupled codebase (massive un...
F I L E S L A N G U A G E B L A N K C O M M E N T C O D E
8 , 2 6 7 P U P P E T 4 3 , 0 5 9 2 1 , 9 3 3 1 9 8 , 9 0 1
1 , ...
F I L E S L A N G U A G E B L A N K C O M M E N T C O D E
2 , 3 7 6 R U B Y 5 2 , 3 5 2 3 0 , 4 6 3 2 2 9 , 6 5 4
4 0 E R ...
T H E J O U R N E Y B E G I N S
U N D E R S TA N D
I D E N T I F Y O U R U S E R S
• Consumers use Puppet to deploy their apps. They just
want Puppet to work.
• Developers m...
• Consumers have a very different use case than
Architects or Developers.
• Most of our users are Consumers, so we should ...
T E A C H
T E A C H I N G P U P P E T
• Most people working with Puppet didn’t really know the
language well
• But they had to get s...
U P G R A D E
A M O D E R N V E R S I O N O F P U P P E T
• Puppet 2.7.3 is buggy and slow
• Couldn’t use any modules from the Forge out...
class	mysql_site	{	
		#	Set	some	defaults	
		$mysql_log_rotate_enable=0	
		$mysql_log_rotation='daily'	
		$mysql_log_reten...
M I N I M U M
V I A B L E R E FA C T O R
• We knew the code was bad, but we didn’t have the
resources to focus on a redesign
• Refactor just enough to get to Puppe...
TA K I N G T H E P L U N G E
• We built out a Puppet 3 infrastructure alongside the
legacy infrastructure
• Shared CA, so the clients could talk to bot...
• We started with the easiest stuff: our own hosts
• Moved on to our CD apps
• In process of moving legacy apps now
M E A N W H I L E …
• We couldn’t stop time and do all this work in a vacuum,
life was still happening around us
• New modules are put into th...
• CI workflow and infrastructure
• Testing environments, harnesses, and helpers
• Deployment tooling
• Knowledge of testin...
N E X T S T E P S
T H I N G S W E ’ R E TA C K L I N G N E X T
• Node classification
• Roles and Profiles
• Decoupling our code
• Desire for...
T H E F O R G E T E S T / P U P P E T A P P R O V E D
T E S T / D E F I N I T I O N O F D O N E
• Can this module be deplo...
I T ’ S A L L A B O U T T H AT I N T E R FA C E
• Make it easier to use our modules
• Our consumers care about the interfa...
T E S T I N G
• Make testing easier
• Make testing automatic
• Make testing a part of the culture
• Teach people about BDD...
L E S S O N S L E A R N E D
• Know your audience
• Don’t try and please everybody
• It’s always going to take longer than you think
• Don’t forget abo...
• Don’t let yourself become paralyzed by complexity
• Take notes, write down things you need to work on,
you’ll get to the...
There’s nothing more successful than success
L E S S O N S T O B E L E A R N E D
• How do we get people to feel invested in the design?
• How can we reenforce good behaviors?
• How can we make sure our m...
T H E E N D
H O W T O F I N D M E
• @djdanzilio on Twitter
• danzilio on Freenode (you can usually find me in
#puppet-community, #pupp...
I M A G E C R E D I T S
• The space photos are from the NASA Image Galleries
and are either in the public domain or licens...
200,000 Lines Later: Our Journey to Manageable Puppet Code
200,000 Lines Later: Our Journey to Manageable Puppet Code
Upcoming SlideShare
Loading in …5
×

200,000 Lines Later: Our Journey to Manageable Puppet Code

397 views

Published on

Slides from a talk I gave at PuppetConf 2015.

Abstract: I joined Constant Contact in the Spring of 2014 to help transform their Puppet infrastructure. Constant Contact was a very early adopter of Puppet and had a hard time keeping up with changes to the language. When I got to Constant Contact we were stuck on a very old version of Puppet 2.7 because our code was heavily dependent on inheritance and dynamic scoping. There was no separation of data and code and 99% of the Puppet modules in use in the environment were homegrown. With over 267,000 lines of ancient code, I was completely overwhelmed with how to get us up to speed. This talk is about how we managed to accomplish this incredible feat in just over a year.

Published in: Software
  • Be the first to comment

200,000 Lines Later: Our Journey to Manageable Puppet Code

  1. 1. 2 0 0 , 0 0 0 L I N E S L AT E R O U R J O U R N E Y T O M A N A G E A B L E P U P P E T C O D E D AV I D D A N Z I L I O C O N S TA N T C O N TA C T
  2. 2. W H O A M I ? • Puppet Evangelist at Constant Contact • Joined Constant Contact in 2014 • SysAdmin background but more of a developer these days • Second time speaking at PuppetConf
  3. 3. W H AT I S C O N S TA N T C O N TA C T ? B A C K G R O U N D
  4. 4. • Founded in 1995 • Headquarters in Waltham, MA • Offices in San Francisco, NYC, London, among others • ~ 1,500 Employees, ~ 350 developers, ~ 70 operations • 160+ apps (SOA - microservices for hipsters) • We’re hiring! (See me after for details)
  5. 5. W H AT ’ S T H I S TA L K A B O U T ? • An overview of our environment and our challenges • Our successes, failures, and lessons from trying to transform Puppet in a large enterprise • Plans for achieving the Puppet singularity • My quest to understand and affect change in a massively complex organization
  6. 6. R U N N I N G P U P P E T AT S C A L E I S H A R D T L ; D R
  7. 7. A L O N G T I M E A G O I N A G A L A X Y FA R FA R A WAY
  8. 8. E A R LY A D O P T E R S Y N D R O M E • Started using Puppet in 2009 with version 0.24.8 • You name the bug, we’ve probably seen it • Forge didn’t exist so we had to write everything • Puppet talent was (still is) hard to come by • Writing Puppet was a little harder back then…
  9. 9. Feature 0.23.x 0.24.x 0.25.x 2.6.x 2.7.x 3.x 3.2.x 3.4.x Dynamic Scope X X X X X       Appending to attributes in class inheritance (+>) X X X X X X X X Multi-line C-style comments   X X X X X X X Arrays of resource references allowed in relationships   X X X X X X X Overrides in class inheritance   X X X X X X X Appending to variables in child scopes (+=)   X X X X X X X Class names starting with 0-9   X X X         Regular expressions in node definitions     X X X X X X Assigning expressions to variables     X X X X X X Regular expressions in conditionals/expresions     X X X X X X elsif in if statements       X X X X X Chaining Resources       X X X X X Hashes       X X X X X Class Parameters       X X X X X Run Stages       X X X X X The “in” operator       X X X X X $title, $name, and $module_name available in parameter lists       X X X X X Optional trailing comma in parameter lists         X X X X Hyphens/dashes allowed in variable names *         X       Automatic class parameter lookup via data bindings           X X X “Unless” conditionals           X X X Iteration over arrays and hashes             X X The modulo (%) operator             X X $trusted hash               X
  10. 10. • Lots of the code we wrote back then is still around… • The company was growing so fast, nobody had time to focus on Puppet • Ops mandated that all configuration and app deployment had to be done with Puppet • Lots of push back • Puppet became a dirty word
  11. 11. T H E G O O D • Everything is in Puppet • Everything is in Git • Our infrastructure is built to be ephemeral • We can provision physical nodes about as fast as we can spin up a virtual machine • Moving an app to OpenStack is just a matter of running Puppet
  12. 12. T H E B A D • Stuck on Puppet 2.7.3; lots of the code was written before parameterized classes • Most of our code lives in a single Git repo • Testing is up to the developer • No design beyond the initial implementation; most of the code just evolved organically • Propensity for home-grown solutions, and a habit of thinking only in terms of the current use case
  13. 13. T H E U G LY • Massive codebase (more on this in a second) • Frightening complexity • Tightly coupled codebase (massive understatement) • Very few tests (effectively none) • Very little documentation • Too easy to get into production • No dependency management
  14. 14. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E 8 , 2 6 7 P U P P E T 4 3 , 0 5 9 2 1 , 9 3 3 1 9 8 , 9 0 1 1 , 0 7 6 E R B 2 5 , 1 9 8 6 6 1 4 3 , 7 2 4 3 9 1 B O U R N E S H E L L 6 , 7 6 1 6 , 5 3 1 5 5 , 1 6 2 1 9 3 X M L 2 , 7 1 4 5 , 1 1 1 3 7 , 3 6 0 5 8 P E R L 4 , 1 9 9 6 , 5 2 0 2 3 , 8 6 6 5 4 2 R U B Y 4 , 9 2 2 1 , 7 2 0 2 2 , 6 6 8 5 1 P Y T H O N 2 , 9 5 4 1 , 6 2 2 1 1 , 2 6 8 6 7 S Q L 4 5 2 1 9 7 9 , 9 9 9 4 6 6 YA M L 6 2 3 8 4 2 8 , 7 9 9 1 0 0 O T H E R 2 , 5 3 9 6 , 0 3 4 8 , 3 2 8 1 1 , 2 1 1 T O TA L 9 3 , 4 2 1 5 0 , 5 7 6 5 2 0 , 0 7 5
  15. 15. F I L E S L A N G U A G E B L A N K C O M M E N T C O D E 2 , 3 7 6 R U B Y 5 2 , 3 5 2 3 0 , 4 6 3 2 2 9 , 6 5 4 4 0 E R B 6 1 7 1 1 2 , 2 7 1 7 X M L 5 0 2 1 , 5 5 4 5 6 YA M L 1 6 4 1 , 5 1 1 3 2 B O U R N E S H E L L 4 0 2 4 7 9 1 , 6 3 8 1 6 J S O N 7 5 0 1 , 1 7 5 8 2 P U P P E T 1 5 9 4 6 6 9 4 2 3 S M A R T Y 3 8 0 3 6 9 2 L I S P 4 4 9 6 3 0 0 1 4 O T H E R 1 3 3 1 1 3 6 2 9 2 , 6 4 8 T O TA L 5 3 , 8 8 6 3 1 , 2 1 4 2 3 9 , 7 9 5
  16. 16. T H E J O U R N E Y B E G I N S
  17. 17. U N D E R S TA N D
  18. 18. I D E N T I F Y O U R U S E R S • Consumers use Puppet to deploy their apps. They just want Puppet to work. • Developers may write Puppet modules or profiles. They may consume Forge modules. • Architects help shape the strategic direction of Puppet in the organization.
  19. 19. • Consumers have a very different use case than Architects or Developers. • Most of our users are Consumers, so we should focus on them. • Consumers aren’t necessarily interested in learning Puppet, but they need to use it to deploy their apps. • Interfaces are very important to these people.
  20. 20. T E A C H
  21. 21. T E A C H I N G P U P P E T • Most people working with Puppet didn’t really know the language well • But they had to get shit done • Rampant propagation of bad patterns • Wrote a 4-hour intro class and started teaching • Get everybody on a common baseline, a good foundation to build good habits • Offer more advanced training to people who want it
  22. 22. U P G R A D E
  23. 23. A M O D E R N V E R S I O N O F P U P P E T • Puppet 2.7.3 is buggy and slow • Couldn’t use any modules from the Forge out of the box because of trailing commas • Puppet 2.7 was holding us back • Offload file serving to Puppet 3 masters, cheap performance win: bit.ly/1VfSwYM • Problem: code isn’t compatible with Puppet 3
  24. 24. class mysql_site { # Set some defaults $mysql_log_rotate_enable=0 $mysql_log_rotation='daily' $mysql_log_retention='14' $mysql_slowlog_rotate_enable=0 $mysql_slowlog_rotation='weekly' $mysql_slowlog_retention='4' $mysql_errlog_rotate_enable=0 $mysql_errlog_rotation='monthly' $mysql_errlog_retention='6' $mysql_genlog_rotate_enable=0 $mysql_genlog_rotation='daily' $mysql_genlog_retention='14' ...AND SO ON… } class mysql_site::dev inherits mysql_site { # Overrides for Dev environment $mysql_log_rotate_enable=1 $mysql_slowlog_rotate_enable=1 $mysql_errlog_rotate_enable=1 $mysql_genlog_rotate_enable=1 } class mysql_site::dev::app inherits mysql_site::dev { # Overrides for app in dev environment $mysql_slowlog_rotation='monthly' $mysql_errlog_rotation='daily' include mysql }
  25. 25. M I N I M U M V I A B L E R E FA C T O R
  26. 26. • We knew the code was bad, but we didn’t have the resources to focus on a redesign • Refactor just enough to get to Puppet 3 • Fix templates • Eliminate dynamic scope • Hiera played a big role in this • Move hosts to Puppet 3 as code becomes compatible
  27. 27. TA K I N G T H E P L U N G E
  28. 28. • We built out a Puppet 3 infrastructure alongside the legacy infrastructure • Shared CA, so the clients could talk to both sets of masters • Move hosts over as the code becomes compatible • Turns out this is really hard when there’s no standard way to classify hosts
  29. 29. • We started with the easiest stuff: our own hosts • Moved on to our CD apps • In process of moving legacy apps now
  30. 30. M E A N W H I L E …
  31. 31. • We couldn’t stop time and do all this work in a vacuum, life was still happening around us • New modules are put into their own repos and deployed with r10k • We’ve kept Puppet 3 environment up to date. Started on 3.4.3, now on 3.8.3 • New modules must have tests and documentation • We’re using roles and profiles for new things
  32. 32. • CI workflow and infrastructure • Testing environments, harnesses, and helpers • Deployment tooling • Knowledge of testing frameworks • No official images
  33. 33. N E X T S T E P S
  34. 34. T H I N G S W E ’ R E TA C K L I N G N E X T • Node classification • Roles and Profiles • Decoupling our code • Desire for bespoke everything • Writing modules with a single entry point • Dependency management • The Forge test
  35. 35. T H E F O R G E T E S T / P U P P E T A P P R O V E D T E S T / D E F I N I T I O N O F D O N E • Can this module be deployed to the Forge? • Is there any proprietary data embedded? • Is it modular enough to meet unforeseen use cases? • Does it have a robust suite of tests? • Does it conform to community style expectations? • Is it fully documented?
  36. 36. I T ’ S A L L A B O U T T H AT I N T E R FA C E • Make it easier to use our modules • Our consumers care about the interface, not the implementation • Design better and fewer interfaces • Modules with well defined interfaces for composability and better dependency management • Documentation and tests as part of the definition of done
  37. 37. T E S T I N G • Make testing easier • Make testing automatic • Make testing a part of the culture • Teach people about BDD • Show them how much easier and more reliable an automated test can be than a manual one • Eventually, make testing required
  38. 38. L E S S O N S L E A R N E D
  39. 39. • Know your audience • Don’t try and please everybody • It’s always going to take longer than you think • Don’t forget about the supporting infrastructure • People, process, and technology. In that order • It’s never going to move as fast as you want it to. Walk, don’t run
  40. 40. • Don’t let yourself become paralyzed by complexity • Take notes, write down things you need to work on, you’ll get to them eventually • Always take the easy wins • Focus on high impact improvements, help as many people as you can • Small improvements add up
  41. 41. There’s nothing more successful than success
  42. 42. L E S S O N S T O B E L E A R N E D
  43. 43. • How do we get people to feel invested in the design? • How can we reenforce good behaviors? • How can we make sure our massive codebase is being regularly evaluated? • How can (should?) we hold people accountable for their Puppet code? • How can we get operations folks to realize they’re software developers?
  44. 44. T H E E N D
  45. 45. H O W T O F I N D M E • @djdanzilio on Twitter • danzilio on Freenode (you can usually find me in #puppet-community, #puppet-dev, and #puppet) • danzilio on GitHub and The Forge • david (at) danzil (dot) io • blog.danzilio.net
  46. 46. I M A G E C R E D I T S • The space photos are from the NASA Image Galleries and are either in the public domain or licensed under CC-BY 2.0 • All other photos are mine or my employer’s

×