Puppet Camp Berlin 2014: Advanced Puppet Design

2,991 views

Published on

"Advanced Puppet Design" presented at Puppet Camp Berlin 2014 by Craig Dunn

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,991
On SlideShare
0
From Embeds
0
Number of Embeds
367
Actions
Shares
0
Downloads
156
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

Puppet Camp Berlin 2014: Advanced Puppet Design

  1. 1. Advanced Puppet Design Craig Dunn, Puppet Camp Berlin 2014 Friday, 11 April 14
  2. 2. Hello • Craig Dunn • Puppet user since 2008 • Previously worked for Puppet Labs • Founder of Enviatics • IT automation engineer and trainer • Based in Spain but work anywhere Friday, 11 April 14
  3. 3. My talk • Avoiding pain • Writing good modules • Challenges of codebase design • Roles / Profiles Friday, 11 April 14
  4. 4. But first.... Everyone loves polls, right? Friday, 11 April 14
  5. 5. Thinking right • Business has requirements • We use technology to fulfill them • But it’s not that simple! Friday, 11 April 14
  6. 6. “Business logic does not often align with technology design” Friday, 11 April 14
  7. 7. Business requirement “We have 3 applications we need to deploy using Puppet” Friday, 11 April 14
  8. 8. Puppetize • Write 3 Puppet modules for 3 applications • Because that was the requirement • Was it? Friday, 11 April 14
  9. 9. It’s not working Friday, 11 April 14
  10. 10. Let’s suppose • Each “application” is a set of shared components implemented different ways Friday, 11 April 14
  11. 11. The business view ApplicationY Application Z Application X Friday, 11 April 14
  12. 12. The technical reality ApplicationYApplication Z Application X Friday, 11 April 14
  13. 13. Danger Signs • Resources being declared in two modules • You don’t know where your implementation “fits” • Lot’s of logic at a node level • Repetition and duplication • The if statement is your go-to-guy Friday, 11 April 14
  14. 14. Catastrophic Signs if ($::hostname =~ /^host[0-3].*/) { package { ‘my-special-app’: ensure => installed, } } Friday, 11 April 14
  15. 15. Catastrophic Signs if !defined(Package[‘httpd’]) { package { ‘httpd’: ensure => installed, } } Friday, 11 April 14
  16. 16. Catastrophic Signs ensure_resource(‘package’,‘httpd’,{‘ensure’ => ‘installed’}) Friday, 11 April 14
  17. 17. Catastrophic Signs ensure_resource(‘package’,‘httpd’,{‘ensure’ => ‘installed’}) if function_defined_with_params(["#{type}[#{item}]", params])       Puppet.debug("Resource #{type}[#{item}] not created because it already exists")     else       Puppet::Parser::Functions.function(:create_resources)       function_create_resources([type.capitalize, { item => params }])     end Friday, 11 April 14
  18. 18. World ending signs • You use parser=future in production • You aren’t regretting it yet • You then implemented order=manifest Friday, 11 April 14
  19. 19. An unhappy Puppet Friday, 11 April 14
  20. 20. Stop thinking about what it looks like • Break everything down into components • Granularity is the key • Think about what it actually is Friday, 11 April 14
  21. 21. Writing modules • Granularity • Portability • Flexibility of implementation Friday, 11 April 14
  22. 22. A bad moduleclass web { $docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, } package { ‘php5’: ensure => installed, require => Package[‘httpd’], } package { ‘mysql-server’: ensure => installed, } file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), } } Friday, 11 April 14
  23. 23. A bad moduleclass web { $docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, } package { ‘php5’: ensure => installed, require => Package[‘httpd’], } package { ‘mysql-server’: ensure => installed, } file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), } } Granularity Friday, 11 April 14
  24. 24. A bad moduleclass web { $docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, } package { ‘php5’: ensure => installed, require => Package[‘httpd’], } package { ‘mysql-server’: ensure => installed, } file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), } } Portability Granularity Friday, 11 April 14
  25. 25. A bad moduleclass web { $docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, } package { ‘php5’: ensure => installed, require => Package[‘httpd’], } package { ‘mysql-server’: ensure => installed, } file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), } } Portability Implementation Granularity Friday, 11 April 14
  26. 26. Keep your modules granular • Manage only resources in the scope of the module • Small modules are ok! Friday, 11 April 14
  27. 27. A granular module class apache { $docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, } file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), } } Friday, 11 April 14
  28. 28. A granular module class apache { $docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, } file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), } } Granularity Friday, 11 April 14
  29. 29. A granular module class apache { $docroot=’/var/www/sites’ $listenaddr=‘10.0.1.10’ $servername=‘myweb.foo.com’ package { ‘httpd’: ensure => installed, } file { ‘/etc/httpd/httpd.conf’: ensure => file, content => template(‘web/httpd.conf.erb’), } } Granularity Portability Friday, 11 April 14
  30. 30. Sharing is good • Re-usable by others in your team • You can publish to the forge! • People will collaborate Friday, 11 April 14
  31. 31. It’s all about sharing! Friday, 11 April 14
  32. 32. You are gonna share your s**t aren’t you? Friday, 11 April 14
  33. 33. Sharing is not just pull requests • Share your ideas in blog posts • What worked? What didn’t? • Discuss and collaborate on mailing lists • IRC Friday, 11 April 14
  34. 34. Making sharing easier • Data separation (Hiera) • Allow the user flexibility of implementation Friday, 11 April 14
  35. 35. defaults params pattern • Use a parameterized class • Default from an inherited class • Allow the user to decide implementation Friday, 11 April 14
  36. 36. defaults pattern class apache { $packagename=‘httpd’ $docroot=’/var/www’ $listenaddr=‘10.0.1.12’ $servername=‘myweb.foo.com’ package { $packagename: ensure => installed, } ..... Friday, 11 April 14
  37. 37. defaults pattern class apache { $packagename=‘httpd’ $docroot=’/var/www’ $listenaddr=‘10.0.1.12’ $servername=‘myweb.foo.com’ package { $packagename: ensure => installed, } ..... In-module private data No way to override Friday, 11 April 14
  38. 38. defaults patternclass apache ( $packagename=‘httpd’, $docroot=’/var/www’, $listenaddr, $servername, ) { package { $packagename: ensure => installed, } ..... Friday, 11 April 14
  39. 39. defaults pattern class { ‘apache’: listenaddr => ‘10.0.1.2’, servername => ‘foo.example.com’, } Override data on implementation Friday, 11 April 14
  40. 40. defaults pattern # /etc/puppet/hieradata/dev.yaml --- apache::docroot: /var/dev/sites Overriding from Hiera Friday, 11 April 14
  41. 41. defaults pattern class apache::defaults { $packagename=$::osfamily ? { redhat => ‘httpd’, debian => ‘apache2’, default => ‘httpd’, } $docroot=’/var/www’ $listenaddr = $::ipaddress $servername = $::fqdn } ..... Friday, 11 April 14
  42. 42. defaults pattern class apache::defaults { $packagename=$::osfamily ? { redhat => ‘httpd’, debian => ‘apache2’, default => ‘httpd’, } $docroot=’/var/www’ $listenaddr = $::ipaddress $servername = $::fqdn } ..... Added logic to defaults Common defaults Friday, 11 April 14
  43. 43. defaults pattern class apache ( $packagename=$::apache::defaults::packagename, $docroot=$::apache::defaults::docroot, $listenaddr=$::apache::defaults::listenaddr, $servername=$::apache::defaults::servername, ) inherits apache::defaults { package { $::apache::packagename: ensure => installed, } ..... Friday, 11 April 14
  44. 44. defaults pattern Granularity Portability Implementation Smaller modules with smaller scope No hard coded data in modules User can decided how / where to override defaults without editing the module Friday, 11 April 14
  45. 45. Component Modules Friday, 11 April 14
  46. 46. Now to build something awesome ... but where? Friday, 11 April 14
  47. 47. Designing Puppet Component Modules Node Classification Friday, 11 April 14
  48. 48. Node-level logic node basil { class { ‘apache’: version => ‘latest’, } class { ‘motd’: } class { ‘ssh’: } class { ‘users’: default_shell => ‘/bin/false’, } Class[‘ssh’] -> Class[‘users’] } Friday, 11 April 14
  49. 49. Node-level logic node basil inherits base { class { ‘apache’: version => ‘latest’, } } Friday, 11 April 14
  50. 50. Node-level logic • Risks duplication and repetition • No guarantee of consistency • Pseudo nodes and inheritance trees will get messy, fast. • TMI! Friday, 11 April 14
  51. 51. Thinking beyond the module.... • Puppet is a code base • You need an effective framework • Gluing everything together Friday, 11 April 14
  52. 52. Introducing Profiles Friday, 11 April 14
  53. 53. Profiles • Wrapper classes implementing component modules • Define a logical technology stack • But just one! Friday, 11 April 14
  54. 54. Profiles class profile::blog { User <| group == ‘webadmins’ |> class { ‘::mysql::server’: } class { ‘::mysql::bindings’: php_enable => true, } class { ‘::wordpress’: install_dir => ‘/var/www/wp’, } } Friday, 11 April 14
  55. 55. Profiles • Component modules manage the resources • Profiles provide a layer of implementation Friday, 11 April 14
  56. 56. Profiles and Components Resources Friday, 11 April 14
  57. 57. Profiles and Components Resources Components: Resource modelling Friday, 11 April 14
  58. 58. Profiles and Components Resources Components: Resource modelling Profiles : Implementation Friday, 11 April 14
  59. 59. Lessons learned • Granularity is good • Don’t assume business logic will directly translate to technology • Abstraction is awesome.... but that’s nothing new.... Friday, 11 April 14
  60. 60. Abstraction is a core principle of coding • Implementation is abstracted by methods • Methods abstracted by classes and modules • They are abstracted with libraries • Puppet is code! Friday, 11 April 14
  61. 61. Puppet is all about abstraction • Providers are abstracted by types • Resources are abstracted by classes • Classes are abstracted by modules Friday, 11 April 14
  62. 62. Puppet is all about abstraction • Providers are abstracted by types • Resources are abstracted by classes • Classes are abstracted by modules •Modules are abstracted by profiles Friday, 11 April 14
  63. 63. Focussing on Abstraction • We’ve turned business logic into a technology stack • Can we translate that back into business logic? • Why would we even want to do that? Friday, 11 April 14
  64. 64. Introducing roles • Translate to business logic • Identify the function of a server in human terms • We never said business logic was a bad thing Friday, 11 April 14
  65. 65. Configuration model include profiles::security include profiles::users include profiles::networking include profiles::blog This is a “acme” server Friday, 11 April 14
  66. 66. Think about the users Meet John, Susan and Bill. Friday, 11 April 14
  67. 67. John is a Sysadmin • Wants to ensure all servers have kernel hardening, NTP and SSH Server installed • Wants to manage what packages, services, files and other resources • Is responsible for maintaining all the components of a any type of server Friday, 11 April 14
  68. 68. Susan is an application specialist • Cares that a the node has Wordpress and MySQL implemented properly • She probably doesn’t care about how sudoers is configured Friday, 11 April 14
  69. 69. Bill is an IT manager • Bill cares that the server is an ACME App server • He probably doesn’t understand what sudoers is Friday, 11 April 14
  70. 70. What do they care about? • John cares about modelling all resources • Susan cares about the technology stack • Bill cares about the business logic Friday, 11 April 14
  71. 71. In Puppet • Resource modelling is done in component modules • The technology stack is defined in profiles • Where do we represent the business logic for Bill? Friday, 11 April 14
  72. 72. Roles • Represent business logic, not technology • Define a set of technology stacks (profiles) that make up the logical role • Allow the business to manage how the infrastructure looks without defining what it is Friday, 11 April 14
  73. 73. A node can only have one role • A role can include as many profiles as required to define itself • If a node requires two roles, it has by definition become a new role Friday, 11 April 14
  74. 74. A node can only have one role • A role can include as many profiles as required to define itself • If a node requires two roles, it has by definition become a new role • Something couldn’t be a lion and a kangaroo at the same time! Friday, 11 April 14
  75. 75. It would be a Lingaroo Friday, 11 April 14
  76. 76. Roles • One-to-one to nodes • One-to-many to profiles • Only implement profiles, that’s it! Friday, 11 April 14
  77. 77. The Stack Resources Friday, 11 April 14
  78. 78. The Stack Resources Components: Resource modelling Friday, 11 April 14
  79. 79. The Stack Resources Components: Resource modelling Profiles : Implementation Friday, 11 April 14
  80. 80. The Stack Resources Components: Resource modelling Profiles : Implementation Roles : Business Logic Friday, 11 April 14
  81. 81. Role classes class role::acme { include profiles::security include profiles::users include profiles::networking include profiles::blog } This is a “acme” server Friday, 11 April 14
  82. 82. Terminology • Profiles and Roles are Puppet modules • They are not special • Everything is a module Friday, 11 April 14
  83. 83. Classification • Assigning classes to a node • You can classify within Puppet code (site.pp) • You can use an External Node Classifier (ENC) Friday, 11 April 14
  84. 84. Classification • You can classify your nodes however you want • Puppet Dashboard • Enterprise Console • Foreman • Site.pp • Custom script Friday, 11 April 14
  85. 85. Classification node ‘craig.puppetlabs.vm’ { include roles::acme_app } Friday, 11 April 14
  86. 86. Classification Friday, 11 April 14
  87. 87. Classification • With roles and profiles we just classify the role to the node Friday, 11 April 14
  88. 88. The Stack Resources Components: Resource modelling Profiles : Implementation Roles : Business Logic Friday, 11 April 14
  89. 89. The Stack Resources Components: Resource modelling Profiles : Implementation Roles : Business Logic Classifier Friday, 11 April 14
  90. 90. Data Separation If you’re not using Hiera you are going to break your data! Friday, 11 April 14
  91. 91. Data Separation • Use parameterized classes • Hiera data bindings • Component modules and profiles can look up data from Hiera • Roles should NOT Friday, 11 April 14
  92. 92. Roles and Profiles for DevOps • Full props to Laurent Bernaille from D2SI • Achieving Continuous Delivery and Devops with Puppet • Puppet Camp Paris, 2014. Friday, 11 April 14
  93. 93. Roles and Profiles for DevOps • Using roles and profiles makes it easier for developers and ops to all collaborate on Puppet • Developers write profiles for the their apps • Ops write profiles for their infrastructure • Roles encompass all of them Friday, 11 April 14
  94. 94. The Roles/Profiles Stack Resources Components: Resource modelling Roles : Business Logic Hiera: Data Classifier Classification Dev profiles Ops profiles Friday, 11 April 14
  95. 95. Roles/Profiles/Devops http://fr.slideshare.net/D2SI/d2-si-puppetcamp Friday, 11 April 14
  96. 96. Key benefits • Reduced node-level logic to a role. • Gain the ability to be flexible with implementation • Business logic improves managability by non-Puppet users • Edge cases are now easy to solve Friday, 11 April 14
  97. 97. Enough Preaching! Friday, 11 April 14
  98. 98. This is not the way to design Puppet... It’s a way. Friday, 11 April 14
  99. 99. Can I implement this design without roles? Friday, 11 April 14
  100. 100. Can I implement this design without roles? • Yes. • You lose the layer of abstraction that exposes business logic Friday, 11 April 14
  101. 101. Can my roles be defined in my ENC? Friday, 11 April 14
  102. 102. Can my roles be defined in my ENC? • Yes. • Keeping it in code makes it versionable Friday, 11 April 14
  103. 103. Can’t I just use Hiera to define profiles? Friday, 11 April 14
  104. 104. Can’t I just use Hiera to define profiles? • Technically yes. • You lose the flexibility to implement code logic in profiles and it may become restrictive Friday, 11 April 14
  105. 105. The fundamental concepts.... Friday, 11 April 14
  106. 106. The fundamental concepts.... • Abstraction, abstraction, abstraction Friday, 11 April 14
  107. 107. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling. Friday, 11 April 14
  108. 108. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling. • Separating data and code Friday, 11 April 14
  109. 109. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling. • Separating data and code • Reducing node-level complexity Friday, 11 April 14
  110. 110. Other resources • http://garylarizza.com/blog/2014/02/17/ puppet-workflow-part-2/ • http://www.craigdunn.org/2012/05/239/ Friday, 11 April 14
  111. 111. Danke. Questions? • Follow me at @crayfishX • Bug me on Freenode: crayfishx In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beer and helped me on my journey of discovery learning how to implement Puppet properly. R.I.P Friday, 11 April 14

×