0
Designing Puppet:                            Roles / Profiles Design Pattern                           Puppet Camp Stockhol...
Hello                     • Craig Dunn                     • Puppet user since 2008 as an IT contractor                   ...
Agenda                     • How people typically design Puppet                     • Real-world case study               ...
Background                     • Originally a blog post written in May 2012                     • Advocated by many Puppet...
Designing Puppet                     • You write awesome modules                     • You classify them to your nodeThurs...
Designing Puppet                             Node Classification                                 ModulesThursday, 7 Februar...
Down the road...                     • Your infrastructure grows                     • Business requirements will change  ...
Danger Signs                     • Resources being declared in two modules                     • You don’t know where your...
Write good modules                     • Should manage only it’s own resources                     • Should be granular   ...
Thinking beyond the                               module....                     • Puppet is a code base                  ...
Node-level logic                     • Risks duplication and repetition                     • No guarantee of consistency ...
Node-level logic          node basil {            class { ‘apache’:              version => ‘latest’,            }        ...
Node-level logic                     • What happens when I have 1000 nodes                     • Or 10,000 nodes!!        ...
Designing Puppet                     • Provide business logic to classification                     • Provide an abstractio...
What is the worse                              thing that is                          going to happen to                  ...
Business requirementsThursday, 7 February 13
Business logic does not                   align with technologyThursday, 7 February 13
Case study                     • Real world problem                     • Solved through designThursday, 7 February 13
Case study                                  “We have 3                                applications we                     ...
The business view                               Application X              Application Y                    Application ZT...
Go forth and                           Puppetize!Thursday, 7 February 13
Go forth and                           Puppetize!                          And we jumped right in...Thursday, 7 February 13
Things got painfulThursday, 7 February 13
Problems                     • These applications aren’t that different                     • They seem to share a whole b...
Our code was hackyThursday, 7 February 13
We are trying to code                        business logic.Thursday, 7 February 13
Stop thinking about                            what it looks like                     • Break everything down into compone...
What we realised                     • Each application stack is a collection of a                          subset of the ...
The business view                               Application X              Application Y                    Application ZT...
The technical reality                                   Application X                              ApplicationApplication ...
We only have one                             application                          Implemented many                        ...
So we had an idea!                     • Reduce each Java sub application into                          granular Puppet mo...
class profiles::x {        include tomcat        include mysql        include componenta        include componentb        ...
Use inheritance for abstraction within profiles     class profiles::application {       include tomcat       include mysql ...
Profiles and                          Components                             ResourcesThursday, 7 February 13
Profiles and                             Components                          Components: Resource modelling                ...
Profiles and                             Components                              Profiles : Implementation                  ...
In reality it was worseThursday, 7 February 13
In reality it was worse                     • 2 different deployment types made up of                          over 15 ser...
In reality it was worse                     • 2 different deployment types made up of                          over 15 ser...
In reality it was worse                     • 2 different deployment types made up of                          over 15 ser...
In reality it was worse                     • 2 different deployment types made up of                          over 15 ser...
In reality it was worse                     • 2 different deployment types made up of                          over 15 ser...
Lessons learned                     • Granularity is good                     • Don’t assume business logic will directly ...
Abstraction is a core                           principle of coding                     • Functions are abstracted by meth...
Puppet is all about                             abstraction                     • Data is abstracted by Hiera             ...
Puppet is all about                             abstraction                     • Data is abstracted by Hiera             ...
Focussing on                                   Abstraction                     • We’ve turned business logic into a       ...
UAT Cluster node                          Our example configuration model:                              include   security ...
Think about the users                          Meet John, Susan and Bill.Thursday, 7 February 13
John is a Sysadmin                     • Wants to ensure all servers have kernel                          hardening, NTP a...
Susan is an application                             specialist                     • Cares that a UAT Cluster node require...
Bill is an IT manager                     • Bill cares that the server is a UAT Cluster                          nodeThurs...
What do they care                               about?                     • John cares about modelling all resources     ...
In Puppet                     • Resource modelling is done in component                          modules                  ...
Introducing Roles                     • Represent business logic, not technology                     • Define a set of tech...
A node can only have                               one role                     • A role can include as many profiles as   ...
A node can only have                               one role                     • A role can include as many profiles as   ...
It would be a LingarooThursday, 7 February 13
Roles                     • One-to-one to nodes                     • One-to-many to profiles                     • Only im...
Example role                class role::uat_server {                  include profiles::base                  include prof...
Classification                     • Node classification simply assigns roles to                          nodes             ...
Classification                          node ‘craig.puppetlabs.vm’ {                            include roles::uat_server  ...
ClassificationThursday, 7 February 13
The Stack                            ResourcesThursday, 7 February 13
The Stack                          Components: Resource modelling                                    ResourcesThursday, 7 ...
The Stack                              Profiles : Implementation                          Components: Resource modelling   ...
The Stack                               Roles : Business Logic                              Profiles : Implementation      ...
Terminology                     • Profiles and Roles are Puppet modules                     • Components are Puppet modules...
Naming conventions                     • Components should be named after what                          they manage (apach...
Hiera Overview                            Let’s talk about data!Thursday, 7 February 13
Managing infrastructure                          DevThursday, 7 February 13
Managing infrastructure                          Dev                          QAThursday, 7 February 13
Managing infrastructure                             Dev                             QA                          Production...
Managing infrastructure                             Dev                             QA        DC1                         ...
Managing infrastructure                             Dev                             QA        DC1   DC2   DC3             ...
Managing data in                          Puppet is hard.Thursday, 7 February 13
Without Hiera?                          if ( $::environment == ‘dev’ ) {                            $ntpserver = ‘192.168....
With Hiera?                          $ntpserver = hiera(‘ntpserver’)Thursday, 7 February 13
Hierarchical lookups                     • Hiera uses facter facts to determine a                          hierarchy      ...
Separation of data from code                     • Puppet modules without hard-coded data                          are eas...
Pluggable Backends                     • Source data from multiple locations                     • Data source is abstract...
Pluggable Backends                     • Source data from multiple locations                     • Data source is abstract...
Data Separation                     • Use Hiera to abstract your data from your                          code             ...
Profiles and Hiera                     • Use Hiera to model your data                     • Use profiles to model your imple...
The Stack                      Roles : Business Logic                  Profiles : Implementation   Components: Resource mod...
The Stack                      Roles : Business Logic                  Profiles : Implementation                           ...
Classification                     • Assigning classes to a node                     • You can classify within Puppet code ...
Leveraging an ENC                     • You can classify your nodes however you                          want             ...
Leveraging an ENC                     • An ENC should classify a node to it’s role                     • Nothing elseThurs...
The Stack                      Roles : Business Logic                  Profiles : Implementation                           ...
The Stack                      Roles : Business Logic   Classifier                  Profiles : Implementation               ...
Key benefits                     • Reduced node-level logic to a role.                     • Gain the ability to be flexible...
Enough Preaching!Thursday, 7 February 13
This is not the way to                          design Puppet... It’s a                                   way.Thursday, 7 ...
Can I implement this                          design without roles?Thursday, 7 February 13
Can I implement this                          design without roles?                     • Yes.                     • You l...
Can my roles be                          defined in my ENC?Thursday, 7 February 13
Can my roles be                          defined in my ENC?                     • Yes.                     • Keeping it in ...
Can’t I just use Hiera                           to define profiles?Thursday, 7 February 13
Can’t I just use Hiera                           to define profiles?                     • Technically yes.                 ...
The fundamental                            concepts....Thursday, 7 February 13
The fundamental                            concepts....                     • Abstraction, abstraction, abstractionThursda...
The fundamental                               concepts....                     • Abstraction, abstraction, abstraction    ...
The fundamental                               concepts....                     • Abstraction, abstraction, abstraction    ...
The fundamental                               concepts....                     • Abstraction, abstraction, abstraction    ...
Other Resources                     • Adrien Thebos’ excellent blog post                           http://                ...
Thank you. Questions?                     • Follow me at @crayfishX                     • Bug me on Freenode: crayfishx     ...
Upcoming SlideShare
Loading in...5
×

Designing Puppet: Roles/Profiles Pattern

36,154

Published on

"Designing Puppet: Roles/Profiles Pattern" by Craig Dunn of Puppet Labs, at Puppet Camp Stockholm 2013.

Published in: Technology

Transcript of "Designing Puppet: Roles/Profiles Pattern"

  1. 1. Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013Thursday, 7 February 13
  2. 2. Hello • Craig Dunn • Puppet user since 2008 as an IT contractor • Started with 0.24 • Joined Puppet Labs in June 2012 • @crayfishX • Freenode IRC: crayfishxThursday, 7 February 13
  3. 3. Agenda • How people typically design Puppet • Real-world case study • Thinking about components • Designing Puppet for your users • Node classification • Data separationThursday, 7 February 13
  4. 4. Background • Originally a blog post written in May 2012 • Advocated by many Puppet Labs Engineers • Based on a real world solution • Several community members have adopted with successThursday, 7 February 13
  5. 5. Designing Puppet • You write awesome modules • You classify them to your nodeThursday, 7 February 13
  6. 6. Designing Puppet Node Classification ModulesThursday, 7 February 13
  7. 7. Down the road... • Your infrastructure grows • Business requirements will change • Your Puppet code feels bulky and high maintenence • There will always be edge cases eventually • You decide it needs refactoringThursday, 7 February 13
  8. 8. Danger Signs • Resources being declared in two modules • You don’t know where your implementation “fits” • Lot’s of logic at a node level • Repetition and duplication • The if statement is your go-to-guyThursday, 7 February 13
  9. 9. Write good modules • Should manage only it’s own resources • Should be granular • Should be portableThursday, 7 February 13
  10. 10. Thinking beyond the module.... • Puppet is a code base • How do I design an effective framework • Gluing everything togetherThursday, 7 February 13
  11. 11. Node-level logic • Risks duplication and repetition • No guarantee of consistency • TMI!Thursday, 7 February 13
  12. 12. Node-level logic node basil { class { ‘apache’: version => ‘latest’, } class { ‘motd’: } class { ‘ssh’: } class { ‘users’: default_shell => ‘/bin/false’, } Class[‘ssh’] -> Class[‘users’] }Thursday, 7 February 13
  13. 13. Node-level logic • What happens when I have 1000 nodes • Or 10,000 nodes!! • That’s a lot of code! • So where should implement this?Thursday, 7 February 13
  14. 14. Designing Puppet • Provide business logic to classification • Provide an abstraction layer for implementation of components • Make code adaptable to complex requirements • Reduce node-level logic • Reduce functionality overlapThursday, 7 February 13
  15. 15. What is the worse thing that is going to happen to your Puppet code?Thursday, 7 February 13
  16. 16. Business requirementsThursday, 7 February 13
  17. 17. Business logic does not align with technologyThursday, 7 February 13
  18. 18. Case study • Real world problem • Solved through designThursday, 7 February 13
  19. 19. Case study “We have 3 applications we need to deploy using Puppet”Thursday, 7 February 13
  20. 20. The business view Application X Application Y Application ZThursday, 7 February 13
  21. 21. Go forth and Puppetize!Thursday, 7 February 13
  22. 22. Go forth and Puppetize! And we jumped right in...Thursday, 7 February 13
  23. 23. Things got painfulThursday, 7 February 13
  24. 24. Problems • These applications aren’t that different • They seem to share a whole bunch of similarities • Implementation differed on different environments and locations • Writing 3 separate modules creates conflicts and duplicationThursday, 7 February 13
  25. 25. Our code was hackyThursday, 7 February 13
  26. 26. We are trying to code business logic.Thursday, 7 February 13
  27. 27. Stop thinking about what it looks like • Break everything down into components • Granularity is the key • Think about what it actually isThursday, 7 February 13
  28. 28. What we realised • Each application stack is a collection of a subset of the same Java apps implemented in different waysThursday, 7 February 13
  29. 29. The business view Application X Application Y Application ZThursday, 7 February 13
  30. 30. The technical reality Application X ApplicationApplication Z YThursday, 7 February 13
  31. 31. We only have one application Implemented many different waysThursday, 7 February 13
  32. 32. So we had an idea! • Reduce each Java sub application into granular Puppet modules • Create a code layer responsible for implementation • Let’s call them profilesThursday, 7 February 13
  33. 33. class profiles::x { include tomcat include mysql include componenta include componentb componentb::resource { ‘name’: ensure => present, } } class profiles::y { include tomcat include mysql include componenta include componentc include componentd } class profiles::z { include tomcat include mysql include componenta include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’] }Thursday, 7 February 13
  34. 34. Use inheritance for abstraction within profiles class profiles::application { include tomcat include mysql include componenta } class profiles::application::x inherits profiles::application { include componentb componentb::resource { ‘name’: ensure => present, } } class profiles::application::y inherits profiles::application { include componentc include componentd } class profiles::application::z inherits profiles::application { include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’] }Thursday, 7 February 13
  35. 35. Profiles and Components ResourcesThursday, 7 February 13
  36. 36. Profiles and Components Components: Resource modelling ResourcesThursday, 7 February 13
  37. 37. Profiles and Components Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  38. 38. In reality it was worseThursday, 7 February 13
  39. 39. In reality it was worse • 2 different deployment types made up of over 15 server types eachThursday, 7 February 13
  40. 40. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locationsThursday, 7 February 13
  41. 41. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locations • 4 environment typesThursday, 7 February 13
  42. 42. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locations • 4 environment types • Every installation was an edge case!Thursday, 7 February 13
  43. 43. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locations • 4 environment types • Every installation was an edge case! • My slides weren’t big enough.Thursday, 7 February 13
  44. 44. Lessons learned • Granularity is good • Don’t assume business logic will directly translate to technology • Abstraction is awesome.... but that’s nothing new....Thursday, 7 February 13
  45. 45. Abstraction is a core principle of coding • Functions are abstracted by methods • Methods abstracted by classes and modules • They are abstracted with libraries • Puppet is code!Thursday, 7 February 13
  46. 46. Puppet is all about abstraction • Data is abstracted by Hiera • Providers are abstracted by types • Resources are abstracted by classes • Classes are abstracted by modulesThursday, 7 February 13
  47. 47. Puppet is all about abstraction • Data is abstracted by Hiera • Providers are abstracted by types • Resources are abstracted by classes • Classes are abstracted by modules • Modules are abstracted by profilesThursday, 7 February 13
  48. 48. Focussing on Abstraction • We’ve turned business logic into a technology stack • Can we translate that back into business logic? • Why would we even want to do that?Thursday, 7 February 13
  49. 49. UAT Cluster node Our example configuration model: include security include users include ntp include ssh::server include customapp include tomcat::server class { ‘jenkins’: require => Class[‘tomcat::server’], } include mysql database { ‘apptest’: ensure => present, }Thursday, 7 February 13
  50. 50. Think about the users Meet John, Susan and Bill.Thursday, 7 February 13
  51. 51. John is a Sysadmin • Wants to ensure all servers have kernel hardening, NTP and SSH Server installed • Wants to manage what packages, services, files and other resources • Is responsible for maintaining all the components of a UAT cluster serverThursday, 7 February 13
  52. 52. Susan is an application specialist • Cares that a UAT Cluster node requires MySQL Server, Tomcat Server and Jenkins server installed.Thursday, 7 February 13
  53. 53. Bill is an IT manager • Bill cares that the server is a UAT Cluster nodeThursday, 7 February 13
  54. 54. What do they care about? • John cares about modelling all resources • Susan cares about the technology stack • Bill cares about the business logicThursday, 7 February 13
  55. 55. In Puppet • Resource modelling is done in component modules • The technology stack is defined in profiles • Where do we represent the business logic for Bill?Thursday, 7 February 13
  56. 56. Introducing Roles • Represent business logic, not technology • Define a set of technology stacks (profiles) that make up the logical role • Allow the business to manage how the infrastructure looks without defining what it isThursday, 7 February 13
  57. 57. A node can only have one role • A role can include as many profiles as required to define itself • If a node requires two roles, it has by definition become a new roleThursday, 7 February 13
  58. 58. A node can only have one role • A role can include as many profiles as required to define itself • If a node requires two roles, it has by definition become a new role • Something couldn’t be a lion and a kangaroo at the same time!Thursday, 7 February 13
  59. 59. It would be a LingarooThursday, 7 February 13
  60. 60. Roles • One-to-one to nodes • One-to-many to profiles • Only implement profilesThursday, 7 February 13
  61. 61. Example role class role::uat_server { include profiles::base include profiles::customapp include profiles::test_tools }Thursday, 7 February 13
  62. 62. Classification • Node classification simply assigns roles to nodes • Roles expose profilesThursday, 7 February 13
  63. 63. Classification node ‘craig.puppetlabs.vm’ { include roles::uat_server }Thursday, 7 February 13
  64. 64. ClassificationThursday, 7 February 13
  65. 65. The Stack ResourcesThursday, 7 February 13
  66. 66. The Stack Components: Resource modelling ResourcesThursday, 7 February 13
  67. 67. The Stack Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  68. 68. The Stack Roles : Business Logic Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  69. 69. Terminology • Profiles and Roles are Puppet modules • Components are Puppet modules responsible for modelling resources • Everything is a moduleThursday, 7 February 13
  70. 70. Naming conventions • Components should be named after what they manage (apache, ssh, mysql) • Profiles should be named after the logical stack they implement (database, bastion, email) • Roles should be named in business logic convention (uat_server, web_cluster, application, archive)Thursday, 7 February 13
  71. 71. Hiera Overview Let’s talk about data!Thursday, 7 February 13
  72. 72. Managing infrastructure DevThursday, 7 February 13
  73. 73. Managing infrastructure Dev QAThursday, 7 February 13
  74. 74. Managing infrastructure Dev QA ProductionThursday, 7 February 13
  75. 75. Managing infrastructure Dev QA DC1 ProductionThursday, 7 February 13
  76. 76. Managing infrastructure Dev QA DC1 DC2 DC3 ProductionThursday, 7 February 13
  77. 77. Managing data in Puppet is hard.Thursday, 7 February 13
  78. 78. Without Hiera? if ( $::environment == ‘dev’ ) { $ntpserver = ‘192.168.2.1’ } else { if ( $::fqdn == ‘host4.mycorp.com’) { $ntpserver = ‘127.0.0.1’ } else { $ntpserver = ‘213.21.6.4’ } }Thursday, 7 February 13
  79. 79. With Hiera? $ntpserver = hiera(‘ntpserver’)Thursday, 7 February 13
  80. 80. Hierarchical lookups • Hiera uses facter facts to determine a hierarchy • Top down hierarchy for overriding configuration values based on roles, environments, locations.... or anything else • And do this without any coding!Thursday, 7 February 13
  81. 81. Separation of data from code • Puppet modules without hard-coded data are easily shared and more re-usable • Infrastructure configuration can be managed without needing to edit Puppet codeThursday, 7 February 13
  82. 82. Pluggable Backends • Source data from multiple locations • Data source is abstracted from codeThursday, 7 February 13
  83. 83. Pluggable Backends • Source data from multiple locations • Data source is abstracted from code • hiera-gpg • hiera-redis • hiera-http • hiera-json • hiera-mysql • hiera-zookeeperThursday, 7 February 13
  84. 84. Data Separation • Use Hiera to abstract your data from your code • Components and profiles can source data from HieraThursday, 7 February 13
  85. 85. Profiles and Hiera • Use Hiera to model your data • Use profiles to model your implementationThursday, 7 February 13
  86. 86. The Stack Roles : Business Logic Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  87. 87. The Stack Roles : Business Logic Profiles : Implementation Hiera: Data Components: Resource modelling ResourcesThursday, 7 February 13
  88. 88. Classification • Assigning classes to a node • You can classify within Puppet code (site.pp) • You can use an External Node Classifier (ENC)Thursday, 7 February 13
  89. 89. Leveraging an ENC • You can classify your nodes however you want • Puppet Dashboard • Enterprise Console • Foreman • Site.pp • Custom scriptThursday, 7 February 13
  90. 90. Leveraging an ENC • An ENC should classify a node to it’s role • Nothing elseThursday, 7 February 13
  91. 91. The Stack Roles : Business Logic Profiles : Implementation Hiera: Data Components: Resource modelling ResourcesThursday, 7 February 13
  92. 92. The Stack Roles : Business Logic Classifier Profiles : Implementation Hiera: Data Components: Resource modelling ResourcesThursday, 7 February 13
  93. 93. Key benefits • Reduced node-level logic to a role. • Gain the ability to be flexible with implementation • Business logic improves managability by non-Puppet users • Edge cases are now easy to solveThursday, 7 February 13
  94. 94. Enough Preaching!Thursday, 7 February 13
  95. 95. This is not the way to design Puppet... It’s a way.Thursday, 7 February 13
  96. 96. Can I implement this design without roles?Thursday, 7 February 13
  97. 97. Can I implement this design without roles? • Yes. • You lose the layer of abstraction that exposes business logicThursday, 7 February 13
  98. 98. Can my roles be defined in my ENC?Thursday, 7 February 13
  99. 99. Can my roles be defined in my ENC? • Yes. • Keeping it in code makes it versionableThursday, 7 February 13
  100. 100. Can’t I just use Hiera to define profiles?Thursday, 7 February 13
  101. 101. Can’t I just use Hiera to define profiles? • Technically yes. • You lose the flexibility to implement code logic in profiles and it may become restrictive • You could possibly use: https://github.com/ ripienaar/hiera-puppet-nodesThursday, 7 February 13
  102. 102. The fundamental concepts....Thursday, 7 February 13
  103. 103. The fundamental concepts.... • Abstraction, abstraction, abstractionThursday, 7 February 13
  104. 104. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling.Thursday, 7 February 13
  105. 105. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling. • Separating data and codeThursday, 7 February 13
  106. 106. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling. • Separating data and code • Reducing node-level complexityThursday, 7 February 13
  107. 107. Other Resources • Adrien Thebos’ excellent blog post http:// sysadvent.blogspot.co.uk/2012/12/day-13-configuration-management-as- legos.html • My original blog post 2012/05/239/ http://www.craigdunn.org/ • Module Structure Redux by R.I.Pienaar http:// www.devco.net/archives/2012/12/13/simple-puppet-module-structure- redux.phpThursday, 7 February 13
  108. 108. Thank you. Questions? • Follow me at @crayfishX • Bug me on Freenode: crayfishx Enjoy the rest of Puppet Camp! In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beer and helped me on my journey of discovery learning how to implement Puppet properly. R.I.PThursday, 7 February 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×