Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Designing Puppet: Roles/Profiles Pattern

79,878 views

Published on

"Designing Puppet: Roles/Profiles Pattern" by Craig Dunn of Puppet Labs, at Puppet Camp Stockholm 2013.

Published in: Technology
  • Hi All, We provide Salesforce training in a traditional Classroom setting at our training center where students gain hands-on training from our Salesforce Instructor as well as interact with other students. Classroom settings provide you a high level of retention and allow collaboration to encourage peer insight and team building. For more information feel free to contact us : siva@keylabstraining.com. For Course Content and Recorded Demo Click Here : http://www.keylabstraining.com/salesforce-online-training-hyderabad-bangalore
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Designing Puppet: Roles/Profiles Pattern

  1. 1. Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013Thursday, 7 February 13
  2. 2. Hello • Craig Dunn • Puppet user since 2008 as an IT contractor • Started with 0.24 • Joined Puppet Labs in June 2012 • @crayfishX • Freenode IRC: crayfishxThursday, 7 February 13
  3. 3. Agenda • How people typically design Puppet • Real-world case study • Thinking about components • Designing Puppet for your users • Node classification • Data separationThursday, 7 February 13
  4. 4. Background • Originally a blog post written in May 2012 • Advocated by many Puppet Labs Engineers • Based on a real world solution • Several community members have adopted with successThursday, 7 February 13
  5. 5. Designing Puppet • You write awesome modules • You classify them to your nodeThursday, 7 February 13
  6. 6. Designing Puppet Node Classification ModulesThursday, 7 February 13
  7. 7. Down the road... • Your infrastructure grows • Business requirements will change • Your Puppet code feels bulky and high maintenence • There will always be edge cases eventually • You decide it needs refactoringThursday, 7 February 13
  8. 8. Danger Signs • Resources being declared in two modules • You don’t know where your implementation “fits” • Lot’s of logic at a node level • Repetition and duplication • The if statement is your go-to-guyThursday, 7 February 13
  9. 9. Write good modules • Should manage only it’s own resources • Should be granular • Should be portableThursday, 7 February 13
  10. 10. Thinking beyond the module.... • Puppet is a code base • How do I design an effective framework • Gluing everything togetherThursday, 7 February 13
  11. 11. Node-level logic • Risks duplication and repetition • No guarantee of consistency • TMI!Thursday, 7 February 13
  12. 12. Node-level logic node basil { class { ‘apache’: version => ‘latest’, } class { ‘motd’: } class { ‘ssh’: } class { ‘users’: default_shell => ‘/bin/false’, } Class[‘ssh’] -> Class[‘users’] }Thursday, 7 February 13
  13. 13. Node-level logic • What happens when I have 1000 nodes • Or 10,000 nodes!! • That’s a lot of code! • So where should implement this?Thursday, 7 February 13
  14. 14. Designing Puppet • Provide business logic to classification • Provide an abstraction layer for implementation of components • Make code adaptable to complex requirements • Reduce node-level logic • Reduce functionality overlapThursday, 7 February 13
  15. 15. What is the worse thing that is going to happen to your Puppet code?Thursday, 7 February 13
  16. 16. Business requirementsThursday, 7 February 13
  17. 17. Business logic does not align with technologyThursday, 7 February 13
  18. 18. Case study • Real world problem • Solved through designThursday, 7 February 13
  19. 19. Case study “We have 3 applications we need to deploy using Puppet”Thursday, 7 February 13
  20. 20. The business view Application X Application Y Application ZThursday, 7 February 13
  21. 21. Go forth and Puppetize!Thursday, 7 February 13
  22. 22. Go forth and Puppetize! And we jumped right in...Thursday, 7 February 13
  23. 23. Things got painfulThursday, 7 February 13
  24. 24. Problems • These applications aren’t that different • They seem to share a whole bunch of similarities • Implementation differed on different environments and locations • Writing 3 separate modules creates conflicts and duplicationThursday, 7 February 13
  25. 25. Our code was hackyThursday, 7 February 13
  26. 26. We are trying to code business logic.Thursday, 7 February 13
  27. 27. Stop thinking about what it looks like • Break everything down into components • Granularity is the key • Think about what it actually isThursday, 7 February 13
  28. 28. What we realised • Each application stack is a collection of a subset of the same Java apps implemented in different waysThursday, 7 February 13
  29. 29. The business view Application X Application Y Application ZThursday, 7 February 13
  30. 30. The technical reality Application X ApplicationApplication Z YThursday, 7 February 13
  31. 31. We only have one application Implemented many different waysThursday, 7 February 13
  32. 32. So we had an idea! • Reduce each Java sub application into granular Puppet modules • Create a code layer responsible for implementation • Let’s call them profilesThursday, 7 February 13
  33. 33. class profiles::x { include tomcat include mysql include componenta include componentb componentb::resource { ‘name’: ensure => present, } } class profiles::y { include tomcat include mysql include componenta include componentc include componentd } class profiles::z { include tomcat include mysql include componenta include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’] }Thursday, 7 February 13
  34. 34. Use inheritance for abstraction within profiles class profiles::application { include tomcat include mysql include componenta } class profiles::application::x inherits profiles::application { include componentb componentb::resource { ‘name’: ensure => present, } } class profiles::application::y inherits profiles::application { include componentc include componentd } class profiles::application::z inherits profiles::application { include componentb include componentd include dependancy Class[‘dependancy’] -> Class[‘componentd’] }Thursday, 7 February 13
  35. 35. Profiles and Components ResourcesThursday, 7 February 13
  36. 36. Profiles and Components Components: Resource modelling ResourcesThursday, 7 February 13
  37. 37. Profiles and Components Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  38. 38. In reality it was worseThursday, 7 February 13
  39. 39. In reality it was worse • 2 different deployment types made up of over 15 server types eachThursday, 7 February 13
  40. 40. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locationsThursday, 7 February 13
  41. 41. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locations • 4 environment typesThursday, 7 February 13
  42. 42. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locations • 4 environment types • Every installation was an edge case!Thursday, 7 February 13
  43. 43. In reality it was worse • 2 different deployment types made up of over 15 server types each • 10+ locations • 4 environment types • Every installation was an edge case! • My slides weren’t big enough.Thursday, 7 February 13
  44. 44. Lessons learned • Granularity is good • Don’t assume business logic will directly translate to technology • Abstraction is awesome.... but that’s nothing new....Thursday, 7 February 13
  45. 45. Abstraction is a core principle of coding • Functions are abstracted by methods • Methods abstracted by classes and modules • They are abstracted with libraries • Puppet is code!Thursday, 7 February 13
  46. 46. Puppet is all about abstraction • Data is abstracted by Hiera • Providers are abstracted by types • Resources are abstracted by classes • Classes are abstracted by modulesThursday, 7 February 13
  47. 47. Puppet is all about abstraction • Data is abstracted by Hiera • Providers are abstracted by types • Resources are abstracted by classes • Classes are abstracted by modules • Modules are abstracted by profilesThursday, 7 February 13
  48. 48. Focussing on Abstraction • We’ve turned business logic into a technology stack • Can we translate that back into business logic? • Why would we even want to do that?Thursday, 7 February 13
  49. 49. UAT Cluster node Our example configuration model: include security include users include ntp include ssh::server include customapp include tomcat::server class { ‘jenkins’: require => Class[‘tomcat::server’], } include mysql database { ‘apptest’: ensure => present, }Thursday, 7 February 13
  50. 50. Think about the users Meet John, Susan and Bill.Thursday, 7 February 13
  51. 51. John is a Sysadmin • Wants to ensure all servers have kernel hardening, NTP and SSH Server installed • Wants to manage what packages, services, files and other resources • Is responsible for maintaining all the components of a UAT cluster serverThursday, 7 February 13
  52. 52. Susan is an application specialist • Cares that a UAT Cluster node requires MySQL Server, Tomcat Server and Jenkins server installed.Thursday, 7 February 13
  53. 53. Bill is an IT manager • Bill cares that the server is a UAT Cluster nodeThursday, 7 February 13
  54. 54. What do they care about? • John cares about modelling all resources • Susan cares about the technology stack • Bill cares about the business logicThursday, 7 February 13
  55. 55. In Puppet • Resource modelling is done in component modules • The technology stack is defined in profiles • Where do we represent the business logic for Bill?Thursday, 7 February 13
  56. 56. Introducing Roles • Represent business logic, not technology • Define a set of technology stacks (profiles) that make up the logical role • Allow the business to manage how the infrastructure looks without defining what it isThursday, 7 February 13
  57. 57. A node can only have one role • A role can include as many profiles as required to define itself • If a node requires two roles, it has by definition become a new roleThursday, 7 February 13
  58. 58. A node can only have one role • A role can include as many profiles as required to define itself • If a node requires two roles, it has by definition become a new role • Something couldn’t be a lion and a kangaroo at the same time!Thursday, 7 February 13
  59. 59. It would be a LingarooThursday, 7 February 13
  60. 60. Roles • One-to-one to nodes • One-to-many to profiles • Only implement profilesThursday, 7 February 13
  61. 61. Example role class role::uat_server { include profiles::base include profiles::customapp include profiles::test_tools }Thursday, 7 February 13
  62. 62. Classification • Node classification simply assigns roles to nodes • Roles expose profilesThursday, 7 February 13
  63. 63. Classification node ‘craig.puppetlabs.vm’ { include roles::uat_server }Thursday, 7 February 13
  64. 64. ClassificationThursday, 7 February 13
  65. 65. The Stack ResourcesThursday, 7 February 13
  66. 66. The Stack Components: Resource modelling ResourcesThursday, 7 February 13
  67. 67. The Stack Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  68. 68. The Stack Roles : Business Logic Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  69. 69. Terminology • Profiles and Roles are Puppet modules • Components are Puppet modules responsible for modelling resources • Everything is a moduleThursday, 7 February 13
  70. 70. Naming conventions • Components should be named after what they manage (apache, ssh, mysql) • Profiles should be named after the logical stack they implement (database, bastion, email) • Roles should be named in business logic convention (uat_server, web_cluster, application, archive)Thursday, 7 February 13
  71. 71. Hiera Overview Let’s talk about data!Thursday, 7 February 13
  72. 72. Managing infrastructure DevThursday, 7 February 13
  73. 73. Managing infrastructure Dev QAThursday, 7 February 13
  74. 74. Managing infrastructure Dev QA ProductionThursday, 7 February 13
  75. 75. Managing infrastructure Dev QA DC1 ProductionThursday, 7 February 13
  76. 76. Managing infrastructure Dev QA DC1 DC2 DC3 ProductionThursday, 7 February 13
  77. 77. Managing data in Puppet is hard.Thursday, 7 February 13
  78. 78. Without Hiera? if ( $::environment == ‘dev’ ) { $ntpserver = ‘192.168.2.1’ } else { if ( $::fqdn == ‘host4.mycorp.com’) { $ntpserver = ‘127.0.0.1’ } else { $ntpserver = ‘213.21.6.4’ } }Thursday, 7 February 13
  79. 79. With Hiera? $ntpserver = hiera(‘ntpserver’)Thursday, 7 February 13
  80. 80. Hierarchical lookups • Hiera uses facter facts to determine a hierarchy • Top down hierarchy for overriding configuration values based on roles, environments, locations.... or anything else • And do this without any coding!Thursday, 7 February 13
  81. 81. Separation of data from code • Puppet modules without hard-coded data are easily shared and more re-usable • Infrastructure configuration can be managed without needing to edit Puppet codeThursday, 7 February 13
  82. 82. Pluggable Backends • Source data from multiple locations • Data source is abstracted from codeThursday, 7 February 13
  83. 83. Pluggable Backends • Source data from multiple locations • Data source is abstracted from code • hiera-gpg • hiera-redis • hiera-http • hiera-json • hiera-mysql • hiera-zookeeperThursday, 7 February 13
  84. 84. Data Separation • Use Hiera to abstract your data from your code • Components and profiles can source data from HieraThursday, 7 February 13
  85. 85. Profiles and Hiera • Use Hiera to model your data • Use profiles to model your implementationThursday, 7 February 13
  86. 86. The Stack Roles : Business Logic Profiles : Implementation Components: Resource modelling ResourcesThursday, 7 February 13
  87. 87. The Stack Roles : Business Logic Profiles : Implementation Hiera: Data Components: Resource modelling ResourcesThursday, 7 February 13
  88. 88. Classification • Assigning classes to a node • You can classify within Puppet code (site.pp) • You can use an External Node Classifier (ENC)Thursday, 7 February 13
  89. 89. Leveraging an ENC • You can classify your nodes however you want • Puppet Dashboard • Enterprise Console • Foreman • Site.pp • Custom scriptThursday, 7 February 13
  90. 90. Leveraging an ENC • An ENC should classify a node to it’s role • Nothing elseThursday, 7 February 13
  91. 91. The Stack Roles : Business Logic Profiles : Implementation Hiera: Data Components: Resource modelling ResourcesThursday, 7 February 13
  92. 92. The Stack Roles : Business Logic Classifier Profiles : Implementation Hiera: Data Components: Resource modelling ResourcesThursday, 7 February 13
  93. 93. Key benefits • Reduced node-level logic to a role. • Gain the ability to be flexible with implementation • Business logic improves managability by non-Puppet users • Edge cases are now easy to solveThursday, 7 February 13
  94. 94. Enough Preaching!Thursday, 7 February 13
  95. 95. This is not the way to design Puppet... It’s a way.Thursday, 7 February 13
  96. 96. Can I implement this design without roles?Thursday, 7 February 13
  97. 97. Can I implement this design without roles? • Yes. • You lose the layer of abstraction that exposes business logicThursday, 7 February 13
  98. 98. Can my roles be defined in my ENC?Thursday, 7 February 13
  99. 99. Can my roles be defined in my ENC? • Yes. • Keeping it in code makes it versionableThursday, 7 February 13
  100. 100. Can’t I just use Hiera to define profiles?Thursday, 7 February 13
  101. 101. Can’t I just use Hiera to define profiles? • Technically yes. • You lose the flexibility to implement code logic in profiles and it may become restrictive • You could possibly use: https://github.com/ ripienaar/hiera-puppet-nodesThursday, 7 February 13
  102. 102. The fundamental concepts....Thursday, 7 February 13
  103. 103. The fundamental concepts.... • Abstraction, abstraction, abstractionThursday, 7 February 13
  104. 104. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling.Thursday, 7 February 13
  105. 105. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling. • Separating data and codeThursday, 7 February 13
  106. 106. The fundamental concepts.... • Abstraction, abstraction, abstraction • Decoupling business logic, implementation and resource modelling. • Separating data and code • Reducing node-level complexityThursday, 7 February 13
  107. 107. Other Resources • Adrien Thebos’ excellent blog post http:// sysadvent.blogspot.co.uk/2012/12/day-13-configuration-management-as- legos.html • My original blog post 2012/05/239/ http://www.craigdunn.org/ • Module Structure Redux by R.I.Pienaar http:// www.devco.net/archives/2012/12/13/simple-puppet-module-structure- redux.phpThursday, 7 February 13
  108. 108. Thank you. Questions? • Follow me at @crayfishX • Bug me on Freenode: crayfishx Enjoy the rest of Puppet Camp! In memory of Giles Constant, who spent many nights debating Puppet design patterns with me over copious amounts of beer and helped me on my journey of discovery learning how to implement Puppet properly. R.I.PThursday, 7 February 13

×