Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Manageable Puppet 
infrastructure 
~September 2014 edition~ 
PuppetConf San Francisco 
Ger Apeldoorn - http://puppetspecia...
Who's this? 
Freelance Puppet Consultant 
Trainer for PuppetLabs Benelux 
Manageable Puppet 
infrastructure 
2 / 44 
~Sept...
Scope 
Also... why this talk? 
3 / 44 
Who's this? 
Freelance Puppet Consultant 
Trainer for PuppetLabs Benelux 
2 / 44
Common 
pitfalls 
4 / 44 
Scope 
Also... why this talk? 
3 / 44
Pitfalls 
Cause & effect Pitfalls 
Lots of Workarounds 
Unmaintainable codebase 
Collaboration difficulties 
Common 
pitfa...
Pitfalls 
Cause & effect 
Quick Wins 
Fix your codebase! 
Quick wins: 
Move data to Hiera 
Implement Code Review 
Use Pupp...
A Manageable Design 
September 2014 edition 
7 / 44 
Pitfalls 
Cause & effect 
Quick Wins 
Fix your codebase! 
Quick wins:...
A Manageable Design 
Requirements 
Whadda we need 
8 / 44 
September 2014 edition 
7 / 44
Our environment should be: 
Easy to Use 
Easy to Comprehend 
Easy to Update 
and... Safe 
9 / 44 
Requirements 
Whadda we ...
This stuff isn't 
exactly easy 
Our environment should be: 
10 / 44 
Easy to Use 
Easy to Comprehend 
Easy to Update 
and....
But we cán make it safe and 
manageable 
11 / 44 
This stuff isn't 
exactly easy 
10 / 44
Requirements 
Easy to: 
Use 
Comprehend 
Update 
Safe 
Safe 
Use environments to test everything 
Create a huge testing en...
Requirements 
Easy to: 
Use 
Comprehend 
Update 
Safe 
Manageable 
Manageable 
Keep a consistent module structure 
Using r...
Domains 
Server Roles 
All things data 
Deployment & Workflow 
14 / 44 
Requirements 
Easy to: 
Use 
Comprehend 
Update 
S...
Overview 
Software Components 
15 / 44 
Domains 
Server Roles 
All things data 
Deployment & Workflow 
14 / 44
Software Components 
Puppet Enterprise or The Foreman 
Hiera and hiera-eyaml (Hierarchical Data lookup) 
Gerrit (Code revi...
Software Components 
Puppet Enterprise or The Foreman 
Hiera and Domain hiera-eyaml #1: 
(Hierarchical Data lookup) 
Gerri...
A layer of abstraction 
18 / 44 
Domain #1: 
Server Roles 
17 / 44
How to do it? 
Create roles module 
root@puppet# puppet module generate gerapeldoorn-role 
Create a base-role to cover gen...
How to do it? -Cont'd- 
Put all required resources in the classes 
# modules/role/manifests/app.pp: 
class role::app { 
in...
Domain #2: 
All things Data 
21 / 44 
How to do it? -Cont'd- 
Put all required resources in the classes 
# modules/role/ma...
Hiera 
Hierarchical data lookup tool 
22 / 44 
Domain #2: 
All things Data 
21 / 44
It's all about Hierarchy 
Configured Hierarchy: 
#/etc/puppet/hiera.yaml: 
:hierarchy: 
- "%{::clientcert}" 
- "%{::enviro...
It's all about Hierarchy 
Types of Hieradata 
Regular values 
# hiera/app01.autiplan.com.yaml 
--- 
examplekey: value 
24 ...
Types of Hieradata 
Arrays 
# hiera/app01.autiplan.com.yaml 
--- 
array: [ item1, item2, item3 ] 
otherarray: 
- item1 
- ...
Types of Hieradata 
Hashes 
# hiera/app01.autiplan.com.yaml 
--- 
hash: 
key1: value 
key2: value 
26 / 44 
Types of Hiera...
Types of Hieradata 
Combinations 
# hiera/app01.autiplan.com.yaml 
--- 
hash: 
key1: value 
key2: value 
key3: 
- arrayval...
Hiera-related functions 
...and what to use them for 
28 / 44 
Types of Hieradata 
Combinations 
# hiera/app01.autiplan.co...
What does it do? 
Retrieves the first-found value in the 
hierarchy. (top-down) 
What to use it for? 
Basic variable-looku...
hiera_array('key' [, default_value]) (and hiera_hash) 
What does it do? 
Retrieves an array or hash value 
in the hierarch...
What does it do? 
hiera_array('key' [, default_value]) (and hiera_hash) 
Includes all classes listed in the 
array that is...
create_resources('type', HASH [, default_values]) 
What does it do? 
Generates resources from a 
HASH. 
What to use it for...
create_resources('type', HASH [, default_values]) 
Data bindings 
Example Hieradata 
# hiera/web01.autiplan.com.yaml 
--- ...
What does it do? 
Automatically loads class 
parameters from Hiera. 
What to use it for? 
Specify all class parameters in ...
Putting it all together 
Anything node-specific should be in Hiera! 
35 / 44 
What does it do? 
Automatically loads class ...
A Puppet Run: What calls what? 
Putting it all together 
36 / 44 
Anything node-specific should be in Hiera! 
35 / 44
Domain #3: 
Deployment & Workflow 
37 / 44 
A Puppet Run: What calls what? 
36 / 44
Environments 
Keeping the environmentalists happy 
38 / 44 
Domain #3: 
Deployment & Workflow 
37 / 44
Environments 
What is an environment? 
Seperate modulepaths/site.pp. 
Common environments: development, testing, productio...
Demo! 
40 / 44 
Environments 
What is an environment? 
Seperate modulepaths/site.pp. 
Common environments: development, te...
R10k overview 
41 / 44 
Demo! 
40 / 44
Final remarks 
Keep public modules as-is, wherever possible 
Create wrapper classes in company-module. 
Create fork if nee...
Questions? 
43 / 44 
Final remarks 
Keep public modules as-is, wherever possible 
Create wrapper classes in company-module...
Thank you! 
A howto of setting up this environment (and the workflow!) is available on my 
blog: http://puppetspecialist.n...
Thank you! 
A howto of setting up this environment (and the workflow!) is available on my 
blog: http://puppetspecialist.n...
Upcoming SlideShare
Loading in …5
×

Manageable Puppet Infrastructure - PuppetConf 2014

1,714 views

Published on

Manageable Puppet Infrastructure - Ger Apeldoorn, Freelance Puppet Consultant

Published in: Technology
  • Be the first to comment

Manageable Puppet Infrastructure - PuppetConf 2014

  1. 1. Manageable Puppet infrastructure ~September 2014 edition~ PuppetConf San Francisco Ger Apeldoorn - http://puppetspecialist.nl 1 / 44
  2. 2. Who's this? Freelance Puppet Consultant Trainer for PuppetLabs Benelux Manageable Puppet infrastructure 2 / 44 ~September 2014 edition~ PuppetConf San Francisco Ger Apeldoorn - http://puppetspecialist.nl 1 / 44
  3. 3. Scope Also... why this talk? 3 / 44 Who's this? Freelance Puppet Consultant Trainer for PuppetLabs Benelux 2 / 44
  4. 4. Common pitfalls 4 / 44 Scope Also... why this talk? 3 / 44
  5. 5. Pitfalls Cause & effect Pitfalls Lots of Workarounds Unmaintainable codebase Collaboration difficulties Common pitfalls 5 / 44 4 / 44
  6. 6. Pitfalls Cause & effect Quick Wins Fix your codebase! Quick wins: Move data to Hiera Implement Code Review Use Puppet-lint in a git-hook REFACTOR CONSTANTLY 6 / 44 Pitfalls Cause & effect Pitfalls Lots of Workarounds Unmaintainable codebase Collaboration difficulties 5 / 44
  7. 7. A Manageable Design September 2014 edition 7 / 44 Pitfalls Cause & effect Quick Wins Fix your codebase! Quick wins: Move data to Hiera Implement Code Review Use Puppet-lint in a git-hook REFACTOR CONSTANTLY 6 / 44
  8. 8. A Manageable Design Requirements Whadda we need 8 / 44 September 2014 edition 7 / 44
  9. 9. Our environment should be: Easy to Use Easy to Comprehend Easy to Update and... Safe 9 / 44 Requirements Whadda we need 8 / 44
  10. 10. This stuff isn't exactly easy Our environment should be: 10 / 44 Easy to Use Easy to Comprehend Easy to Update and... Safe 9 / 44
  11. 11. But we cán make it safe and manageable 11 / 44 This stuff isn't exactly easy 10 / 44
  12. 12. Requirements Easy to: Use Comprehend Update Safe Safe Use environments to test everything Create a huge testing environment Use Git to promote your code But we cán make it safe and 12 / 44 manageable 11 / 44
  13. 13. Requirements Easy to: Use Comprehend Update Safe Manageable Manageable Keep a consistent module structure Using roles for abstraction Facilitate collaboration 13 / 44 Requirements Easy to: Use Comprehend Update Safe Safe Use environments to test everything Create a huge testing environment Use Git to promote your code 12 / 44
  14. 14. Domains Server Roles All things data Deployment & Workflow 14 / 44 Requirements Easy to: Use Comprehend Update Safe Manageable Manageable Keep a consistent module structure Using roles for abstraction Facilitate collaboration 13 / 44
  15. 15. Overview Software Components 15 / 44 Domains Server Roles All things data Deployment & Workflow 14 / 44
  16. 16. Software Components Puppet Enterprise or The Foreman Hiera and hiera-eyaml (Hierarchical Data lookup) Gerrit (Code review system) Git (what else?) Git Flow, adapted version for Gerrit R10K (Environment deployment tool) 16 / 44 Overview Software Components 15 / 44
  17. 17. Software Components Puppet Enterprise or The Foreman Hiera and Domain hiera-eyaml #1: (Hierarchical Data lookup) Gerrit Server (Code review Roles system) Git (what else?) Git Flow, adapted version for Gerrit R10K (Environment deployment tool) 17 / 44 16 / 44
  18. 18. A layer of abstraction 18 / 44 Domain #1: Server Roles 17 / 44
  19. 19. How to do it? Create roles module root@puppet# puppet module generate gerapeldoorn-role Create a base-role to cover generic settings # modules/role/manifests/base.pp: class role::base { include users include ssh include motd ... 19 / 44 A layer of abstraction 18 / 44
  20. 20. How to do it? -Cont'd- Put all required resources in the classes # modules/role/manifests/app.pp: class role::app { include apache include tomcat apache::virtualhost { 'default': ... Include role in node definition # site.pp: node 'app01.autiplan.com' { include role::base include role::app } 20 / 44 How to do it? Create roles module root@puppet# puppet module generate gerapeldoorn-role Create a base-role to cover generic settings # modules/role/manifests/base.pp: class role::base { include users include ssh include motd ... 19 / 44
  21. 21. Domain #2: All things Data 21 / 44 How to do it? -Cont'd- Put all required resources in the classes # modules/role/manifests/app.pp: class role::app { include apache include tomcat apache::virtualhost { 'default': ... Include role in node definition # site.pp: node 'app01.autiplan.com' { include role::base include role::app } 20 / 44
  22. 22. Hiera Hierarchical data lookup tool 22 / 44 Domain #2: All things Data 21 / 44
  23. 23. It's all about Hierarchy Configured Hierarchy: #/etc/puppet/hiera.yaml: :hierarchy: - "%{::clientcert}" - "%{::environment}" - common Node app01.autiplan.com: environment: testing Hieradata # hiera/app01.autiplan.com.yaml --- examplekey: value for app01.autiplan.com # hiera/testing.yaml --- examplekey: value for nodes in testing environment # hiera/common.yaml --- examplekey: value for all nodes What will be in $test? $test = hiera('examplekey') 23 / 44 Hiera Hierarchical data lookup tool 22 / 44
  24. 24. It's all about Hierarchy Types of Hieradata Regular values # hiera/app01.autiplan.com.yaml --- examplekey: value 24 / 44 Configured Hierarchy: #/etc/puppet/hiera.yaml: :hierarchy: - "%{::clientcert}" - "%{::environment}" - common Node app01.autiplan.com: environment: testing Hieradata # hiera/app01.autiplan.com.yaml --- examplekey: value for app01.autiplan.com # hiera/testing.yaml --- examplekey: value for nodes in testing environment # hiera/common.yaml --- examplekey: value for all nodes What will be in $test? $test = hiera('examplekey') 23 / 44
  25. 25. Types of Hieradata Arrays # hiera/app01.autiplan.com.yaml --- array: [ item1, item2, item3 ] otherarray: - item1 - item2 - item3 Note: Never use tabs in Hiera files! 25 / 44 Types of Hieradata Regular values # hiera/app01.autiplan.com.yaml --- examplekey: value 24 / 44
  26. 26. Types of Hieradata Hashes # hiera/app01.autiplan.com.yaml --- hash: key1: value key2: value 26 / 44 Types of Hieradata Arrays # hiera/app01.autiplan.com.yaml --- array: [ item1, item2, item3 ] otherarray: - item1 - item2 - item3 Note: Never use tabs in Hiera files! 25 / 44
  27. 27. Types of Hieradata Combinations # hiera/app01.autiplan.com.yaml --- hash: key1: value key2: value key3: - arrayvalue1 - arrayvalue2 key4: subhashkey1: value subhashkey2: value 27 / 44 Types of Hieradata Hashes # hiera/app01.autiplan.com.yaml --- hash: key1: value key2: value 26 / 44
  28. 28. Hiera-related functions ...and what to use them for 28 / 44 Types of Hieradata Combinations # hiera/app01.autiplan.com.yaml --- hash: key1: value key2: value key3: - arrayvalue1 - arrayvalue2 key4: subhashkey1: value subhashkey2: value 27 / 44
  29. 29. What does it do? Retrieves the first-found value in the hierarchy. (top-down) What to use it for? Basic variable-lookup. Very easy to create exceptions! How to use it? $smarthost = hiera('smarthost') Example Hieradata # hiera/mail.autiplan.com.yaml --- smarthost: smtp.myprovider.nl # hiera/testing.yaml --- smarthost: testsmtp.autiplan.com # hiera/common.yaml --- smarthost: mail.autiplan.com hiera('key' [, default_value]) 29 / 44 Hiera-related functions ...and what to use them for 28 / 44
  30. 30. hiera_array('key' [, default_value]) (and hiera_hash) What does it do? Retrieves an array or hash value in the hierarchy, concatinates all found results What to use it for? Combining data from all hierarchy levels. How to use it? $users = hiera_array('users') Example Hieradata # hiera/app01.autiplan.com.yaml --- users: [ 'user1', 'user2' ] # hiera/testing.yaml --- users: [ 'testuser' ] # hiera/common.yaml --- users: [ 'user3', 'user4' ] 30 / 44 What does it do? Retrieves the first-found value in the hierarchy. (top-down) What to use it for? Basic variable-lookup. Very easy to create exceptions! How to use it? $smarthost = hiera('smarthost') Example Hieradata # hiera/mail.autiplan.com.yaml --- smarthost: smtp.myprovider.nl # hiera/testing.yaml --- smarthost: testsmtp.autiplan.com # hiera/common.yaml --- smarthost: mail.autiplan.com hiera('key' [, default_value]) 29 / 44
  31. 31. What does it do? hiera_array('key' [, default_value]) (and hiera_hash) Includes all classes listed in the array that is loaded from Hiera. Takes elements from ALL hierarchy levels. What to use it for? Lightweight ENC. Put all classes / roles in Hiera. How to use it? node default { hiera_include('roles') } Example Hieradata # hiera/web01.autiplan.com.yaml --- roles: - role::web # hiera/common.yaml --- roles: - role::base hiera_include('classes') 31 / 44 What does it do? Retrieves an array or hash value in the hierarchy, concatinates all found results What to use it for? Combining data from all hierarchy levels. How to use it? $users = hiera_array('users') Example Hieradata # hiera/app01.autiplan.com.yaml --- users: [ 'user1', 'user2' ] # hiera/testing.yaml --- users: [ 'testuser' ] # hiera/common.yaml --- users: [ 'user3', 'user4' ] 30 / 44
  32. 32. create_resources('type', HASH [, default_values]) What does it do? Generates resources from a HASH. What to use it for? Generate any resource based on data from Hiera. Can also be used with hiera_hash to create resources from all levels! How to use it? Example Hieradata # hiera/web01.autiplan.com.yaml --- vhosts: autiplan.com: alias: www.autiplan.com autiplan.dk: alias: www.autiplan.dk docroot: /var/www/html/autiplan.dk autiplan.nl: alias: www.autiplan.nl cdn.autiplan.com: port: 81 docroot: /var/www/html/cdn create_resources ('apache::vhost', hiera('vhosts', {})) 32 / 44 What does it do? Includes all classes listed in the array that is loaded from Hiera. Takes elements from ALL hierarchy levels. What to use it for? Lightweight ENC. Put all classes / roles in Hiera. How to use it? node default { hiera_include('roles') } Example Hieradata # hiera/web01.autiplan.com.yaml --- roles: - role::web # hiera/common.yaml --- roles: - role::base hiera_include('classes') 31 / 44
  33. 33. create_resources('type', HASH [, default_values]) Data bindings Example Hieradata # hiera/web01.autiplan.com.yaml --- vhosts: autiplan.com: alias: www.autiplan.com autiplan.dk: alias: www.autiplan.dk docroot: /var/www/html/autiplan.dk autiplan.nl: alias: www.autiplan.nl cdn.autiplan.com: port: 81 docroot: /var/www/html/cdn Auto-loading of Hiera data for parameterized classes. 33 / 44 What does it do? Generates resources from a HASH. What to use it for? Generate any resource based on data from Hiera. Can also be used with hiera_hash to create resources from all levels! How to use it? create_resources ('apache::vhost', hiera('vhosts', {})) 32 / 44
  34. 34. What does it do? Automatically loads class parameters from Hiera. What to use it for? Specify all class parameters in Hiera. Use all hierarchical benefits for class parameters. Simplify the use of parameterized classes. How to use it? include mysql::server Example Hieradata # hiera/web01.autiplan.com.yaml --- mysql::server::root_password: m0ars3cr3t # hiera/common.yaml --- mysql::server::root_password: t0ps3cr3t mysql::server::package_name: mysql-server mysql::server::restart: true Data bindings 34 / 44 Data bindings Auto-loading of Hiera data for parameterized classes. 33 / 44
  35. 35. Putting it all together Anything node-specific should be in Hiera! 35 / 44 What does it do? Automatically loads class parameters from Hiera. What to use it for? Specify all class parameters in Hiera. Use all hierarchical benefits for class parameters. Simplify the use of parameterized classes. How to use it? include mysql::server Example Hieradata # hiera/web01.autiplan.com.yaml --- mysql::server::root_password: m0ars3cr3t # hiera/common.yaml --- mysql::server::root_password: t0ps3cr3t mysql::server::package_name: mysql-server mysql::server::restart: true Data bindings 34 / 44
  36. 36. A Puppet Run: What calls what? Putting it all together 36 / 44 Anything node-specific should be in Hiera! 35 / 44
  37. 37. Domain #3: Deployment & Workflow 37 / 44 A Puppet Run: What calls what? 36 / 44
  38. 38. Environments Keeping the environmentalists happy 38 / 44 Domain #3: Deployment & Workflow 37 / 44
  39. 39. Environments What is an environment? Seperate modulepaths/site.pp. Common environments: development, testing, production. Nodes request a specific environment. Why? Essential to prevent mistakes. NEVER edit code in production! The workflow helps us to 'promote' our code to production. 39 / 44 Environments Keeping the environmentalists happy 38 / 44
  40. 40. Demo! 40 / 44 Environments What is an environment? Seperate modulepaths/site.pp. Common environments: development, testing, production. Nodes request a specific environment. Why? Essential to prevent mistakes. NEVER edit code in production! The workflow helps us to 'promote' our code to production. 39 / 44
  41. 41. R10k overview 41 / 44 Demo! 40 / 44
  42. 42. Final remarks Keep public modules as-is, wherever possible Create wrapper classes in company-module. Create fork if needed, submit pull request for fixes. Add forked module (gitrepo) to Puppetfile. Think ahead Always try to anticipate future applications. If it feels overly complicated, yer doin it wrong. Refactor! 42 / 44 R10k overview 41 / 44
  43. 43. Questions? 43 / 44 Final remarks Keep public modules as-is, wherever possible Create wrapper classes in company-module. Create fork if needed, submit pull request for fixes. Add forked module (gitrepo) to Puppetfile. Think ahead Always try to anticipate future applications. If it feels overly complicated, yer doin it wrong. Refactor! 42 / 44
  44. 44. Thank you! A howto of setting up this environment (and the workflow!) is available on my blog: http://puppetspecialist.nl/mpi Freelance Puppet Consultant Trainer for PuppetLabs Benelux 44 / 44 Questions? 43 / 44
  45. 45. Thank you! A howto of setting up this environment (and the workflow!) is available on my blog: http://puppetspecialist.nl/mpi Freelance Puppet Consultant Trainer for PuppetLabs Benelux 44 / 44

×