Puppet Camp Paris 2014: Achieving Continuous Delivery and DevOps with Puppet

1,421 views

Published on

Puppet Camp Paris 2014: "Achieving Continuous Delivery and DevOps with Puppet" presented by Laurent Bernaille, D2SI

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,421
On SlideShare
0
From Embeds
0
Number of Embeds
146
Actions
Shares
0
Downloads
66
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Puppet Camp Paris 2014: Achieving Continuous Delivery and DevOps with Puppet

  1. 1. Achieving Continuous Delivery with puppet Laurent Bernaille, D2SI #PuppetCampParis @D2SI@D2SI #PuppetCampParis
  2. 2. Past (current?) situation Why do we need continuous delivery ?
  3. 3. Applications do not answer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly June
  4. 4. Applications do not answer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly June What the end user say he needs in January January
  5. 5. What the dev team delivers in June June Applications do not answer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly What the end user say he needs in January January
  6. 6. What the dev team delivers in June June What the end user needs in June Applications do not answer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly June What the end user say he needs in January January
  7. 7. Development processes are inefficient Bug are detected too late Code TESTOther work Fix
  8. 8. Development processes are inefficient Bug are detected too late Integration Hell Code TESTOther work Fix Start integration
  9. 9. Development processes are inefficient Bug are detected too late Integration Hell A lot of wasted time Code TESTOther work Fix Request Approve & Prioritize Technical Assessment Code & Test Verify & Fix Deploy 20 min 2 min 15 min 2 h 4 h 3 min ½ week 2 weeks 2 weeks 1week ½ week Processing Time = 6 h 40 min Waiting Time = 6 weeks Adapted from Implementing Lean Software Development: From Concept to Cash, Mary & Tom Poppendieck. Start integration
  10. 10. OPS view Relationship with OPS can be “difficult” Application deployment is a nightmare
  11. 11. OPS view Relationship with OPS can be “difficult” Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare
  12. 12. OPS view ? Relationship with OPS can be “difficult” Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers»
  13. 13. OPS view ? Relationship with OPS can be “difficult” DEV view Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers»
  14. 14. OPS view ? Relationship with OPS can be “difficult” DEV view Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers» Identical servers are always “slightly” different
  15. 15. OPS view ? Relationship with OPS can be “difficult” DEV view Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers» Identical servers are always “slightly” different OPS always say “no” Standards do not evolve 10 deploys per day, Dev & ops cooperation at Flickr John Allspaw & Paul Hammond (Velocity 2009)
  16. 16. 6+ months to setup a new environment Infrastructure is not very agile Server “hoarding”
  17. 17. Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
  18. 18. Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
  19. 19. Production Infra setup Deploy Deploy Deploy Deploy Utilization : 100% Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
  20. 20. Production Infra setup Deploy Deploy Deploy Deploy Utilization : 100% Preproduction Utilization : 10% Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
  21. 21. Production Infra setup Deploy Deploy Deploy Deploy Utilization : 100% Preproduction Utilization : 10% Test Utilization : 40% Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
  22. 22. Business Development Operations • Applications do not answer business needs well • Too long to get new features • Integration and bug fixing is painful • A lot of wasted time • Deployments are very painful • A lot of misunderstanding • Environment setup is too slow • No on-demand resources Summary of the issues WHAT?
  23. 23. IT should be easier Continuous Delivery
  24. 24. Agile Manifesto, 2001 Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan Agile Development Business Development Operations A G I L E
  25. 25. Test Discover Design Develop Waterfall method Developing Incrementally and Iteratively Long cycles High-risk Adaptation is very difficult 6-12 month
  26. 26. 2-4 weeks Test Discover Design Develop Discover Design Develop Test Discover Design Develop Test Discover Design Develop Test Waterfall method Developing Incrementally and Iteratively Agile method Long cycles High-risk Adaptation is very difficult 6-12 month Low-risk Many short cycles Adaptation is much easier VS
  27. 27. Maintain a single source repository Automate the build Make your build self-testing Every commit should build on an integration machine Keep the build fast Test in a clone of the production environment Make it easy for anyone to get the latest executable Everyone can see what’s happening Detect problems early and solve them quickly Continuous Integration C I Business Development Operations
  28. 28. Continuous Integration Discover Design Develop Test Repository Version Control System Source Code Continuous Integration Feedback Source Code Build Analyze Code Test Continuous Integration Agile
  29. 29. DevOps : bring the wall down Business Development Operations D E V O P S Measure, Analyze & Describe Constraints (from DEV and OPS) Best practices, methods Automation Measure Share Culture Align objectives on business needs Innovate Automate application delivery
  30. 30. Continuous Delivery Discover Design Develop Test Repository Version Control System Source Code Continuous Integration Feedback Source Code Build Analyze Code Test Deploy Env Measure & analyze Continuous Integration Agile DevOps
  31. 31. Agile infra Control resources Configure resources Deploy applications Create, Delete, start, stop Servers (physical, virtual, in the cloud) Storage volumes, networks Define system states, when possible Verify system states Reconfigure systems when necessary Provide a service to deploy applications Automated, with rollbacks Agile Infrastructure Business Development Operations
  32. 32. Agile infra Control resources Configure resources Deploy applications Create, Delete, start, stop Servers (physical, virtual, in the cloud) Storage volumes, networks Define system states, when possible Verify system states Reconfigure systems when necessary Provide a service to deploy applications Automated, with rollbacks Automation API API API Agile Infrastructure Business Development Operations
  33. 33. Continuous Delivery Discover Design Develop Test Repository Version Control System Source Code Continuous Integration Feedback Source Code Build Analyze Code Test Deploy Env Measure & analyze Continuous Integration Agile DevOps ConfigureProvision Agile infrastructure
  34. 34. What about puppet ?
  35. 35. Discover Design Develop Test Repository Version Control System Source Code Continuous Integration Feedback Source Code Build Analyze Code Test Deploy Env Measure & analyze Continuous Integration Agile DevOps ConfigureProvision Agile infrastructure Puppet use cases
  36. 36. Puppet Master • Resolvers, time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes Server team uses puppet as a configuration tool
  37. 37. Puppet Master • Resolvers, time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes 4: apply/test catalog Puppet agentPuppet agentPuppet agent Server team uses puppet as a configuration tool
  38. 38. Puppet Master • Resolvers, time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes 4: apply/test catalog McollectiveGet info on nodes Run agent on subset of nodes Puppet agentPuppet agentPuppet agent Server team uses puppet as a configuration tool
  39. 39. Puppet Master • Resolvers, time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes 4: apply/test catalog McollectiveGet info on nodes Run agent on subset of nodes Puppet agentPuppet agentPuppet agent  Most common puppet usage.  Proven for large scale deployment.  Very “Infra oriented”: not opened to applications. Server team uses puppet as a configuration tool
  40. 40. Binaries of the application • Puppet manifests and modules describing deployments Repository Binaries Manifest Modules 1: put Use case 2 : deploy applications Developers supply :
  41. 41. Binaries of the application • Puppet manifests and modules describing deployments Dev servers Test servers Dev serversOther env Repository Binaries Manifest Modules 1: put 2: get 4: run Use case 2 : deploy applications • Chooses servers (env) where the deployment should happen • Runs puppet and gathers reports; if run fails, forwards to DEV Developers supply : OPS team :
  42. 42. Binaries of the application • Puppet manifests and modules describing deployments Dev servers Test servers Dev serversOther env Repository Binaries Manifest Modules 1: put 2: get 4: run Use case 2 : deploy applications • Chooses servers (env) where the deployment should happen • Runs puppet and gathers reports; if run fails, forwards to DEV  Much more efficient than written deployment processes.  Much easier to understand what fails. Developers supply : OPS team :
  43. 43. Configuration as a service ProfilsLevels ConfigurationReport Tools
  44. 44. Using a “configuration service” Env Configuration service Deploy API GUI • Associate “profiles” to nodes, define variables • Run configuration and get reports Give application teams the possibility to configure servers1
  45. 45. Using a “configuration service” Env Configuration service Deploy API GUI • Associate “profiles” to nodes, define variables • Run configuration and get reports Give application teams the possibility to configure servers1 Different levels of configuration, different responsibilities2 Base OS configuration Standard middlewares Application middlewares Application
  46. 46. Can we do that with puppet? Sure, but tricky with classic DEV / OPS model
  47. 47. Can we do that with puppet? Sure, but tricky with classic DEV / OPS model DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
  48. 48. Base OS configuration Standard middlewares Application middlewares Application2 1 Can we do that with puppet? Some options : Sure, but tricky with classic DEV / OPS model • Second puppet master, or puppet apply (non root) • Other tool Tool separation1 DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
  49. 49. Base OS configuration Standard middlewares Application middlewares Application2 1 Can we do that with puppet? Some options : Sure, but tricky with classic DEV / OPS model • Second puppet master, or puppet apply (non root) • Other tool Tool separation1 Ok to run as root but under full control2 • Custom “profile” facts (facts.d) and hiera • Run with mcollective (limit to some tags) • Read-only console access DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
  50. 50. Base OS configuration Standard middlewares Application middlewares Application2 1 Can we do that with puppet? Some options : Sure, but tricky with classic DEV / OPS model • Second puppet master, or puppet apply (non root) • Other tool Tool separation1 Ok to run as root but under full control2 • Custom “profile” facts (facts.d) and hiera • Run with mcollective (limit to some tags) • Read-only console access Many other ways3 DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
  51. 51. Approach 1 : OPS write all modules Version Control System I need a mongodb module CI Puppet master mongodb module What if DEV need custom modules (they will)
  52. 52. Approach 1 : OPS write all modules  Impossible to scale  Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module What if DEV need custom modules (they will)
  53. 53. Approach 1 : OPS write all modules  Impossible to scale  Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module What if DEV need custom modules (they will) Approach 2 : Pull request mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
  54. 54. Approach 1 : OPS write all modules  Impossible to scale  Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module  Very limited scalabity What if DEV need custom modules (they will) Approach 2 : Pull request mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
  55. 55. Approach 1 : OPS write all modules  Impossible to scale  Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module  Very limited scalabity What if DEV need custom modules (they will) Approach 2 : Pull request Approach 3: DEV can push to some repositories mongodb module Version Control System CI Puppet master mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
  56. 56. Approach 1 : OPS write all modules  Impossible to scale  Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module  Very limited scalabity  Complex permissions  DEV are still basically root What if DEV need custom modules (they will) Approach 2 : Pull request Approach 3: DEV can push to some repositories mongodb module Version Control System CI Puppet master mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
  57. 57. Sure, we can tweak puppet
  58. 58. Sure, we can tweak puppet Is this this the way??
  59. 59. Sure, we can tweak puppet Maybe we should adapt processes and not just tools Is this this the way??
  60. 60. A NEW relationship between DEV & OPS
  61. 61. • Provide application • Ask for env From separation and control to shared responsiblities
  62. 62. Storage / Network Servers • Provide application • Ask for env • Provide env • Run production From separation and control to shared responsiblities
  63. 63. Strict separation of roles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production From separation and control to shared responsiblities
  64. 64. Strict separation of roles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production • Provide programmable resources • Provide advice • Delegate some Prod responsability Storage Servers Servers Network From separation and control to shared responsiblities
  65. 65. Strict separation of roles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production • Provide programmable resources • Provide advice • Delegate some Prod responsability Storage Servers Servers Network • Provide application • Consume environments • Share responsibility From separation and control to shared responsiblities
  66. 66. Strict separation of roles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production • Provide programmable resources • Provide advice • Delegate some Prod responsability Storage Servers Servers Network • Provide application • Consume environments • Share responsibility Shared responsibilities From separation and control to shared responsiblities
  67. 67. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources What it could look like with the profile/role pattern
  68. 68. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules OPS provide core OS modules What it could look like with the profile/role pattern
  69. 69. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules mysql apache OPS provide core OS modules OPS provide middleware modules What it could look like with the profile/role pattern
  70. 70. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules mysql apache OS BaseProfiles OPS provide core OS modules OPS provide middleware modules OPS provide Base profile What it could look like with the profile/role pattern
  71. 71. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules What it could look like with the profile/role pattern
  72. 72. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules What it could look like with the profile/role pattern
  73. 73. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress Roles Roles Wordpress-server OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules DEV & OPS define roles What it could look like with the profile/role pattern
  74. 74. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress Roles Roles Wordpress-server OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules DEV & OPS define roles DEV & OPS define variables Hiera What it could look like with the profile/role pattern
  75. 75. “Designing Puppet: Roles / Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress Roles Roles Wordpress-server OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules DEV & OPS define roles DEV & OPS associate roles to nodes DEV & OPS define variables Hiera Classifier What it could look like with the profile/role pattern
  76. 76. Final words
  77. 77. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Our feedback on puppet Puppet is an amazing tool
  78. 78. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Our feedback on puppet Puppet is an amazing tool
  79. 79. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? Our feedback on puppet Puppet is an amazing tool
  80. 80. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply Our feedback on puppet Puppet is an amazing tool
  81. 81. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? Our feedback on puppet Puppet is an amazing tool
  82. 82. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Our feedback on puppet Puppet is an amazing tool
  83. 83. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Our feedback on puppet Puppet is an amazing tool
  84. 84. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Our feedback on puppet Puppet is an amazing tool
  85. 85. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Too big! We are lost * Variables? * Classification * Module conflicts Our feedback on puppet Puppet is an amazing tool
  86. 86. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Too big! We are lost * Variables? * Classification * Module conflicts Best practices * Roles / Profiles * Variable location Our feedback on puppet Puppet is an amazing tool
  87. 87. • Automate configuration • Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Too big! We are lost * Variables? * Classification * Module conflicts Best practices * Roles / Profiles * Variable location Our feedback on puppet • Setups can be complex • Many solutions to a problem • Use it for what it does best Try adapting processes first • Look for best practices Puppet is an amazing tool You can do (almost) anything with puppet, but
  88. 88. The pace of innovation in IT is accelerating New time-to-market challenges will require continuous delivery We will not get continuous delivery without DEVOPS Puppet is an amazing DEVOPS tool and will help you Conclusion
  89. 89. The pace of innovation in IT is accelerating New time-to-market challenges will require continuous delivery We will not get continuous delivery without DEVOPS Puppet is an amazing DEVOPS tool and will help you Conclusion But tools cannot do everything: puppet is not a magic solution
  90. 90. The pace of innovation in IT is accelerating New time-to-market challenges will require continuous delivery We will not get continuous delivery without DEVOPS Puppet is an amazing DEVOPS tool and will help you Conclusion • Finding the best way to use puppet for you will take time • Providing a configuration service will be a challenge • Processes will need to change DEV and OPS roles are evolving and Organizations will need to adapt But tools cannot do everything: puppet is not a magic solution
  91. 91. Thank you Laurent Bernaille, D2SI @lbernail @D2SI #PuppetCampParis

×