Downloaded 75 times
Uploaded byPuppet
Puppet Camp Paris 2014: Achieving Continuous Delivery and DevOps with Puppet
This document discusses achieving continuous delivery with Puppet. It notes that currently, development cycles are long, integration is painful, and deployments are difficult. It proposes that continuous integration, continuous delivery, DevOps practices, and an agile infrastructure using automation can help address these issues. Puppet is presented as a tool that can be used to help achieve an agile infrastructure and automate application deployments, though some challenges with its use are also discussed. The document advocates for changing the relationship between development and operations teams to one of more shared responsibility.
More Related Content
Similar to Puppet Camp Paris 2014: Achieving Continuous Delivery and DevOps with Puppet
Puppet Camp Paris 2014: Achieving Continuous Delivery and DevOps with Puppet
- 1. Achieving Continuous Deliverywith puppet Laurent Bernaille, D2SI #PuppetCampParis @D2SI@D2SI #PuppetCampParis
- 2. Past (current?) situation Whydo we need continuous delivery ?
- 3. Applications do notanswer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly June
- 4. Applications do notanswer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly June What the end user say he needs in January January
- 5. What the devteam delivers in June June Applications do not answer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly What the end user say he needs in January January
- 6. What the devteam delivers in June June What the end user needs in June Applications do not answer business needs well Long development cycle Difficult to get clear specifications Market can shift quickly June What the end user say he needs in January January
- 7. Development processes areinefficient Bug are detected too late Code TESTOther work Fix
- 8. Development processes areinefficient Bug are detected too late Integration Hell Code TESTOther work Fix Start integration
- 9. Development processes areinefficient Bug are detected too late Integration Hell A lot of wasted time Code TESTOther work Fix Request Approve & Prioritize Technical Assessment Code & Test Verify & Fix Deploy 20 min 2 min 15 min 2 h 4 h 3 min ½ week 2 weeks 2 weeks 1week ½ week Processing Time = 6 h 40 min Waiting Time = 6 weeks Adapted from Implementing Lean Software Development: From Concept to Cash, Mary & Tom Poppendieck. Start integration
- 10. OPS view Relationship withOPS can be “difficult” Application deployment is a nightmare
- 11. OPS view Relationship withOPS can be “difficult” Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare
- 12. OPS view ? Relationship withOPS can be “difficult” Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers»
- 13. OPS view ? Relationship withOPS can be “difficult” DEV view Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers»
- 14. OPS view ? Relationship withOPS can be “difficult” DEV view Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers» Identical servers are always “slightly” different
- 15. OPS view ? Relationship withOPS can be “difficult” DEV view Performance is not only related to hardware « Make my website faster in Asia» Application deployment is a nightmare « Our application is too slow because of your servers» Identical servers are always “slightly” different OPS always say “no” Standards do not evolve 10 deploys per day, Dev & ops cooperation at Flickr John Allspaw & Paul Hammond (Velocity 2009)
- 16. 6+ months to setupa new environment Infrastructure is not very agile Server “hoarding”
- 17. Infrastructure is notvery agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
- 18. Infrastructure is notvery agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
- 19. Production Infra setup Deploy DeployDeploy Deploy Utilization : 100% Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
- 20. Production Infra setup Deploy DeployDeploy Deploy Utilization : 100% Preproduction Utilization : 10% Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
- 21. Production Infra setup Deploy DeployDeploy Deploy Utilization : 100% Preproduction Utilization : 10% Test Utilization : 40% Infrastructure is not very agile Server “hoarding” Resources are heavily shared Most environments are underutilized 6+ months to setup a new environment
- 22. Business Development Operations • Applications donot answer business needs well • Too long to get new features • Integration and bug fixing is painful • A lot of wasted time • Deployments are very painful • A lot of misunderstanding • Environment setup is too slow • No on-demand resources Summary of the issues WHAT?
- 23. IT should beeasier Continuous Delivery
- 24. Agile Manifesto, 2001 Individualsand interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan Agile Development Business Development Operations A G I L E
- 25. Test Discover Design Develop Waterfall method Developing Incrementallyand Iteratively Long cycles High-risk Adaptation is very difficult 6-12 month
- 26. 2-4 weeks Test Discover Design Develop Discover Design Develop Test Discover Design Develop Test Discover Design Develop Test Waterfall method DevelopingIncrementally and Iteratively Agile method Long cycles High-risk Adaptation is very difficult 6-12 month Low-risk Many short cycles Adaptation is much easier VS
- 27. Maintain a singlesource repository Automate the build Make your build self-testing Every commit should build on an integration machine Keep the build fast Test in a clone of the production environment Make it easy for anyone to get the latest executable Everyone can see what’s happening Detect problems early and solve them quickly Continuous Integration C I Business Development Operations
- 28.
- 29. DevOps : bringthe wall down Business Development Operations D E V O P S Measure, Analyze & Describe Constraints (from DEV and OPS) Best practices, methods Automation Measure Share Culture Align objectives on business needs Innovate Automate application delivery
- 30.
- 31. Agile infra Control resources Configure resources Deploy applications Create, Delete,start, stop Servers (physical, virtual, in the cloud) Storage volumes, networks Define system states, when possible Verify system states Reconfigure systems when necessary Provide a service to deploy applications Automated, with rollbacks Agile Infrastructure Business Development Operations
- 32. Agile infra Control resources Configure resources Deploy applications Create, Delete,start, stop Servers (physical, virtual, in the cloud) Storage volumes, networks Define system states, when possible Verify system states Reconfigure systems when necessary Provide a service to deploy applications Automated, with rollbacks Automation API API API Agile Infrastructure Business Development Operations
- 33.
- 34.
- 35.
- 36. Puppet Master • Resolvers,time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes Server team uses puppet as a configuration tool
- 37. Puppet Master • Resolvers,time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes 4: apply/test catalog Puppet agentPuppet agentPuppet agent Server team uses puppet as a configuration tool
- 38. Puppet Master • Resolvers,time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes 4: apply/test catalog McollectiveGet info on nodes Run agent on subset of nodes Puppet agentPuppet agentPuppet agent Server team uses puppet as a configuration tool
- 39. Puppet Master • Resolvers,time servers, standard packages • Authentication, security • Monitoring,… Use case 1 : core OS configuration Hiera Modules Console ENC / API 1:Create modules Define variables Assign classes to nodes 4: apply/test catalog McollectiveGet info on nodes Run agent on subset of nodes Puppet agentPuppet agentPuppet agent Most common puppet usage. Proven for large scale deployment. Very “Infra oriented”: not opened to applications. Server team uses puppet as a configuration tool
- 40. Binaries of theapplication • Puppet manifests and modules describing deployments Repository Binaries Manifest Modules 1: put Use case 2 : deploy applications Developers supply :
- 41. Binaries of theapplication • Puppet manifests and modules describing deployments Dev servers Test servers Dev serversOther env Repository Binaries Manifest Modules 1: put 2: get 4: run Use case 2 : deploy applications • Chooses servers (env) where the deployment should happen • Runs puppet and gathers reports; if run fails, forwards to DEV Developers supply : OPS team :
- 42. Binaries of theapplication • Puppet manifests and modules describing deployments Dev servers Test servers Dev serversOther env Repository Binaries Manifest Modules 1: put 2: get 4: run Use case 2 : deploy applications • Chooses servers (env) where the deployment should happen • Runs puppet and gathers reports; if run fails, forwards to DEV Much more efficient than written deployment processes. Much easier to understand what fails. Developers supply : OPS team :
- 43. Configuration as aservice ProfilsLevels ConfigurationReport Tools
- 44. Using a “configurationservice” Env Configuration service Deploy API GUI • Associate “profiles” to nodes, define variables • Run configuration and get reports Give application teams the possibility to configure servers1
- 45. Using a “configurationservice” Env Configuration service Deploy API GUI • Associate “profiles” to nodes, define variables • Run configuration and get reports Give application teams the possibility to configure servers1 Different levels of configuration, different responsibilities2 Base OS configuration Standard middlewares Application middlewares Application
- 46. Can we dothat with puppet? Sure, but tricky with classic DEV / OPS model
- 47. Can we dothat with puppet? Sure, but tricky with classic DEV / OPS model DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
- 48. Base OS configuration Standardmiddlewares Application middlewares Application2 1 Can we do that with puppet? Some options : Sure, but tricky with classic DEV / OPS model • Second puppet master, or puppet apply (non root) • Other tool Tool separation1 DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
- 49. Base OS configuration Standardmiddlewares Application middlewares Application2 1 Can we do that with puppet? Some options : Sure, but tricky with classic DEV / OPS model • Second puppet master, or puppet apply (non root) • Other tool Tool separation1 Ok to run as root but under full control2 • Custom “profile” facts (facts.d) and hiera • Run with mcollective (limit to some tags) • Read-only console access DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
- 50. Base OS configuration Standardmiddlewares Application middlewares Application2 1 Can we do that with puppet? Some options : Sure, but tricky with classic DEV / OPS model • Second puppet master, or puppet apply (non root) • Other tool Tool separation1 Ok to run as root but under full control2 • Custom “profile” facts (facts.d) and hiera • Run with mcollective (limit to some tags) • Read-only console access Many other ways3 DEV cannot execute anything as root SURE, but tricky with classic DEV / OPS model
- 51. Approach 1 :OPS write all modules Version Control System I need a mongodb module CI Puppet master mongodb module What if DEV need custom modules (they will)
- 52. Approach 1 :OPS write all modules Impossible to scale Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module What if DEV need custom modules (they will)
- 53. Approach 1 :OPS write all modules Impossible to scale Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module What if DEV need custom modules (they will) Approach 2 : Pull request mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
- 54. Approach 1 :OPS write all modules Impossible to scale Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module Very limited scalabity What if DEV need custom modules (they will) Approach 2 : Pull request mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
- 55. Approach 1 :OPS write all modules Impossible to scale Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module Very limited scalabity What if DEV need custom modules (they will) Approach 2 : Pull request Approach 3: DEV can push to some repositories mongodb module Version Control System CI Puppet master mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
- 56. Approach 1 :OPS write all modules Impossible to scale Not efficient Version Control System I need a mongodb module CI Puppet master mongodb module Very limited scalabity Complex permissions DEV are still basically root What if DEV need custom modules (they will) Approach 2 : Pull request Approach 3: DEV can push to some repositories mongodb module Version Control System CI Puppet master mongodb module Version Control System CI Puppet master Pull request Version Control System validated module
- 57. Sure, we cantweak puppet
- 58. Sure, we cantweak puppet Is this this the way??
- 59. Sure, we cantweak puppet Maybe we should adapt processes and not just tools Is this this the way??
- 60. A NEW relationshipbetween DEV & OPS
- 61. • Provide application •Ask for env From separation and control to shared responsiblities
- 62. Storage / Network Servers •Provide application • Ask for env • Provide env • Run production From separation and control to shared responsiblities
- 63. Strict separation ofroles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production From separation and control to shared responsiblities
- 64. Strict separation ofroles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production • Provide programmable resources • Provide advice • Delegate some Prod responsability Storage Servers Servers Network From separation and control to shared responsiblities
- 65. Strict separation ofroles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production • Provide programmable resources • Provide advice • Delegate some Prod responsability Storage Servers Servers Network • Provide application • Consume environments • Share responsibility From separation and control to shared responsiblities
- 66. Strict separation ofroles Storage / Network Servers • Provide application • Ask for env • Provide env • Run production • Provide programmable resources • Provide advice • Delegate some Prod responsability Storage Servers Servers Network • Provide application • Consume environments • Share responsibility Shared responsibilities From separation and control to shared responsiblities
- 67. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources What it could look like with the profile/role pattern
- 68. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules OPS provide core OS modules What it could look like with the profile/role pattern
- 69. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules mysql apache OPS provide core OS modules OPS provide middleware modules What it could look like with the profile/role pattern
- 70. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules mysql apache OS BaseProfiles OPS provide core OS modules OPS provide middleware modules OPS provide Base profile What it could look like with the profile/role pattern
- 71. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules What it could look like with the profile/role pattern
- 72. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules What it could look like with the profile/role pattern
- 73. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress Roles Roles Wordpress-server OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules DEV & OPS define roles What it could look like with the profile/role pattern
- 74. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress Roles Roles Wordpress-server OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules DEV & OPS define roles DEV & OPS define variables Hiera What it could look like with the profile/role pattern
- 75. “Designing Puppet: Roles/ Profiles Design Pattern Puppet Camp Stockholm, Feb 2013 (Craig Dunn Puppet Labs) Resources word press ssh ntp dns ldap Modules mysql apache OS BaseProfiles Wordpress Roles Roles Wordpress-server OPS provide core OS modules OPS provide middleware modules OPS provide Base profile DEV create profiles using modules DEV create some custom modules DEV & OPS define roles DEV & OPS associate roles to nodes DEV & OPS define variables Hiera Classifier What it could look like with the profile/role pattern
- 76.
- 77. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Our feedback on puppet Puppet is an amazing tool
- 78. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Our feedback on puppet Puppet is an amazing tool
- 79. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? Our feedback on puppet Puppet is an amazing tool
- 80. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply Our feedback on puppet Puppet is an amazing tool
- 81. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? Our feedback on puppet Puppet is an amazing tool
- 82. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Our feedback on puppet Puppet is an amazing tool
- 83. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Our feedback on puppet Puppet is an amazing tool
- 84. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Our feedback on puppet Puppet is an amazing tool
- 85. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Too big! We are lost * Variables? * Classification * Module conflicts Our feedback on puppet Puppet is an amazing tool
- 86. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Too big! We are lost * Variables? * Classification * Module conflicts Best practices * Roles / Profiles * Variable location Our feedback on puppet Puppet is an amazing tool
- 87. • Automate configuration •Declare state, keep configuration on track • Puppet syntax is very expressive • Variable management with hiera is very efficient Time Expectations Puppet??? OK, looks interesting First puppet apply What the hell are: * Modules (and classes) * Hiera * erb * spaceships?? First modules Wow this is big Ok not that simple Too big! We are lost * Variables? * Classification * Module conflicts Best practices * Roles / Profiles * Variable location Our feedback on puppet • Setups can be complex • Many solutions to a problem • Use it for what it does best Try adapting processes first • Look for best practices Puppet is an amazing tool You can do (almost) anything with puppet, but
- 88. The pace ofinnovation in IT is accelerating New time-to-market challenges will require continuous delivery We will not get continuous delivery without DEVOPS Puppet is an amazing DEVOPS tool and will help you Conclusion
- 89. The pace ofinnovation in IT is accelerating New time-to-market challenges will require continuous delivery We will not get continuous delivery without DEVOPS Puppet is an amazing DEVOPS tool and will help you Conclusion But tools cannot do everything: puppet is not a magic solution
- 90. The pace ofinnovation in IT is accelerating New time-to-market challenges will require continuous delivery We will not get continuous delivery without DEVOPS Puppet is an amazing DEVOPS tool and will help you Conclusion • Finding the best way to use puppet for you will take time • Providing a configuration service will be a challenge • Processes will need to change DEV and OPS roles are evolving and Organizations will need to adapt But tools cannot do everything: puppet is not a magic solution
- 91. Thank you Laurent Bernaille,D2SI @lbernail @D2SI #PuppetCampParis