Puppet Camp London April 2014: Increasing Agility by Understanding Risk

861 views
651 views

Published on

"Increasing Agility by Understanding
Risk" given at Puppet Camp London April 2014 by Simon Croome

Published in: Software, Technology, Business
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
861
On SlideShare
0
From Embeds
0
Number of Embeds
107
Actions
Shares
0
Downloads
18
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Puppet Camp London April 2014: Increasing Agility by Understanding Risk

  1. 1. Increasing Agility by Understanding Risk Simon Croome simon@croome.org
  2. 2. $ whoami Simon Croome Infrastructure Engineer Financial Industry Independent Consultant
  3. 3. Why Puppet?
  4. 4. Velocity
  5. 5. Configuration Management is only part of the solution
  6. 6. Software is deployable throughout its lifecycle
  7. 7. Software is deployable throughout its lifecycle ! Priority is keeping the software deployable over new features
  8. 8. Software is deployable throughout its lifecycle ! Priority is keeping the software deployable over new features ! Anybody can get fast, automated feedback on the production readiness of their systems any time a change is made
  9. 9. Software is deployable throughout its lifecycle ! Priority is keeping the software deployable over new features ! Anybody can get fast, automated feedback on the production readiness of their systems any time a change is made ! Push-button deployments of any version to any environment
  10. 10. Continuous Delivery
  11. 11. Continuous Delivery Reduces the transaction cost of making change
  12. 12. Continuous Delivery Reduces the transaction cost of making change Faster ROI
  13. 13. Continuous Delivery Reduces the transaction cost of making change Faster ROI Reduces Risk
  14. 14. Risk
  15. 15. Concept: John Allspaw http://blog.vincentbrouillet.com/costs-and-risks-benefits-of- continuous-delivery-in-one-picture/ !
  16. 16. If it hurts Do it more often
  17. 17. Controls
  18. 18. Controls Change Management
  19. 19. Controls Change Management Separation of Duties
  20. 20. Controls Change Management Separation of Duties Audit
  21. 21. An example workflow
  22. 22. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. Peer Review Code Format Tests Integr’tn Tests Unit Tests Code Release Once code is merged into Stash’s Release Branch, a UNIX team member may create a release containing one or more changes. Bamboo automates deployment of the release to UAT, further automated testing, then release to Production. Note that it will not yet be deployed to client systems. Create Release Deploy to UAT Integration Tests Deploy to PROD New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test failures block the release. Initiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk. Scheduling & Change Management Each impacted server role enters separate change management workflow Maintenance windows and change risk feed into scheduling. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Deployment Completed approvals trigger deployment during change window. . Approved CHG Tickets Apply Change Review Request Risk Maintenance Windows ServiceNow CHG Tickets Reporting Web console to track deployment of changes across the environment, including summary view of health by server role and business area. Un-approved tickets reset change workflow. Code Review UNIX team member creates “Pull Request”, indicating change is ready to be reviewed. Bamboo automates code quality review, unit and integration testing. If all tests pass, Engineering team alerted to perform peer review in Stash.
  23. 23. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test fail
  24. 24. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test failures block the release. Review Request Code Review UNIX team member change is ready to b Bamboo automates integration testing. If all tests pass, Eng peer review in Stash
  25. 25. Puppet Data Puppet Modules et. sing Stash Peer Review Code Format Tests Integr’tn Tests Unit Tests Code Release Once code is merged into member may create a rel Bamboo automates deplo automated testing, then r Note that it will not yet be Create Release Deploy UAT Test failures block the release. Review Request Code Review UNIX team member creates “Pull Request”, indicating change is ready to be reviewed. Bamboo automates code quality review, unit and integration testing. If all tests pass, Engineering team alerted to perform peer review in Stash.
  26. 26. Peer Review Code Release Once code is merged into Stash’s Release Branch, a UNIX team member may create a release containing one or more changes. Bamboo automates deployment of the release to UAT, further automated testing, then release to Production. Note that it will not yet be deployed to client systems. Create Release Deploy to UAT Integration Tests Deploy to PROD st”, indicating unit and to perform
  27. 27. Initiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk.
  28. 28. nitiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk. Scheduling & Ch Each impacted serv workflow Maintenance window DE Impacted Roles Reporting
  29. 29. nge ail ld tus Manual Review read-only mode. ack changes. t and assigns risk. Scheduling & Change Management Each impacted server role enters separate change management workflow Maintenance windows and change risk feed into scheduling. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Deployment Completed app window. . Approved CHG Ticke Risk Maintenance Windows ServiceNow CHG Tickets Reporting Un-approved tickets r change workflow.
  30. 30. Management nters separate change management hange risk feed into scheduling. T EBF PERF EBF BCP PROD Deployment Completed approvals trigger deployment during change window. . Approved CHG Tickets Apply Change Risk Maintenance Windows ServiceNow CHG Tickets Un-approved tickets reset change workflow.
  31. 31. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Approved CHG Tickets Risk Maintenance Windows ServiceNow CHG Tickets Reporting Web console to track deployment of changes across the environment, including summary view of health by server role and business area. Un-approved tickets reset change workflow.
  32. 32. Jira Ticket Engineering Puppet Data Code Branch Puppet Modules Test Servers Dev Puppet Engineering UNIX team member is assigned ticket. Jira creates development sandbox using Stash integration. Peer Review Code Format Tests Integr’tn Tests Unit Tests Code Release Once code is merged into Stash’s Release Branch, a UNIX team member may create a release containing one or more changes. Bamboo automates deployment of the release to UAT, further automated testing, then release to Production. Note that it will not yet be deployed to client systems. Create Release Deploy to UAT Integration Tests Deploy to PROD New Requirement Ticket is raised against UNIX Jira Issue tracker, e.g. New project, change to OS build. Test failures block the release. Initiate Deployment Release is selected by UNIX team member. Select Release Dry-run Change Detail Impacted Roles Build Status Non- Impacted Servers Impacted Servers Manual Review Impact Analysis Change is run across the environment in read-only mode. Servers that would be changed report back changes. UNIX Change Manager assesses impact and assigns risk. Scheduling & Change Management Each impacted server role enters separate change management workflow Maintenance windows and change risk feed into scheduling. DEV SIT DIT EBF PERF EBF Impacted Roles BCP PROD CAT Deployment Completed approvals trigger deployment during change window. . Approved CHG Tickets Apply Change Review Request Risk Maintenance Windows ServiceNow CHG Tickets Reporting Web console to track deployment of changes across the environment, including summary view of health by server role and business area. Un-approved tickets reset change workflow. Code Review UNIX team member creates “Pull Request”, indicating change is ready to be reviewed. Bamboo automates code quality review, unit and integration testing. If all tests pass, Engineering team alerted to perform peer review in Stash.
  33. 33. Technology Stack ! Puppet - Dynamic environments MCollective - Comms CouchDB - Reports PouchDB - Replication to browser AngularJS - Web interface Rails - API (to be replaced) Resque / Redis - Job scheduling !
  34. 34. github.com/croomes/gonzo ! Questions?

×