Dev "Programming" Ops For DevOps Success

1,004 views

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1mz2piq.

Damon Edwards explores the successful patterns - and damaging anti-patterns - observed at dozens of companies going through DevOps transformations. The main focus is on how Development teams can influence and take a leading role in the closing of the DevOps divide. Filmed at qconlondon.com.

Damon Edwards is the co-founder and managing partner of the DTO Solutions consulting group. Damon is also a frequent contributor to the Web Operations focused dev2ops.org blog, the co-host of the DevOps Cafe podcast series, and a co-author of the DevOps Cookbook from IT Revolution Press.

Published in: Technology, Business

Dev "Programming" Ops For DevOps Success

  1. 1. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /devops-patterns http://www.infoq.com/presentati ons/nasa-big-data http://www.infoq.com/presentati ons/nasa-big-data
  2. 2. Presented at QCon London www.qconlondon.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  3. 3. Dev "Programming" Ops for DevOps Success
  4. 4. Damon Edwards @damonedwards dev2ops.org DevOps Cafe
  5. 5. Disclosure: DevOps (to me)
  6. 6. Disclosure: DevOps (to me) DevOps is not •a specific methodology or prescriptive steps •only achievable by “one true way” •a job title or group name
  7. 7. Disclosure: DevOps (to me) DevOps is not •a specific methodology or prescriptive steps •only achievable by “one true way” •a job title or group name DevOps is •a way of seeing your problems •a way of evaluating solutions •a way of communicating these things •always evolving
  8. 8. Damon Edwards DevOps Consulting Automation Design Open Source Tools
  9. 9. What every business wants Time-to-market Quality
  10. 10. What every business wants Time-to-market Quality Effectiveness
  11. 11. What every business wants Time-to-market Quality Effectiveness Do more, but don’t spend any more!( )
  12. 12. What’s stopping them?
  13. 13. Photo credit: Doc Searls on Flickr
  14. 14. Photo credit: Doc Searls on Flickr OpsDev
  15. 15. Photo credit: Doc Searls on Flickr WallofConfusion OpsDev
  16. 16. Photo credit: Doc Searls on Flickr Ah-ha! OpsDev Wall of iononfusC Remove Shorten
  17. 17. Photo credit: Doc Searls on Flickr You are here DEV
  18. 18. “...but how can we start a DevOps transformation from our Dev silo?”
  19. 19. Dev initiated DevOps Transformation
  20. 20. Dev initiated DevOps Transformation 1. Take an “Operations First” mindset
  21. 21. Improved Time to Market & Quality Quicker Feedback from Users Business Idea Running Services Dev Ops DevOps and Agile Ops Security Auditing Compliance Availability Understand the pressure on Ops
  22. 22. What is the product?
  23. 23. What is the product? • Are customers paying for a running service?
  24. 24. What is the product? • Are customers paying for a running service? • Then the running service is the product
  25. 25. What is the product? • Are customers paying for a running service? • Then the running service is the product • Operations is your “factory floor” and “storefront”
  26. 26. What is the product? • Are customers paying for a running service? • Then the running service is the product • Operations is your “factory floor” and “storefront” • Everything else is a “parts supplier”
  27. 27. What is the product? • Are customers paying for a running service? • Then the running service is the product • Operations is your “factory floor” and “storefront” • Everything else is a “parts supplier” • (Yes, that includes Developers)
  28. 28. What is the product? • Are customers paying for a running service? • Then the running service is the product • Operations is your “factory floor” and “storefront” • Everything else is a “parts supplier” • (Yes, that includes Developers) • If the service isn’t running, there is no product or business
  29. 29. “-ilities” are product features
  30. 30. “-ilities” are product features • Stability, scalability, availability, security, etc... are all features
  31. 31. “-ilities” are product features • Stability, scalability, availability, security, etc... are all features • Like all features, you get what you invest
  32. 32. “-ilities” are product features • Stability, scalability, availability, security, etc... are all features • Like all features, you get what you invest CommanderWorkbench ControlTier software Commander Workbench ControlTier software Commander Workbench ControlTier software Backlog Funcational Requirements Operational Requirements • Operations Requirements should be first class citizens in your backlog
  33. 33. You are developing services (not software)
  34. 34. You are developing services (not software) • Software is a service when it is running and managed
  35. 35. You are developing services (not software) • Software is a service when it is running and managed • Deployment and configuration is automated
  36. 36. You are developing services (not software) • Software is a service when it is running and managed • Deployment and configuration is automated • Standard operating procedures are automated
  37. 37. You are developing services (not software) • Software is a service when it is running and managed • Deployment and configuration is automated • Standard operating procedures are automated • Tests/Health Checks/Monitoring at service level
  38. 38. You are developing services (not software) • Software is a service when it is running and managed • Deployment and configuration is automated • Standard operating procedures are automated • Tests/Health Checks/Monitoring at service level • You are developing a service... so this is part of your deliverable
  39. 39. Redefine “done”
  40. 40. Redefine “done” Old “Done”
  41. 41. Redefine “done” Old “Done” •Work is "done" when it moves downstream
  42. 42. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done"
  43. 43. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository
  44. 44. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete!
  45. 45. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete! o The work just begins for Operations to figure out how to run it in production
  46. 46. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete! o The work just begins for Operations to figure out how to run it in production New “Done”
  47. 47. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete! o The work just begins for Operations to figure out how to run it in production New “Done” •It's running
  48. 48. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete! o The work just begins for Operations to figure out how to run it in production New “Done” •It's running •It's managed
  49. 49. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete! o The work just begins for Operations to figure out how to run it in production New “Done” •It's running •It's managed •Customer accessible
  50. 50. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete! o The work just begins for Operations to figure out how to run it in production New “Done” •It's running •It's managed •Customer accessible •Behaving properly
  51. 51. Redefine “done” Old “Done” •Work is "done" when it moves downstream •No shared sense of "done" o Developers commit code to repository o Time for a party: code complete! o The work just begins for Operations to figure out how to run it in production New “Done” •It's running •It's managed •Customer accessible •Behaving properly Development of a feature can be “done”. But a service is never “done” until it is turned off!
  52. 52. If you want freedom, take responsibility
  53. 53. If you want freedom, take responsibility Old Way
  54. 54. If you want freedom, take responsibility Old Way • Developers owned feature requirements
  55. 55. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime
  56. 56. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality
  57. 57. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality • Security owned security
  58. 58. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality • Security owned security • etc...
  59. 59. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality • Security owned security • etc... New Way
  60. 60. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality • Security owned security • etc... New Way • Developers own their application
  61. 61. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality • Security owned security • etc... New Way • Developers own their application • Operations owns infrastructure and common tooling
  62. 62. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality • Security owned security • etc... New Way • Developers own their application • Operations owns infrastructure and common tooling • Everybody owns quality, availability, security for the thing they produce and shared responsibility for things that consume it
  63. 63. If you want freedom, take responsibility Old Way • Developers owned feature requirements • Operations owned performance and uptime • QA owned quality • Security owned security • etc... New Way • Developers own their application • Operations owns infrastructure and common tooling • Everybody owns quality, availability, security for the thing they produce and shared responsibility for things that consume it
  64. 64. Dev initiated DevOps Transformation 1. Take an “operations first” mindset 2. Build organizational alignment
  65. 65. What is organizational alignment?
  66. 66. What is organizational alignment? • 1000’s of small decisions made daily • How do we harness that?
  67. 67. What is organizational alignment? • 1000’s of small decisions made daily • How do we harness that? • Top down is brittle • Conditions, people, tools are always changing... how do we handle that?
  68. 68. What is organizational alignment? • 1000’s of small decisions made daily • How do we harness that? • Top down is brittle • Conditions, people, tools are always changing... how do we handle that? • Alignment is when you know that different individuals would independently see a set of conditions and arrive at the same decision that is correct for the company’s goal
  69. 69. What is organizational alignment? • 1000’s of small decisions made daily • How do we harness that? • Top down is brittle • Conditions, people, tools are always changing... how do we handle that? • Alignment is when you know that different individuals would independently see a set of conditions and arrive at the same decision that is correct for the company’s goal
  70. 70. What does an aligned organization “see”?
  71. 71. What does an aligned organization “see”? expanded from
  72. 72. What does an aligned organization “see”? 1. See the system expanded from
  73. 73. What does an aligned organization “see”? 1. See the system 2. Focus on flow expanded from
  74. 74. What does an aligned organization “see”? 1. See the system 2. Focus on flow 3. Recognize feedback loops expanded from
  75. 75. What does an aligned organization “see”? 1. See the system 2. Focus on flow 3. Recognize feedback loops 4. Look for continuous improvement opportunities expanded from
  76. 76. Sure... but how do you do that?
  77. 77. Building organizational alignment
  78. 78. Building organizational alignment 1. Socialize the concepts and vocabulary
  79. 79. Building organizational alignment 1. Socialize the concepts and vocabulary 2. Visualize the system
  80. 80. Building organizational alignment 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping
  81. 81. Building organizational alignment 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis
  82. 82. Building organizational alignment 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis
  83. 83. Value Stream Mapping Creative Business Release Engineering Development QA Technical Operations Program Management L/T Lead time P/T Process time H/C Head count S/R Scrap rate D Defects EP Extra processes M Motion PD Partially done TS Task switching W Waiting Core business service delivery process Current state value stream map Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Production Environment Merge project branch Deploy to Performance Web Assets, flows and wireframes Delta features and fixesSet of service branches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Develop web assets Bob McNulty L/T 6 weeks P/T H/C S/R 4 weeks 12 5% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy to Production Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Cut project branch Chris Holmes L/T 15 minutes P/T H/C S/R 5 minutes 1 0% Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Cannata L/T 9 hours P/T H/C S/R 8 hours 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Fernando Gomez L/T 30 minutes P/T H/C S/R 30 minutes 1 33% 4 weeks 2 weeks 4 weeks 4 weeks 4 hours 1.5 days 1 day 7 hours 5 hours 1 hours 4.5 days 2.5 weeks 1.5 weeks 90 minutes 5.75 days 3.5 days 1.5 days 8 hours 1 hour Process Time = 62 days Lead Time = 113 days D EP Dx2 EP Mx2 W D PD Dx3 Mx3 TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2
  84. 84. Value Stream Mapping Creative Business Release Engineering Development QA Technical Operations Program Management L/T Lead time P/T Process time H/C Head count S/R Scrap rate D Defects EP Extra processes M Motion PD Partially done TS Task switching W Waiting Core business service delivery process Current state value stream map Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Production Environment Merge project branch Deploy to Performance Web Assets, flows and wireframes Delta features and fixesSet of service branches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Develop web assets Bob McNulty L/T 6 weeks P/T H/C S/R 4 weeks 12 5% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy to Production Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Cut project branch Chris Holmes L/T 15 minutes P/T H/C S/R 5 minutes 1 0% Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Cannata L/T 9 hours P/T H/C S/R 8 hours 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Fernando Gomez L/T 30 minutes P/T H/C S/R 30 minutes 1 33% 4 weeks 2 weeks 4 weeks 4 weeks 4 hours 1.5 days 1 day 7 hours 5 hours 1 hours 4.5 days 2.5 weeks 1.5 weeks 90 minutes 5.75 days 3.5 days 1.5 days 8 hours 1 hour Process Time = 62 days Lead Time = 113 days D EP Dx2 EP Mx2 W D PD Dx3 Mx3 TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2
  85. 85. Value Stream Mapping Customer Shared Drive Test Shared Drive Prod Commits Rollout Schedule README MOP Release Schedule PRD PRD Release Memos Tasks QA Forum Ticket Remedy Ticket Estimates Patch Calendar QA forum MOP README ERR ERR MOP, SOP PRD Design Specs crit bugs email Lockdown control checklist New Targets Remedy Ticket Single Image Server XML BRD ERR BTS QA Environment Documentum Production Packages Customer communication L/T = 28d P/T = 7d H/C = 1 S/R = Stephen / Xi Product Program Planning L/T = 105d P/T = 46d H/C = 15 S/R = 100% John Robert Release Program Management L/T = P/T = H/C = S/R = Erica Smith Engineering Planning Process L/T = 45d P/T = 18d H/C = 23 S/R = Bob SmithPreliminary Development L/T = 45d P/T = 21d H/C = 140 S/R = Bob Smith Full Development L/T = 75d P/T = 43d H/C = 130 S/R = Bob Smith Build L/T = 1d P/T = 0.3d H/C = 2 S/R = 33% John Doe Selective Promotion L/T = 90d P/T = 15d H/C = 5 S/R = Steve Young QA Test L/T = 105d P/T = 11d H/C = 42 S/R = Sam Young Engineering Release L/T = 60d P/T = 1d H/C = 1 S/R = >5% Victoria Doe Release Promotion L/T = 60d P/T = 0.2d H/C = 1 S/R = >5% Victoria Doe Cloud Services Release L/T = 60d P/T = 16d H/C = 3 S/R = 3% Reggie / Carlos Change Control L/T = 42d P/T = H/C = S/R = Peter Lee Deploy Release L/T = 90d P/T = 8d H/C = 8 S/R = 2% Lewis S./Peter Y. Server Provisioning L/T = 24d P/T = 4d H/C = 3 S/R = 50% Jen Garza BRD Server Acceptance L/T = 14d P/T = 1d H/C = 4.5 S/R = 15% Lynn A. etc derived reqs. PRD QA Forum Ticket Service pack review L/T = 56d P/T = 7d H/C = 6 S/R = 100% Suresh Wu M PD(3) PD(3) EPM EP(2) PD M(3) W(2)M TS M(3) M(2) W(2) M(2) EP EP EP(3) W W EP W PD TS(2) M M M(2) M W(2) EP M D EPM(3) D W EP PD D(3) Current state value stream map L/T Lead time P/T Process time H/C Head count S/R Scrap rate D Defects EP Extra processes M Motion PD Partially done TS Task switching W Waiting Product Management Engineering Cloud Services Technical Support
  86. 86. Value Stream Mapping Customer Shared Drive Test Shared Drive Prod Commits Rollout Schedule README MOP Release Schedule PRD PRD Release Memos Tasks QA Forum Ticket Remedy Ticket Estimates Patch Calendar QA forum MOP README ERR ERR MOP, SOP PRD Design Specs crit bugs email Lockdown control checklist New Targets Remedy Ticket Single Image Server XML BRD ERR BTS QA Environment Documentum Production Packages Customer communication L/T = 28d P/T = 7d H/C = 1 S/R = Stephen / Xi Product Program Planning L/T = 105d P/T = 46d H/C = 15 S/R = 100% John Robert Release Program Management L/T = P/T = H/C = S/R = Erica Smith Engineering Planning Process L/T = 45d P/T = 18d H/C = 23 S/R = Bob SmithPreliminary Development L/T = 45d P/T = 21d H/C = 140 S/R = Bob Smith Full Development L/T = 75d P/T = 43d H/C = 130 S/R = Bob Smith Build L/T = 1d P/T = 0.3d H/C = 2 S/R = 33% John Doe Selective Promotion L/T = 90d P/T = 15d H/C = 5 S/R = Steve Young QA Test L/T = 105d P/T = 11d H/C = 42 S/R = Sam Young Engineering Release L/T = 60d P/T = 1d H/C = 1 S/R = >5% Victoria Doe Release Promotion L/T = 60d P/T = 0.2d H/C = 1 S/R = >5% Victoria Doe Cloud Services Release L/T = 60d P/T = 16d H/C = 3 S/R = 3% Reggie / Carlos Change Control L/T = 42d P/T = H/C = S/R = Peter Lee Deploy Release L/T = 90d P/T = 8d H/C = 8 S/R = 2% Lewis S./Peter Y. Server Provisioning L/T = 24d P/T = 4d H/C = 3 S/R = 50% Jen Garza BRD Server Acceptance L/T = 14d P/T = 1d H/C = 4.5 S/R = 15% Lynn A. etc derived reqs. PRD QA Forum Ticket Service pack review L/T = 56d P/T = 7d H/C = 6 S/R = 100% Suresh Wu M PD(3) PD(3) EPM EP(2) PD M(3) W(2)M TS M(3) M(2) W(2) M(2) EP EP EP(3) W W EP W PD TS(2) M M M(2) M W(2) EP M D EPM(3) D W EP PD D(3) Current state value stream map L/T Lead time P/T Process time H/C Head count S/R Scrap rate D Defects EP Extra processes M Motion PD Partially done TS Task switching W Waiting Product Management Engineering Cloud Services Technical Support
  87. 87. Timeline Analysis Creative Business Release Engineering Development QA Technical Operations Program Management L/T Lead time P/T Process time H/C Head count S/R Scrap rate D Defects EP Extra processes M Motion PD Partially done TS Task switching W Waiting Core business service delivery process Current state value stream map Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Production Environment Merge project branch Deploy to Performance Web Assets, flows and wireframes Delta features and fixesSet of service branches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Develop web assets Bob McNulty L/T 6 weeks P/T H/C S/R 4 weeks 12 5% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy to Production Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Cut project branch Chris Holmes L/T 15 minutes P/T H/C S/R 5 minutes 1 0% Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Cannata L/T 9 hours P/T H/C S/R 8 hours 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Fernando Gomez L/T 30 minutes P/T H/C S/R 30 minutes 1 33% 4 weeks 2 weeks 4 weeks 4 weeks 4 hours 1.5 days 1 day 7 hours 5 hours 1 hours 4.5 days 2.5 weeks 1.5 weeks 90 minutes 5.75 days 3.5 days 1.5 days 8 hours 1 hour Process Time = 62 days Lead Time = 113 days D EP Dx2 EP Mx2 W D PD Dx3 Mx3 TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2
  88. 88. Waste Analysis Creative Business Release Engineering Development QA Technical Operations Program Management L/T Lead time P/T Process time H/C Head count S/R Scrap rate D Defects EP Extra processes M Motion PD Partially done TS Task switching W Waiting Core business service delivery process Current state value stream map Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Production Environment Merge project branch Deploy to Performance Web Assets, flows and wireframes Delta features and fixesSet of service branches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Develop web assets Bob McNulty L/T 6 weeks P/T H/C S/R 4 weeks 12 5% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy to Production Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Cut project branch Chris Holmes L/T 15 minutes P/T H/C S/R 5 minutes 1 0% Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Cannata L/T 9 hours P/T H/C S/R 8 hours 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Fernando Gomez L/T 30 minutes P/T H/C S/R 30 minutes 1 33% 4 weeks 2 weeks 4 weeks 4 weeks 4 hours 1.5 days 1 day 7 hours 5 hours 1 hours 4.5 days 2.5 weeks 1.5 weeks 90 minutes 5.75 days 3.5 days 1.5 days 8 hours 1 hour Process Time = 62 days Lead Time = 113 days D EP Dx2 EP Mx2 W D PD Dx3 Mx3 TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2
  89. 89. Waste Analysis   (Mary Poppendick’s “Seven Wastes of Software Development”)
  90. 90. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for?
  91. 91. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time)
  92. 92. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues
  93. 93. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes
  94. 94. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer
  95. 95. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash”
  96. 96. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency
  97. 97. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration
  98. 98. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification
  99. 99. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification • Injecting requirements late in the process
  100. 100. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification • Injecting requirements late in the process • Fixing in place (vs rebuilding)
  101. 101. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification • Injecting requirements late in the process • Fixing in place (vs rebuilding) • “Snowflake” servers and environments
  102. 102. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification • Injecting requirements late in the process • Fixing in place (vs rebuilding) • “Snowflake” servers and environments • Manual dependency management
  103. 103. Core business service deliver Current state value stream m Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Productio Environme Deploy to Performance b Assets, ws and eframes Delta features and fixesSet of service ranches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy Product Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Can L/T 9 hou P/T H/C S/R 8 hou 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% D DD PD TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should we be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification • Injecting requirements late in the process • Fixing in place (vs rebuilding) • “Snowflake” servers and environments • Manual dependency management • Ad-hoc or unstructured artifact management
  104. 104. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis Building organizational alignment
  105. 105. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter Building organizational alignment
  106. 106. Are you getting better as an organization?
  107. 107. Are you getting better as an organization? • Cycle Time
  108. 108. Are you getting better as an organization? • Cycle Time • MTTD (Mean Time To Detect)
  109. 109. Are you getting better as an organization? • Cycle Time • MTTD (Mean Time To Detect) • MTTR (Mean Time to Repair)
  110. 110. Are you getting better as an organization? • Cycle Time • MTTD (Mean Time To Detect) • MTTR (Mean Time to Repair) • Quality at the Source (Scrap)
  111. 111. Metrics chains tie the individual to the goal What matters to the business Capability that influences what matters to the business Activity over which an individual can cause/influence outcomes
  112. 112. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter Building organizational alignment
  113. 113. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter 4. Identify projects / experiments against baseline Building organizational alignment
  114. 114. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter 4. Identify projects / experiments against baseline 5. Repeat steps 2 - 4 (continuous improvement program) Building organizational alignment
  115. 115. Start with a burst of energy = Principles = Analysis = Design Agenda Day 1 Day 2 Day 3 Day 4 Lunch Lunch Kickoff DevOps Goals Key Concepts Case Study Discussion Current State Analysis Current State Analysis Future State / Solution Design Future State / Solution Design Future State / Solution Design Wrap-up LunchLunch DevOps Workshop
  116. 116. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter 4. Identify projects / experiments against baseline 5. Repeat steps 2 - 4 (continuous improvement program) Building organizational alignment
  117. 117. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter 1. See the system 4. Identify projects / experiments against baseline 5. Repeat steps 2 - 4 (continuous improvement program) Building organizational alignment
  118. 118. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter 1. See the system 2. Focus on flow 4. Identify projects / experiments against baseline 5. Repeat steps 2 - 4 (continuous improvement program) Building organizational alignment
  119. 119. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter 1. See the system 2. Focus on flow 4. Identify projects / experiments against baseline 5. Repeat steps 2 - 4 (continuous improvement program) 3. Recognize feedback loops Building organizational alignment
  120. 120. 1. Socialize the concepts and vocabulary 2. Visualize the system ! a. value stream mapping ! b. timeline analysis ! c. waste analysis 3. Pick metrics that matter 1. See the system 2. Focus on flow 4. Identify projects / experiments against baseline 5. Repeat steps 2 - 4 (continuous improvement program) 3. Recognize feedback loops 4. Look for continuous improvement opportunities Building organizational alignment
  121. 121. Dev initiated DevOps Transformation 1. Take an “operations first” mindset 2. Build organizational alignment 3. Establish a new model for working with Ops
  122. 122. Example
  123. 123. What does Dev want?
  124. 124. What does Dev want? • What they need to get their job done
  125. 125. What does Dev want? • What they need to get their job done • When they need it
  126. 126. What does Dev want? • What they need to get their job done • When they need it • Fast feedback
  127. 127. What does Dev want? • What they need to get their job done • When they need it • Fast feedback • Dependable and predictable systems to integrate with
  128. 128. What does Dev want? • What they need to get their job done • When they need it • Fast feedback • Dependable and predictable systems to integrate with • Limit extraneous information or tasks
  129. 129. What does Dev want? • What they need to get their job done • When they need it • Fast feedback • Dependable and predictable systems to integrate with • Limit extraneous information or tasks • For everyone to get out of their way
  130. 130. What does Ops want?
  131. 131. What does Ops want? • Enough time to do their work • Deployment / provisioning • Stability and performance engineering • Hardening and security • Paying down technical debt • Compliance
  132. 132. What does Ops want? • Enough time to do their work • Deployment / provisioning • Stability and performance engineering • Hardening and security • Paying down technical debt • Compliance • To have their requirements considered earlier in the lifecycle
  133. 133. What does Ops want? • Enough time to do their work • Deployment / provisioning • Stability and performance engineering • Hardening and security • Paying down technical debt • Compliance • To have their requirements considered earlier in the lifecycle • Confidence that changes are not going to break the system or create a vulnerability
  134. 134. Dev and Ops interact through request queues
  135. 135. Dev and Ops interact through request queues X X Ticket System ?? X Leads to... • Bottlenecks • Increased lead times • Reinforces organizational silos • Misinterpretation or omissions
  136. 136. Replace request queues with self-service interfaces
  137. 137. Replace request queues with self-service interfaces • Fully automate what used to be done by humans
  138. 138. Replace request queues with self-service interfaces • Fully automate what used to be done by humans • Put behind self-service interfaces for on-demand consumption
  139. 139. Replace request queues with self-service interfaces • Fully automate what used to be done by humans • Put behind self-service interfaces for on-demand consumption • Benefits to Ops • Less time spent “doing”, more time adding value • Stop being the blocker
  140. 140. Replace request queues with self-service interfaces • Fully automate what used to be done by humans • Put behind self-service interfaces for on-demand consumption • Benefits to Ops • Less time spent “doing”, more time adding value • Stop being the blocker • Benefits to rest of organization • Decouple processes and avoid bottlenecks • Each team can move at their own pace • Cuts down on scrap and communication overhead • Enables a pull-based lifecycle
  141. 141. Service provider mindset is already familiar Customers Ops Web UI API
  142. 142. Customers Ops Web UI API Dev QA PM / Analysts Web UI / API Web UI / API Web UI / API Extend concept to internal interfaces as well
  143. 143. Why Ops will initially say no • Low confidence that new changes won’t break things • Governance / Compliance • Auditing • Access Control • Accounting
  144. 144. ...so lets show them what’s possible step-by-step
  145. 145. Fully automated, specification driven lifecycle
  146. 146. Example Shell Powershell Shell Powershell
  147. 147. Example Shell Powershell Shell Powershell Governance / Compliance
  148. 148. Mitigate quality and security risks • Repeatedly “rehearse” all operations from earliest possible point in lifecycle • Everybody should be deploying and testing with the same toolchain, automation, and tests • QA and InfoSec provide standards and tooling that are used by Dev upfront
  149. 149. DEV1 DEV2 DEV3 PERF/STAGE PROD ConfidenceBuild confidence from the start
  150. 150. Pull-based model to control promotions DEV1 DEV2 DEV3 PERF/STAGE PROD Confidence query promote publish 1 2 3
  151. 151. Don’t forget to give it a name Operations as a Service (OaaS) Ticketless IT
  152. 152. Don’t forget to give it a name Operations as a Service (OaaS) Ticketless IT More web / cloud friendly Bigger impact with traditional enterprise IT
  153. 153. Dev initiated DevOps Transformation 1. Take an “operations first” mindset 2. Build organizational alignment 3. Establish a new model for working with Ops
  154. 154. Damon Edwards @damonedwards dev2ops.org DevOps Cafe http://www.dtosolutions.com http://www.simplifyops.com
  155. 155. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/devops- patterns

×