Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DOES14 - Scott Prugh - CSG - DevOps and Lean in Legacy Environments

2,150 views

Published on

10 Techniques for Flow & Continuous Delivery

Startups are continually evangelizing DevOps to be able to reduce risk, hasten feedback and deploy 1000’s of times a day. But what about the rest of the world that comes from Waterfall, Mainframes, Long Release Cycles and Risk Aversion? Learn how one company went from 480 day lead times and 6 month releases to 3 month releases with high levels of automation and increased quality across disparate legacy environments. We will discuss how Optimizing People & Organizations, Increasing the Rate of Learning, Deploying Innovative Tools and Lean System Thinking can help large scale enterprises increase throughput while decreasing cost and risk.

Published in: Software
  • Be the first to comment

DOES14 - Scott Prugh - CSG - DevOps and Lean in Legacy Environments

  1. 1. DevOps & Lean in Legacy Environments 10 Techniques for Flow & Continuous Delivery Scott Prugh Chief Architect & VP NA Software Development @ScottPrugh Scott.Prugh@csgi.com Copyright © 2012 CSG Systems International, Inc. and/or its 1 affiliates (“CSG International”). All rights reserved.
  2. 2. CSG in NA: Who Are We? Customer Care and Billing Operations • ~50M Subscribers across 120 customers • 100k Call Center Seats • ~6B External Transactions/month • 40 Dev. Teams & 1000 Practitioners • ACP: ~20 Technology Stacks: JS to HLASM • Integrated Suite of 50+ applications Challenges: • Time to Market & Quality/Release Impact • Technology Stovepipes • Role Stovepipes Print and Mail Factory • ~70M statements/month • Lean & Efficient Challenges • Continuous Optimization Copyright © 2012 CSG Systems International, Inc. and/or its 2 affiliates (“CSG International”). All rights reserved.
  3. 3. Release Improvements From Lean & DevOps Impact Score: Critical=4 High=3 Medium=2 Low=1 Reducing the batch size from 28 to 14 weeks resulted in a 66.6%(3x) improvement And value is delivered in half the time! Copyright © 2012 CSG Systems International, Inc. and/or its 3 affiliates (“CSG International”). All rights reserved.
  4. 4. More Practice & Smaller Batches “Same Code(with fixes)” deployed 2nd time… © 2014 CSG 4 Systems International, Inc. Confidential and Proprietary
  5. 5. Goal: Improve Quality & TTM But, how do we improve quality, reduce batch size & move towards flow when the system is inherently structured to prevent this? Techniques to Address 1. Accelerate Learning & Lean Thinking 2. Inverse Taylor Maneuver 3. Inverse Conway Maneuver 4. Shared Service Continuous Delivery 5. Environment Congruency & Practice 6. Application Telemetry 7. Visualize Your Work 8. Work Release & WIP Limits 9. Cadence & Synchronization 10. Reduce Batch Size System Constraints 1. Structure 2. Stovepipes & Handoffs 3. Technology Variance 4. Defects & Quality 5. Low Automation 6. Fragility 3 year journey! © 2014 CSG 5 Systems International, Inc. Confidential and Proprietary
  6. 6. 1. Accelerate Learning & Lean Thinking People First: Build a culture of Learning and Self Improvement by embracing Lean & Systems Thinking. Find a Lean Framework Encourage Cross Training T-Shaped I-Shaped E-Shaped © 2014 CSG 6 Systems International, Inc. Confidential and Proprietary
  7. 7. 2. Inverse Taylor Maneuver Structure & responsibility will enforce behavior and prevents learning. Architect Developer Product Owner Developer Developer QA QA Ops Traditional Agile Optimize Optimize Optimize Analysis Architecture Developer QA Handoff Handoff Handoff Which structure incents learning and removes queues? SM Organize teams to optimize the entire flow of value. QA Optimize © 2014 CSG 7 Systems International, Inc. Confidential and Proprietary
  8. 8. 3. Inverse Conway Maneuver Structure will enforce technology and architecture. Technology: Fat CS Desktop Technology: Standard SOA Architecture Technology: Legacy Tuxedo MW Technology: Standard SOA Architecture Change the structure to help create the architecture you want. © 2014 CSG 8 Systems International, Inc. Confidential and Proprietary
  9. 9. 4. Shared Service Continuous Delivery Pave one automated high-way to production so all teams can get there with speed and predictability. Having many automated roads to production is better than no road or “dirt roads”. But having one road encourages reuse at scale and enforces consistency by buffering downstream teams from variance. © 2014 CSG 9 Systems International, Inc. Confidential and Proprietary
  10. 10. 5a. Environment Congruency & Practice Separate deployment and operations processes via infrequent handoffs create large batch transfers and high-failure rates. Ops. Optimize Agile DBT Team(7+/-2) Agile DBT Team(7+/-2) Agile DBT Team(7+/-2) Deploy Everyday “The Practice Team” Deploy 4x/year “The Game Team” Handoff 4x/year Dev./Test Environment Production Environment Do you play the “Game” with a different team than you “Practice” with? © 2014 CSG 10 Systems International, Inc. Confidential and Proprietary
  11. 11. 5b. Environment Congruency & Practice Make environments as close to production as possible and have the same team practice daily. Shared Operations Team(7+/-2) Agile DBT Team(7+/-2) Agile DBT Team(7+/-2) Agile DBT Team(7+/-2) Dev./Test Environment Production Environment Deploy Everyday “The Game Team Practices” Deploy 4x/year “Practice Made Perfect” Continuous Delivery We let our teams practice at least 70 times before “Game Day”…. © 2014 CSG 11 Systems International, Inc. Confidential and Proprietary
  12. 12. 6. Application Telemetry Build or embed deep telemetry into your application. Telemetry allows the teams to learn and greatly reduces TTR when there are issues. NASA doesn’t launch a rocket without millions of automated sensors reporting the status of this valuable asset. Why don’t we take the same care with software? Process Process Realtime Telemetry Trace/Activity Collection Queues StatHub Repository Analytics Reporting © 2014 CSG 12 Systems International, Inc. Confidential and Proprietary
  13. 13. 7. Make All Work Visible Haphazard and chaotic work intake processes invite context switching and high WIP(Work in Process) levels. Worker1 Worker2 Worker3 Dependency Worker4 Why am I waiting? Why am I waiting? © 2014 CSG 13 Systems International, Inc. Confidential and Proprietary
  14. 14. 8. Work Release & WIP Limits An intake buffer and process can take many forms but allows a quick triage, unified visibility, and WIP management. Worker1 WIP Limit Worker2 Work Work Work Work Work Work Worker3 Worker4 I’m ready! I’m ready One List of Work! © 2014 CSG 14 Systems International, Inc. Confidential and Proprietary
  15. 15. Print Center: Work Visibility Example Treat virtualized work with the same care that manufacturing treats physical work. Do you know how your work comes in and is scheduled? © 2014 CSG 15 Systems International, Inc. Confidential and Proprietary
  16. 16. Print Center: Continuous Delivery with Robots How do you get your code in production? © 2014 CSG 16 Systems International, Inc. Confidential and Proprietary
  17. 17. 9. Cadence & Synchronization If you don’t force cadence and synchronization in your processes and resources, nature will do it for you. Program Increment Program Increment Iterate, Synchronize, Release Iterate, Synchronize Release Manage new work injection! Make unpredictable events like release planning predictable across teams. Synchronize start and finish times to create a pull effect and prevent hidden work from creating problems. © 2014 CSG 17 Systems International, Inc. Confidential and Proprietary
  18. 18. 10. Reduce Batch Size Once you have Learning, Infrastructure & Process in place: Reduce Batch Size PI/Release PI/Release 28 Weeks/14 Iterations 28 Weeks/14 Iterations PI/Release PI/Release PI/Release PI/Release 14 Weeks 14 Weeks 14 Weeks 14 Weeks Smaller/fewer things go through the system faster with less impact. © 2014 CSG 18 Systems International, Inc. Confidential and Proprietary
  19. 19. More Practice & Smaller Batches Where we were. Where we are. Where we want to be. © 2014 CSG 19 Systems International, Inc. Confidential and Proprietary
  20. 20. 10 Techniques to Take Home 1. Accelerate Learning & Lean Thinking 2. Inverse Taylor Maneuver 3. Inverse Conway Maneuver 4. Shared Service Continuous Delivery 5. Environment Congruency & Practice 6. Application Telemetry 7. Visualize Your Work 8. Work Release & WIP Limits 9. Cadence & Synchronization 10. Reduce Batch Size © 2014 CSG 20 Systems International, Inc. Confidential and Proprietary
  21. 21. Credits  The Phoenix Project, Gene Kim, Kevin Behr, George Spofford  Continuous Delivery, Jez Humble  Scaled Agile Framework: www.scaledagileframework.com  The Fifth Discipline, Peter Senge  Martin Fowler, Technology Radar(Inverse Conway Maneuver)  Cultural Cartography(I,T,E Shaped Resources): http://culturecartography.wordpress.com/2012/07/26/busines s-trend-e-shaped-people-not-t-shaped/  Kevin Behr, Seven Traits Similar to DevOps: http://www.kevinbehr.com/kevins-blog © 2014 CSG 21 Systems International, Inc. Confidential and Proprietary
  22. 22. How can you help me?  Problem Areas • Standardizing applications at scale - Proving the business case/savings - Making progress quickly…. • Balancing standards with innovation © 2014 CSG 22 Systems International, Inc. Confidential and Proprietary

×