Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev

When Ops Swallows Dev

Scott Prugh, Chief Architect & VP Software Development & Operations, CSG International
Erica Morrison, Director, Software Development, CSG International

CSG has been on an Agile and Lean journey to continually shorten feedback loops in its SDLC and Operations Processes. This began with moving from waterfall to agile and deploying cross functional dev teams. Today, we have taken this transformation further by deploying cross functional product delivery teams that Design, Build, Test and Run their products. Join us to discover the things that went as expected and the surprises we discovered in this journey.

DevOps Enterprise Summit San Francisco 2016

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

DOES16 San Francisco - Scott Prugh & Erica Morrison - When Ops Swallows Dev

  1. 1. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. When Ops Swallows Dev Our Cultural and Technical Journey to Realize DevOps at CSG Scott Prugh, Chief Architect & VP Software Development & Operations, scott.prugh@csgi.com, @ScottPrugh Erica Morrison, Director Software Development, erica.morrison@csgi.com, @ericarmorrison
  2. 2. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. When Ops Swallows Dev Techniques(v2) 1. Holistically Improve Work Visibility a) Incidents b) Dependencies c) Single Intake of Planned Work 2. Challenging Shared KPIs 3. Go See & Role Rotation 4. Infra as Code & Shared Understanding 5. Legacy Test Automation / ATDD 6. Telemetry & Shared Understanding Pressures & Constraints Demand for Quality & Speed SoRs being pressured to become SoEs Org & Process Debt, Technical Debt Strive for Unimodal IT Invest in: Culture, Empathy, Understanding, Simplicity & Automation Techniques(v1) 1. Accelerate Learning & Lean Thinking 2. Inverse Taylor Maneuver 3. Inverse Conway Maneuver 4. Shared Service Continuous Delivery 5. Environment Congruency & Practice 6. Application Telemetry 7. Visualize Your Work 8. Work Release & WIP Limits 9. Cadence & Synchronization 10. Reduce Batch Size Accelerate Feedback & Learning Understanding Accountability Engineering 2014 2015 2016 Journey and Principles 1. Org Archetype Journey 2012-2016  Creating Service Delivery Teams 2. What we learned…. 3. What is next… Scott Prugh, Chief Architect & VP Software Development & Operations, scott.prugh@csgi.com, @ScottPrugh Erica Morrison, Director Software Development, erica.morrison@csgi.com, @ericarmorrison Our Cultural and Technical Journey to Realize DevOps at CSG YouTube: f4et0EGvKXA YouTube: tKdIHCL0DUg
  3. 3. CSG International Confidential and Proprietary Information Copyright © 2015 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. CSG in NA: Who Are We? SaaS Based Customer Care and Billing • ~50M Subscribers • 100k Call Center Seats • ~6B External Transactions/month • 40 Dev. Teams & 1000 Practitioners • ACP: ~20 Technology Stacks: JS to HLASM • Integrated Suite of 50+ applications Challenges: • Time to Market & Quality/Release Impact • Technology Stovepipes • Role Stovepipes • Operational Quality Print and Mail Factory • ~70M statements/month • Lean & Efficient Challenges • Continuous Optimization
  4. 4. CSG International Confidential and Proprietary Information Copyright © 2015 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Context: Opposing Forces Development Operations Change Management Release Management Production Operations PMO Speed! Stability! Your code sucks!Your environments suck! Your SR tool sucks! Your Agile tool sucks! We hate CRQs! We hate, CRQs! © Alphaspirit @ 123RF.com , Wavebreakmediamicro @ 123RF.com, Warrengoldswain @ 123RF.com
  5. 5. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Courage and Feedback Automated Unit Tests Automated System Tests CI Build w/ Automated Tests Exploration Tests Manual regression testing Operational Quality/Feedback Analysis of production metrics(Telemetry) Customer Opinion Time DOES15 - Elisabeth Hendrickson - Its All About Feedback https://www.youtube.com/watch?v=r2BFTXBundQ Latency: How do we move from days to hours? Faster Loops means Faster Learning
  6. 6. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Functional Archetype: Role specific organizations “locally optimize” around their structure and role instead of globally optimizing the flow of business value. Customer Reqs. Design Dev. Testing Ops. Production Optimize Optimize Optimize Optimize Optimize Flow of Business Value Big product/feature a client wants Queues CSG Pre-2012 Role ShapeQueues WaitTime=(%Busy)/(%Idle) Queues delay feedback and learning Queues help enforce I-Shaped resources
  7. 7. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Agile Archetype: Cross functional, self-empowered agile teams optimize the SDLC value-stream and remove impediments to do so. Customer Ops. Production Optimize Flow of Business Value Big product/feature a client wants Agile DBT Team(7+/-2) Agile DBT Team(7+/-2) Agile DBT Team(7+/-2) CSG 2012-2015 Role Shape Batch Size Reduction Test Automation Cadence & Synchronization Shared Operations Teams Inverse Conway & Taylor Work Visibility Continuous Delivery V1 “little t”
  8. 8. CSG International Confidential and Proprietary Information Copyright © 2015 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Release Incidents ImpactScore I-ImpactScore 13.1(04-07) 201 4550%(1x) 14.1(01-12) 76 15666%(3x) 14.2(04-13) 38 8681%(5x) 14.3(07-27) 39 8382%(5x) 14.4(10-26) 47 11176%(4x) 15.1(02-08) 20 4091%(11x) 15.2(05-17) 33 8182%(6x) 15.3(08-09) 18 4391%(11x) 15.4(11-15) 26 6885%(7x) 16.1(03-06) 36 8282%(6x) 16.2(06-12) 28 7085%(7x) 16.3(09-25) 24 5089%(9x) Results: Release Quality Improvements CSG 2014-2016
  9. 9. CSG International Confidential and Proprietary Information Copyright © 2015 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. System View of Quality Facts 1. 98% of incidents outside release 2. 92% of incidents fixed by ops OperationsDevelopment Your code sucks! Works in test! Incidents Resolved By Group/Priority 12mo period ending 3/16 My code does suck
  10. 10. CSG International Confidential and Proprietary Information Copyright © 2015 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Before: Traditional Dev and Ops Structure Hypotheses • Different org goals were working against system goals • Lack of op understanding = hard to run software • Lack of shared mission = lack of empathy across teams • Handoffs caused elongated lead time • Lack of engineering skills in operations prevented improvements and encouraged duct-tape engineering Observations • Release quality optimized but system quality not • Manual processes around hard to run software • Dev lacks understanding around operations • Lack of feedback into development • Collaboration was unnatural and scripted • Lack of “Esprit De Corps” • Lots of context switching and work management chaos Market Archetype*: Service Delivery(DevOps) Teams
  11. 11. CSG International Confidential and Proprietary Information Copyright © 2015 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Before: Traditional Dev and Ops Structure After: Cross Functional DevOps Structure The future is here* IAC vs IAT Market Archetype*: Service Delivery(DevOps) Teams
  12. 12. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Market Archetype*: Team Level Architect Product Owner Prod Ops Developer QA SM Platform via Self Serve(TBD) Service Delivery Teams Build and Run their own software. This shortens feedback, increases learning and rapidly improves quality. Why Bring Dev and Ops Together? Understanding: Team now has the ability to understand, learn and improve the delivery chain. Accountability: Teams now accountable for the majority of the delivery chain: Design, Build, Test, Operations. Engineering: 1) Team can inject and cross train engineering principles/skills into operations, 2) Team can now evolve operations from being solely a process activity to an engineering activity with true continuous improvement. Shift left: no duct-tape engineering. Other Benefits: Communication, Meetings, Planning, Collaboration, Work Visibility, Shared Leadership Vision Role Shape “Big T”
  13. 13. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. What We Learned
  14. 14. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Going All In
  15. 15. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Title: “When a Dev Manager Takes Over NLB -> The Horrors and Realizations that Follow” Title: "Pizza, Beer, and Illegal Drugs - An Overview of Motivational Techniques in DevOps" Timing: “We should present something on NLB, but we would need to present at 1AM” A Whole New World
  16. 16. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. I’m In the Phoenix Project! Invisible work/Work in multiple systems Competing priorities from across the company WIP/Overutilization Manual configuration Technical debt Brent Poor visibility into our specific changes
  17. 17. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Applying DevOps • Made resource changes • Implemented automated reporting • Applied dev best practices • Implementing configuration as code • Completed work tracking system overhaul • Changed workload management • Increased change process visibility
  18. 18. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Ops is hard! Change process and volumes can be daunting Change can be scary Application architecture is needed Enablement is key Support is truly 24x7 Ops is forced to tolerate a lot of pain Ops from a Development Perspective
  19. 19. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Summary / What’s Next Accelerate Feedback & Learning: Understanding, Accountability, Engineering Challenge Org Norms(Courage) Structure for Feedback/Outcome  Speed, Quality Things Learned Work Chaos: Change and BAU Manual Processes Resource Bottlenecks Tech Debt Testing(lack of) Process Change: Lead Time, SDLC Rigor, Ownership Bridge ITIL/SDLC(Process & Tools) Impact Reduction Centers of Enablement(vs Excellence) Technology More engineering less duct-tape Mainframe ASM  Java: 30-1 reduction CD v2.0: From IAT to IAC/Cloud Pipeline People Engineering Culture Cross Skilling / Upskilling What’s Next….
  20. 20. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. Change Lead Time & Schedule Buffers

×