ESEconf2011 - Guckenheimer Sam: "Agile in the Very Large"
1. Impossibly X Agile in the Very LargeExperiences from Microsoft Developer Division Originally presented at Agile 2009 Sam Guckenheimer samgu@microsoft.com
6. How Large is Very Large? 3586 Active “devs” last 7 days 2008 Builds per month 716,858 Work items Source code files 23,681,882 15 Terabytes of data Stats as of July 2009 from Divisional TFS Server…one divisional server instance out of 42 in company
9. DevDiv: Servant of Three Masters Creates conflicting priorities among stakeholders Should force a discussion of single backlog At divisional level, reconciled largely through resourcing 7
10. 2005– Starting Culture Conway’s Law The Best and Brightest Autonomy, Job rotation, promotion The Currency of Love Headcount Dysfunctional Tribalism Don’t ask, don’t tell Schedule chicken Metrics are for others Our tribe is better Our customers are different Waste Easy Credit No interest charge for debt 8
11.
12. Background: Peanut Butter a.k.a. Overproduction Next Version Current Version Goodness Features Can’t message Intermittent satisfaction, no excitement No clear priorities for scoping mean risk accumulates Don’t know when we’re done
14. Product Cycle pre 2005 3-4 Years to Build It M0: Plan and cost the release M1…M3: Develop the code M3.1…M3.3 Recode Beta1: Integrate and pray Beta2: Test like hell RC (release candidate) 0..n: Final builds RTM: Ship it! Service It (10 years): QFEs and hot fixes Service packs
17. Experience ReportGet Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results Observations Lessons Applied to VS 2010 Discussion 14 Scheduling Practice
18. Moving from 2005 to 2008 Release Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results 15
19. Changes to Pre- and Post- Game Build it (~2 years): Planning: Start and groom product backlog MQ: Improve our practices, get ready for the next version M0: What are our goals for this release? What customer value do we deliver? M1…M35-week sprints: Develop and test the code. CTP (customer technical preview): Targeted customer release to collect feedback on mainline scenarios Beta1: First broad customer visibility; collect feedback Beta2: Validate recent changes with customers; collect feedback RC (release candidate) 0..n: Final builds RTM: Ship it! Service Release (10 years) Quarterly ±: Powertools and Feature Packs for current release Business Transformation from Packaged Product to Subscription
20. Focus of MQ (Milestone for Quality) Get Clean Stay Clean Understand our Debt Test Automation Product Code Bugs Test Case Bugs Test Infrastructure Bugs Put our Debt on the Table Cost our Debt Eliminate the Debt Measure our Quality Quickly & Consistently Anytime, Anywhere Quality earlier in the cycle Make life easier 17
21. Common Practices Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results 18
27. Customer Rating of Value Props “Invest less” ranking by audience “Invest more” ranking by audience Value Props within scenario
28. Common Practices Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results 25
29. Quality Gates & Frequency Main Nightlies – CTP, TP, Beta, RTM Full Auto – TP, Beta, RTM, (weekly coverage runs) PU Branches Nightlies – nightly Full Auto – prior to RI to Main App Weeks, Exploratory, Integration RPS (Perf & Stress) etc. Feature Branches New feature tests – as tests developed Directed Exploratory Rolling Testing etc.
31. Quality Gates Applied to Features Example ROTR Architectural Review Ensure that redist changes and additions do not compromise architectural consistency of the platform Advance the platform Minimize the barriers to upgrade Do not create architectural legacy we will later regret Assemblies that we would otherwise not have Performance issues Assemblies and features we will be “stuck” with List of Gates Functional Specification Dev Design Test Plan Threat Model IP Protection Code Coverage Static Analysis Testing Pseudo Loc Defects DDBasics Performance RPS API Review WinFx Architectural Layering ROTR Architectural Review Feature Complete Compatiblity
32. Common Practices Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results 29
34. Scrum Team (Feature Crew) Delivers ≥1 Features Team responsible for producing a feature Work together as a team to hit Feature Complete Small Interdisciplinary Team PM, Dev, Test Other disciplines as appropriate Typical team 1 Program Manager 4 or fewer Developers 4 or fewer Testers 3-6 weeks in duration 31
35. Considerations When Sizing Work When sizing a feature for development Size the time to get to Feature Complete Not just time to get to Code Complete Remember quality work happens up front Before Feature Complete Not in the bug tail No credit if you just get to “code complete”. Must get to Done. Developers write automated Unit Tests Testers write automated feature level tests Can write scenario level tests afterwards Threat Analysis Performed Don’t defer to a Security Push Feature level testing performed Most bugs found and fixed 32
37. A More Iterative Release Shape Full resourcing as prior release ships, attempt to groom and scope full product backlog Limited resource start, limiting planning to 1-2 sprints ahead
38. Common Practices Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog & Tooling Engineering Principles Measurement and Hardening Results 35
43. Drilldown: Value Prop -> Feature Status review rolls up features and experiences back into value props to assess release readiness Drilldown from value prop shows detailed feature status
47. Common Practices Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results 44
48. Engineering Principles Drive quality upstream Increase efficiency by doing things right the first time Don’t defer work Bug tail not filled with hidden work Keep main branch customer ready Enables consistent delivery of high quality CTPs every four weeks Product not tattooed with incomplete or unstable code Ready to ship when the time is right Use Feature Branches to isolate new feature development One branch per feature Only RI feature branch into PU branch once when quality gates are met Feature Branch PU Staging branch Main Common definition of what done means Consistent definition of quality across the division Enables meaningful metrics for progress and decisions Automate testing Automated testing at multiple levels Tests run anywhere
50. Integration Cadence into Main 47 CTP CTP Self Test Drive Self Test Drive Self Test Drive Self Test Drive Integration Integration Stabilization Stabilization Integration Integration Stabilization Stabilization 1 wk Release high quality CTPs every four weeks If a feature misses one CTP it catches the next Self Test Drive runs in main One week of feature branch integrations And bug fixes for features that have previously integrated Followed by one week of stabilization Only Self Test (BVT) fixes are accepted
51. Common Practices Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results 48
60. Telemetry Assessed from Dogfood and CTPs Great for qualities of service (e.g. MTBF) Tread carefully for usability conclusions
61. Common Practices Get Clean, Stay Clean Product Backlog Defining Done Scheduling Iteration Backlog Engineering Principles Measurement and Hardening Results 58
62. 2005 Debt vs. 2008 Debt 10x Debt Reduction! Minimal Deferral 2x Schedule Improvement High Predictability Huge Customer Sat Rise VS 2005 Beta 1 Product Bugs only VS 2008 Beta 1 ALL bug debt
67. Newton’s Third Law 62 Actions beget reactions, good and bad Paradoxical effects from Isolation and integration Quality early and change impedance Frequent integration and done done Test Automation and testing Pain and memory
70. “The Real Reason People Won’t Change” People and teams have Competing commitments Big assumptions Discovery has to be intentional This is the real maturation process Language and authenticity matter Kegan & Lahey, Harvard Business Review, Nov 2001
71. Pigs, Chickens and Al Haig The Organizational Paradox Healthy career development can lead to impaired memories (team and personal) Pigs tend to be centrifugal Chickens tend to be centripetal Tribalism abhors a vacuum What goes up can come down Celebrate successes, but don’t declare victory 66
72. Cost of Change Curve Revisited 67 Moral: Do not undersell the investment to start and the cost of Done
78. Aspirational Lean Scenarios No more production mismatches No more waiting for build setup No more UI regressions No more performance regressions No more missed requirements or changes No more no repro No more planning black box No more late surprises No more build breaks No more parallel development pain No more bewildering admin No more butterfly effects or legacy fear No more code & fix 70
79. Burndown Dashboard 71 How well are we estimating hours? And burning down the tasks? How quickly are implementing requirements? Are we removing impediments in time? Which impediments are still open? 71
80. QualityDashboard 72 Are we making progress on running test plans? Are build failures blocking progress? How quickly are we fixing bugs? Do the “fixes” actually fix the bugs? Are the tests covering the code on builds? How fast is code changing?
81. Bugs Dashboard 73 How quickly are we fixing bugs and what’s our current debt? What are our weekly find/fix/closure rates? How is the debt distributed by priority? What is still open right now?
82. Test Dashboard 74 Are we making progress on running test plans? Are test cases ready to run? What are the current test results for each requirement? How much testing is manual? What are the causes of failure?
83. VS 2010 Recognition Community votes for best product: Industry analysts recognize Microsoft leadership: VOKE: Test Market Mover Array July 2010 Evaluating VS2010 IDC: IT PPM Market Landscape December 2010 Evaluating VS2010 FORRESTER: Agile Development Management Tools Q2 2010 Evaluating VS2008
Democratization of ALM – a reality TFS in all MSDN Subscriptions and Visual Studio is the ALM brand 50% of SCCM market will shift in next 2 yearsVendor landscape - Integrated ALM now central to all vendors plans Two major competitors for single vendor ‘integrated ALM’ – IBM and Atlassian Price differential vs. competitors erodingCloud addressing core friction points in transitioning through the maturity model Every vendor has cloud plans, however majority positioning products as cloud
![Visual Studio Pro: The Developer Hub Visual Studio Playbook 1) Support the Microsoft Platforms 2) Create a world-class developer experience 3) Extend Visual Studio Platform to enable a vibrant ecosystem Centralized investment leads to success for All: ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)