Downloaded 302 times
Uploaded byXBOSoft
Top 10 Agile Metrics
The document is a presentation by Xbosoft about agile metrics, highlighting the importance of measurement in software quality and project management. It discusses various metrics, frameworks, and objectives that can help teams track progress, quality, and efficiency while aiming for continuous improvement in agile practices. Additionally, it features insights from experts and encourages interactive participation from attendees.
More Related Content
Top 10 Agile Metrics
- 1. Agile Metrics XBOSoft,Inc. All Rights Reserved. 1
- 2. XBOSoft Founded in2006 Dedicated to software quality Software QA Consulting Software Testing Offices in San Francisco, Beijing and Amsterdam XBOSoft, Inc. All Rights Reserved. 2
- 3. House Rules •Everyone except the speakers is muted • Questions via gotowebinar control panel OR via Twitter @XBOSoft using #Top10AgileMetrics • Questions can be asked throughout the webinar, and will be answered if possible • You’ll receive information for recording after the webinar XBOSoft, Inc. All Rights Reserved. 3
- 4. Meet Our Speakers • CEO and Founder of XBOSoft • Relevant specialties and passions – Software quality process, evaluation, measurement and improvement – Mobile User Experience and usability – Cycling and travel Philip Lew • Go2Group Vice President • ALM Expert • Over 30 years with over 100 client installations of ALM toolsets in a variety of development methodologies Doug Bass XBOSoft, Inc. All Rights Reserved. 4 Sabrina Gasson • Marketing Manager of XBOSoft • Emails you all regularly to join our industry hot topic webinars • And invites you all to download our latest trends in software testing whitepapers.
- 5. WHAT IS ONEOF THE MOST FAMOUS THINGS HE SAID? “LIFE IS LIKE A BOX OF CHOCOLATES, YOU NEVER KNOW WHAT YOU’RE GOING TO GET” XBOSoft, Inc. All Rights Reserved. 5
- 6. Agile Metrics Byusing metrics, we can know what we are going to get!
- 7. Agenda 1. Agile,Frameworks and GQM 2. Applying GQM to Agile 3. Top 10 Agile Metrics XBOSoft, Inc. All Rights Reserved. 7
- 8. Poll XBOSoft, Inc.All Rights Reserved. 8
- 9. What We CanAgree On • Want to minimize mistakes and doing things over (rework) • Want to utilize resources effectively • Want to predict when things could go wrong • Want to do what we say we will do, when we say - timelines • Fix defects early – later is more expensive • Production defects are bad news XBOSoft, Inc. All Rights Reserved. 10
- 10. What We CanAgree On About Metrics • Want easy to get/capture • Should have a valuable purpose toward improvement – Evaluate – Guide – Insightful – Actionable XBOSoft, Inc. All Rights Reserved. 11
- 11. Agile – HowCan We Improve? • Some say no metrics needed • Is velocity all you track? • What is sacrificed for velocity? • How can we know what and how we are doing or where to improve? • How do we know if we’re using resources effectively? • How do we know we are spending our effort/time in the right place? XBOSoft, Inc. All Rights Reserved. 12
- 12. Poll XBOSoft, Inc.All Rights Reserved. 13
- 13. Using Frameworks to Organizing Our Thoughts • Putting metrics into a framework helps us to categorize and organize • Helps us to make sure we have all functions in our organization and process covered • Helps us to develop relationships between different elements in our process
- 14. Applying Basili’s GQM • Many software metrics programs have failed because they had poorly defined, or even non-existent objectives. Goal Metric Question
- 15. Stakeholders and GQM Goal 1 Goal 2 Goal 3 Question 1 Question 2 Question 3 Question 4 Question 5 Metric 1 Metric 2 Metric 3 Metric 4 Metric 5 Metric 6 Metric 7 CEO CTO Dev Mgr QA Manager
- 16. GQM - Example • Goals – Reduce total cost of development – Reduce total cost of testing-effort – Reduce the number of new feature bugs • Questions – Which areas have the highest re-work rates? – Which functional areas have the most defects? – How long does it take to repair defects? – How complete are my requirements? – How much time do I spend testing?
- 17. Metric Q&A Worksheet Category Question Metric Tool to Use Needed Customer Are we meeting Schedule Satisfaction schedule? Variance Project Management Tool None Software Testing Efficiency Software Quality What kinds of defects are most common? Usability Are end users getting lost, understand our UI Software Test Effectiveness ?
- 18. Agile Considerations Waterfall • Speed • Quality • Cost Agile • Speed • Quality • Cost Where’s the beef? XBOSoft, Inc. All Rights Reserved. 20
- 19. Agile Manifesto RETURNTO BASICS
- 20. Excerpts from theAgile Manifesto 1. Working software over comprehensive documentation 2. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. 3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. 4. Business people and developers must work together daily throughout the project. 5. Working software is the primary measure of progress. 6. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. 7. Continuous attention to technical excellence and good design enhances agility.
- 21. Agile – UnderstandingWhere We Came From? XBOSoft, Inc. All Rights Reserved. 23
- 22. Poll XBOSoft, Inc.All Rights Reserved. 24
- 23. What are theGoals of Agile? Speed Quality Customer-User Satisfaction
- 24. Matching Agile Characteristicsand Objectives Agile Characteristic Speed Quality Customer and End User Satisfaction Other Collaborative x x Working x Software Adapt to changing requirements x x From the agile manifesto
- 25. What Questions TellMe If I’m Reaching Those Objectives? Speed Quality Working Product Requirements Owner Participation Technical debt Schedule Planned versus unplanned
- 26. What Questions TellMe If I’m Reaching Those Objectives? Speed Requirements Did I get them in time? Do I understand them? Did we deliver working product? Schedule Am I on schedule? Can I be more productive? How much wasted time? Rework? Unplanned versus planned Did I estimate well? Did I finish what I said I would? Do I make mistakes that cause more (unplanned) work?
- 27. What Questions TellMe If I’m Reaching Those Objectives? Quality Product owner Did my end users find defects? Did product owner give feedback regularly? Are there less defects than before? Requirements Are my requirements delivered when needed? Did the delivered product match the requirement? Or reqt. wrong? Do engineers have trouble understanding the requirements? Technical Debt Is my software performance getting worse and worse? Am I prepared if my lead engineer leaves? Do I mess things up when fixing things?
- 28. ARE YOU READYFOR THE COUNTDOWN? METRICS THAT HELP US ANSWER THESE QUESTIONS!
- 29. 10. VELOCITY AmI on schedule? Did I estimate well? XBOSoft, Inc. All Rights Reserved. 32
- 30. Velocity • Trackingour velocity over time tells us our progress and capacity and helps us forecast based on our capabilities (now and in the future). • Measurement – Sum of all the approved estimates of the stories • One point in time measurement has little meaning, so need to track over time or wrt another metric • Metric – Velocity (T)-(T-1)/Velocity (T)
- 31. Velocity Velocity Variance XBOSoft, Inc. All Rights Reserved. 34 30% 25% 20% 15% 10% 5% 0% -5% -10% -15% -20% -25% 1 2 3 4 5 6 7 8 9
- 32. Burn Down –(Distance over Time) XBOSoft, Inc. All Rights Reserved. 35
- 33. 9. DEFECT REMOVALEFFECTIVENESS Did my end users find defects? Are there less defects than before? XBOSoft, Inc. All Rights Reserved. 36
- 34. Defect Removal Effectiveness • We don’t want the end user to find defects • Any defect found by the user influences quality • Metric – Defects in found in production during (90 day) period post release (DFP)/DFP + Defects found Pre-release
- 35. What Should YourGoal Be For DRE • Estimated (by Capers Jones) that defect removal effectiveness differs by different levels of process capability maturity levels: – Level 1: 85% – Level 2: 89% – Level 3: 91% – Level 4: 93% – Level 5: 95% XBOSoft, Inc. All Rights Reserved. 38
- 36. DRE Distribution C A XBOSoft, Inc. All Rights Reserved. 39
- 37. 8. ADDED WORK Am I on schedule? Am I overcommitting? Is my rate of work sustainable? XBOSoft, Inc. All Rights Reserved. 40
- 38. Added Work •This will help us see if we are making promises that maybe we should not – overcommitting and unsustainable. • Measurement – Sum of work (new stories added) over and above the original plan. • Metric – New stories added/Original Stories Planned
- 39. Piling On XBOSoft,Inc. All Rights Reserved. 42 35 30 25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 Revised Planned Original Plan
- 40. 7. WORK CATEGORYALLOCATION Am I allocating resources optimally? Do I have trouble understanding requirements? XBOSoft, Inc. All Rights Reserved. 43
- 41. Work Category Allocation • This will help us see where we are spending our time and adjust our priorities. • Measurement – Sum of work (for each category) Metric – Divide by total work to get % allocation • Categories could include: – Technical debt – refactor - redo – Regression testing – New feature development – Research – Requirements review – Defect fixing – Other
- 42. Is Effort inAlignment? XBOSoft, Inc. All Rights Reserved. 45 Are we spending enough time showing our stakeholders what we’ve done? 30% 25% 20% 15% 10% 5% 0% Are we fixing things too much or making lots of mistakes?
- 43. 6. WORK SUSTAINABILITY Do I have the right people on my team? Did I estimate work well? Can I attain quality sustainably? XBOSoft, Inc. All Rights Reserved. 46
- 44. Work Sustainability •“Sustainable” - We can’t sustain our effort if we are continuously putting in overtime • Overtime is necessary, but needs to be managed and causes mistakes • Mistakes manifest themselves in many ways, such as technical debt or defects • Overtime often results from poor work estimates and directly proportional to technical debt • Metric – Unplanned Overtime Work/Total Work
- 45. Unplanned Overtime % XBOSoft, Inc. All Rights Reserved. 48 16% 14% 12% 10% 8% 6% 4% 2% 0% 1 2 3 4 5 6 7 8 9 Sprint What caused this?
- 46. Unplanned Overtime andDefects XBOSoft, Inc. All Rights Reserved. 49 7 6 5 4 3 2 1 0 1 2 3 4 5 6 7 Defects Found Unplanned OT
- 47. 5. DEFECT REWORKRATE Is my technical debt accumulating? Am I wasting time? XBOSoft, Inc. All Rights Reserved. 50
- 48. Defect Rework Rate • We can’t get faster if we are always fixing things. • Depends on valid data whereby people are logging real hours to tasks they take on. • Metric - Fix Effort Ratio : Time to fix defects/Total effort expended (hours)
- 49. Defect ReWork % XBOSoft, Inc. All Rights Reserved. 52 35% 30% 25% 20% 15% 10% 5% 0% 1 2 3 4 5 6 7 8 9
- 50. Accumulated Debt XBOSoft,Inc. All Rights Reserved. 53 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 6 7 8 9 Defect Rework% Unfixed %
- 51. 4. TOTAL UNEXPECTEDWORK What is throwing me off schedule? Am I biting off too much? XBOSoft, Inc. All Rights Reserved. 54
- 52. Total Unexpected Work • Measurement – Sum of work done above the original estimates for those stories in the plan. • Are we underestimating difficulty? • Many reasons: – Incomplete user stories – Misunderstood user stories – Bigger and more complex than I thought? – Not enough time for research – Too hard… • Metric – (Total work done – Planned work)/Planned work
- 53. Are We EstimatingWell? XBOSoft, Inc. All Rights Reserved. 56 60 50 40 30 20 10 0 1 2 3 4 5 6 7 8 9 Planned Unplanned Per Person/Week During the Sprint
- 54. What are wedoing that is unplanned? Research Requirements Coding Testing Other 20% XBOSoft, Inc. All Rights Reserved. 57 29% 9% 19% 23%
- 55. 3. DEFECTS INPRODUCTION AVG. FIX TIME Are my customers/end users happy? Is the architecture prohibiting me? XBOSoft, Inc. All Rights Reserved. 58
- 56. Defect in ProductionAvg. Fix Time • If defects make it to production, that’s bad ! • But what is worse if an important defect stays unfixed. • Defects that take a long time to fix represent either technical debt or poor process • Metric – Time to fix Defects (released to production-usually P1 defects found) / Total P1 Defects (post production)
- 57. Don’t Make ThemWait Hey, we’re improving! WooHoo! XBOSoft, Inc. All Rights Reserved. 60 60 50 40 30 20 10 0 1 2 3 4 5 6 7 8 9 #P1 Defects Found Average Fix Time (Hrs)
- 58. 2. DEFECT CATEGORIZATION ALLOCATION % Why are things breaking? Can I get faster? XBOSoft, Inc. All Rights Reserved. 61
- 59. Defect Categorization Allocation • Reduce rework = Increase speed • Discover when defects are injected • Reduce and prevent defects from happening in the first place. • Drive down the Fix Effort Ratio leading to better quality and resulting in higher velocity. • Examine defect rates per category
- 60. Defect Rework Allocation Requirements Coding Environment Testing 38% XBOSoft, Inc. All Rights Reserved. 63 29% 21% 13% What’s going on here?
- 61. Rework-Requirements Errors Requirements • Change in requirements • Don’t understand requirements • Incomplete requirement • Requirement does not exist • ? ‘oh, I meant this…’
- 62. Poll XBOSoft, Inc.All Rights Reserved. 65
- 63. 1. WORKING SOFTWAREDELIVERY RATE Am I delivering what I said I would? Am I getting feedback from the PO? Will I meet the release date? XBOSoft, Inc. All Rights Reserved. 67
- 64. Working Software DeliveryRate • Does the software work or not? • Working does not mean perfect • Working means can show in order to get feedback • Metric - # story points delivered that ‘work’ / # story points that were to be shown to the PO but could not be shown to work – Many variations
- 65. Am I GettingBetter? XBOSoft, Inc. All Rights Reserved. 69 25 20 15 10 5 0 1 2 3 4 5 6 7 8 9 Planning Working Visible Actual Working
- 66. Am I DeliveringSo I can Get Feedback? % Working XBOSoft, Inc. All Rights Reserved. 70 100% 80% 60% 40% 20% 0% 1 2 3 4 5 6 7 8 9
- 67. Be Agile Isa Way of Thinking “Be-Do-Have” People • Communicative • Collaborative/Cooperative • Flexible and willing • Knowledgeable-multi • Initiative/responsible • Responsive/adaptive Do Have Process • Iterative (sprints) • Daily standups • Face to face communication • Post mortem – end of sprint • Delivery meeting – end of sprint • Planning meeting – before sprint • Self organizing Results • Speed • Quality XBOSoft, Inc. All Rights Reserved. 71
- 68. TAKE AWAYS-SUMMARY •What and why • Goals and objective s Understand Ask yourself questions Evaluate Measure to answer those questions and take action Improve
- 69. Q+A Post yourquestions on Twitter and we'll answer them @XBOSoft Join us to keep updated on all our webinars, reports and whitepapers: facebook.com/xbosoft +xbosoft linkedin.com/company/xbosoft We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/ Why not download our free Whitepapers, available here: http://xbosoft.com/knowledge-center/ You will receive an email with information on slides and recording. Any further queries regarding our services or ideas for future webinars please email us! services@xbosoft.com www.xbosoft.com Our latest whitepaper on Agile Metrics will be released next week so check back soon!
Editor's Notes
- #7 Business people call them KPI or Key Performance Indicators but basically we are trying to see how we are doing. And you can only do that with measurement.
- #8 This comes from half a dozen workshops over the last 18 months, where I’ve taught at STPCON, PNSQC, Techwell Quest and many others where I’ve been able to summarize all the learning in both directions!
- #11 OF course there are many more, but here are some general principles to think about, and I think we all agree on. These general thoughts help us to form our TOP 10
- #12 Valuable is the important word here, not just to look at charts and graphs
- #13 I will ask Doug on his experience in what he sees people doing.
- #17 GGM is really fitting for me and its not a mistake that I like this paradigm, When I was young My Dad would tell me to do things, and I’d always ask WHY? Overall GOAL is to ? What questions that if answered, tell us information pertaining to the goal? Analyze each question to determine what must be measured (and an associated METRIC determined) in order to answer the question. Generate those metrics that are related to the goal. Several metrics may answer a single question Single metric may apply to more than one question
- #24 Agile is about working product in order to get feedback Agile is about involving the end user/product owner to get buy in == quality Agile is about delivering 20-80 and 80-20, get the critical stuff up and working
- #27 The end goal is to satisfy the customer, that goes with these and is in alignment with these. Any others?
- #29 Maintain schedule – be on time with respect to delivery – speed Working product – show to owners to get real feedback, no working product, no feedback Planned versus unplanned – rework, injection, pull forward, bad estimates, padding Owner participation – no participation, bad quality, we can’t document everything, this is agile! Requirements – Toss off, we want them nailed down early, but we don’t want them nailed down. Many related elements that impact velocity Requirements clarity Product owner participation Unplanned work Many related elements that impact quality Requirements clarity Product owner participation
- #30 Group exercise listing out questions
- #31 Group exercise listing out questions
- #32 I had over 30 to start with and narrowed down to these based on our logic of the previous slides.
- #33 I would bet he could almost predict how fast he would run? How much he ate How much he slept How much he weighed What he ate
- #37 Widely touted by Capers Jones as THE METRIC to collect and monitor --- Again is a resultant metric, like measuring your weight, rather than measuring what you eat.
- #38 NO MATTER HOW SMALL IT IS
- #40 This is how they graded us in school based on a curve right doug?
- #42 We don’t want to over commit --- leads to user or PO dissatisfaction
- #49 Doug, what kinds of things would cause this?
- #50 Gee what’s going on here Doug? Have you seen this before?
- #54 We fixed them all versus we didn’t fix many at all!
- #57 What kinds of things do you find Doug? What happens when this kind of thing continues?
- #60 NO MATTER HOW SMALL IT IS
- #65 Perhaps misrepresented, ‘oh, I meant this…’
- #69 NO MATTER HOW SMALL IT IS