Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Measuring Agile: A Disciplined Approach To Metrics


Published on

This presentation works through important questions that people have about metrics on agile teams, principles around how to be effective with your agile metrics strategy, how to measure agile teams following a lightweight approach to GQM, potential metrics to collect about agile teams, and how to support IT governance through effective metrics rollups.

Published in: Software
  • Get the best essay, research papers or dissertations. from ⇒ ⇐ A team of professional authors with huge experience will give u a result that will overcome your expectations.
    Are you sure you want to  Yes  No
    Your message goes here
  • Did you try ⇒ ⇐?. They know how to do an amazing essay, research papers or dissertations.
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here

Measuring Agile: A Disciplined Approach To Metrics

  1. 1. © Disciplined Agile Consortium 1 Measuring Agile A Disciplined Agile Approach to Metrics
  2. 2. Agenda •  Important questions •  Principles •  Measuring agile teams •  Potential metrics •  Supporting IT governance •  Parting thoughts •  Questions © Disciplined Agile Consortium 2
  3. 3. Important Questions •  How do you measure solution delivery teams? •  What metrics should we collect? •  How do you roll up team measures to the portfolio level? •  What metrics are right for you? •  How do agile and traditional metrics fit together? •  What are the success factors for metrics programs? •  What are the failure factors? •  How does this fit into your overall governance strategy? © Disciplined Agile Consortium 3
  4. 4. © Disciplined Agile Consortium 4
  5. 5. Principle: There is No Easy Answer © Disciplined Agile Consortium 5
  6. 6. Principle: Every Metric Has Strengths and Weaknesses © Disciplined Agile Consortium 6
  7. 7. Principle: Compete Against Yourself © Disciplined Agile Consortium 7
  8. 8. Principle: Measure to Improve © Disciplined Agile Consortium 8
  9. 9. Principle: You Get What You Measure © Disciplined Agile Consortium 9
  10. 10. Principle: Measure Outcomes at the Team Level © Disciplined Agile Consortium 10
  11. 11. Principle: Every Team Needs a Unique Set of Metrics © Disciplined Agile Consortium 11
  12. 12. Principle: Teams Use Metrics to Self Organize © Disciplined Agile Consortium 12
  13. 13. Principle: Trust But Verify © Disciplined Agile Consortium 13
  14. 14. Principle: Adopt Common Metrics Categories Across Teams © Disciplined Agile Consortium 14 Time to Market Financial Stakeholder Satisfaction Quality Staff Morale Transform
  15. 15. Principle: Don’t Manage to the Metrics © Disciplined Agile Consortium 15
  16. 16. Principle: Automate Wherever Possible © Disciplined Agile Consortium 16
  17. 17. Principle: Prefer Trends Over Scalars © Disciplined Agile Consortium 17
  18. 18. Prefer Leading Over Trailing Metrics © Disciplined Agile Consortium 18
  19. 19. Principle: Prefer Pull (Dashboards) Over Push (Reports) © Disciplined Agile Consortium 19
  20. 20. Measuring Agile Teams © Disciplined Agile Consortium 20
  21. 21. Goal Question Metric (GQM) A lightweight GQM process: 1.  Identify improvement goals 2.  Generate questions that define those goals as completely as possible in a quantifiable way 3.  Identify potential measures needed to be collected to answer those questions 4.  Introduce mechanisms for data collection 5.  Collect, validate and analyze the data in real time to provide feedback to teams for corrective action Similar to Google’s OKR (Objectives and Key Results) © Disciplined Agile Consortium 21
  22. 22. •  Some goals are qualitative: –  Invest in IT wisely –  Improve staff morale –  Improve stakeholder satisfaction with delivered solutions •  Some goals are quantitative: –  Decrease cycle time by 20% this quarter –  Reduce technical debt by 10% by year end –  Get our net promoter score (NPS) to 60% by quarter end •  Our advice: –  Short time frames (less than three months) work well –  Quantitative goals are generally better than qualitative © Disciplined Agile Consortium 22 Identify Improvement Goals
  23. 23. © Disciplined Agile Consortium 23
  24. 24. Goal: Invest in IT Wisely Questions 1.  How effective is the investment in IT activities? 2.  What future spending do we require? 3.  Is productivity increasing? Potential Metrics 1.  Value delivered, net present value (NPV), return on investment (ROI) 2.  Effort/cost projection (burn ups/burn downs) 3.  Acceleration, business value delivered, velocity (trend) © Disciplined Agile Consortium 24
  25. 25. Goal: Decrease cycle time by 20% this quarter Questions 1.  Is the team working at a sufficient pace? 2.  Is the team working together effectively? 3.  Are changing requirements putting the release date at risk? Potential Metrics 1.  Release burndown, ranged release burndown 2.  Team morale, stakeholder satisfaction, blocking work items 3.  Age of items, ranged release burndown © Disciplined Agile Consortium 25
  26. 26. Goal: Reduce Technical Debt by 10% by Year End Questions 1.  What is our current level of technical debt? 2.  Are we improving quality over time? Potential Metrics 1.  Code quality, defect density 2.  Build health, defect density, test coverage, code quality © Disciplined Agile Consortium 26
  27. 27. Potential Metrics © Disciplined Agile Consortium 27
  28. 28. Iteration Burndown •  Indicates the likelihood of delivering as promised this iteration •  Application: –  By team to manage their time •  Advantages: –  Helps to avoid surprises at end of iteration –  Motivates a continuous and sustainable pace •  Disadvantages: –  Overhead to update estimated work remaining © Disciplined Agile Consortium 28
  29. 29. Velocity •  Velocity is a measure of the team’s delivery capacity for each iteration measured in “points” •  Options: –  Gross velocity: Number of points delivered in an iteration –  Net velocity: Delta in the number of points on the work item list –  Smoothed velocity: Velocity averaged over several iterations •  Application: –  By team to extrapolate the schedule. e.g. If total points in Work Item List is 400, with velocity of 40 there is 10 iterations left •  Advantages: –  Simple way to project schedule/cost •  Disadvantage: –  Easy to manipulate because points are manually identified –  Not easily comparable across teams © Disciplined Agile Consor1um 29
  30. 30. Release Burndown Chart © Disciplined Agile Consortium 30
  31. 31. Release Burnup Chart © Disciplined Agile Consortium 31
  32. 32. Ranged Release Burndown •  A ranged estimate of number of iteration required to complete work •  Range of uncertainty decreases over time •  BUT… many people don’t like ranged estimates © Disciplined Agile Consortium 32
  33. 33. Ranged Burnup Chart © Disciplined Agile Consortium 33
  34. 34. Acceleration •  The change in velocity over time •  Application: –  Use to calculate productivity improvement (acceleration * number of people on team * charge rate) •  Advantages: –  Comparable across teams –  Easy to calculate •  Disadvantages: –  Easy to manipulate because points are manually identified © Disciplined Agile Consortium 34 0 5 10 15 20 25 30 Velocity Iterations
  35. 35. Value Delivered – Value points •  Assign “value points” to a story that captures the estimated financial value of the delivered functionality. •  Application: –  Estimate the value delivered by a team •  Advantage: –  Provides insight into the value being produced by a team •  Disadvantage: –  Easy to game as it’s a manual estimation –  Doesn’t take cost of delay into account (without re- estimating) © Disciplined Agile Consortium 35
  36. 36. Code Quality Metrics •  A category covering a very wide range of metrics that provide insight into how well source code has been written. •  Options –  Halstead, Cyclomatic complexity, Defect density, Coupling, and many more •  Applications: –  Determine the existing quality of code –  Pinpoint potential code quality problems •  Advantages: –  Easy to automate –  Many commercial and open source tools •  Disadvantage: –  Easy to get overwhelmed by the data © Disciplined Agile Consortium 36
  37. 37. Defect Trends •  Track when and where defects are found against when they are addressed •  Options: –  By severity –  By source •  Application: –  Monitor defects/bugs that escape the delivery team •  Advantage: –  Motivates teams to fix defects –  Can provide insight into where you need to improve your quality strategy •  Disadvantage: –  Very easy to game because defects are often identified manually © Disciplined Agile Consortium 37
  38. 38. Test/Code Coverage •  Calculated as the number of lines invoked by tests divided by the total number of lines of relevant code •  Application: –  Determine how much of your code base has automated tests •  Advantages: –  Easy to automate –  Low test coverage tells you that you need to automate more of your tests •  Disadvantage: –  High test coverage may not tell you much because this metric is easy to game with low-quality tests © Disciplined Agile Consortium 38
  39. 39. Build Status •  Monitor the status of the build •  Options: –  Trend –  Current status •  Advantage: –  Trend report useful to teams new to continuous integration (CI) –  Very useful indicator if build broken for long time •  Disadvantage: –  Once you have implemented CI effectively the trend metric loses value © Disciplined Agile Consortium 39
  40. 40. Net Promoter Score (NPS) •  Net Promoter Score (NPS) measures the loyalty that exists between a provider and a consumer •  Application: –  Gauge the loyalty of your customers •  Advantage: –  Indicates potential for revenue growth •  Disadvantage: –  Manually generated and easy to “fix” with enticements © Disciplined Agile Consortium 40
  41. 41. Resource Utilization •  Measure of the percentage of time that people spend working. Typically calculated as number of hours worked divided by number of hours they should have worked •  Application: –  Matrix organizations where some (or most) people aren’t “self starters” who need to be assigned to work •  Advantages: –  Identify people available for work •  Disadvantages: –  Motivates you to build teams out of whom you have available instead of whom would be a good fit –  Increases stress and motivates people to fill in their time instead of to add value –  Often results in little or no slack time © Disciplined Agile Consortium 41
  42. 42. Skill Level •  People are rated against desired skills on a scale of 1 to N •  Application: –  Identify skills gaps –  Justify training or coaching efforts •  Options: –  Individual or Team –  Self or external assessment •  Advantages: –  Focuses training/coaching efforts •  Disadvantages: –  Easy to game, particularly with self- assessment © Disciplined Agile Consortium 42
  43. 43. Disciplined Agile Certifications •  Measure adoption rate of the DA certifications across your organization •  Application: –  Indicates skill and experience level of team members •  Advantage: –  Measured from outside so difficult to game –  When combined with other metrics can provide an accurate estimate of the value of your investment in training and coaching –  Focuses training and coaching efforts •  Disadvantage: –  Need to invest in the training and coaching of your staff over a long term basis © Disciplined Agile Consortium 43 •  Knowledge •  Knowledge + 2+ years experience •  Knowledge + 5+ years experience + Give back
  44. 44. Lead and Cycle Time •  Measure the length of time to develop a feature from concept to cash •  Application: –  Identify places where you need to improve your process –  Indication of time productivity: (Cycle Time-Internal Wait Time Within Cycle)/Lead Time •  Advantages: –  Measures time-to-market, a critical business success factor –  Looks at overall process, not just development •  Disadvantage: –  Can lead to finger pointing © Disciplined Agile Consortium 44
  45. 45. Cost of Delay •  A measure of the impact of time on the business outcomes we hope to achieve •  Application: –  Input into work (feature or project) prioritization •  Advantages: –  Surfaces important financial considerations •  Disadvantages: –  Easy to game as you need to guesstimate the value curve(s) –  Requires understanding of basic financial concepts © Disciplined Agile Consortium 45
  46. 46. Cumulative Flow Diagram •  Visualize the work effort of a team over time •  Advantages: –  Indicates potential bottlenecks in your process (bumps on the diagram) –  Simple report commonly generated by agile management tools such as JIRA, Rally, or VersionOne •  Disadvantage: –  Easy to “fix” because it shows points of work, which are manually guesstimated © Disciplined Agile Consortium 46 0 10 20 30 40 50 60 70 80 90 May '16 June '16 July '16 Aug '16 Sept '16 Oct '16 Backlog In Progress Done
  47. 47. Supporting IT Governance © Disciplined Agile Consortium 47
  48. 48. IT Intelligence (Automated Dashboards) •  Automated dashboard that summarizes the status for all of IT •  Shows the entire portfolio: –  Potential/suggested endeavors –  Ongoing development endeavors –  Operational solutions •  Enables drill down into details © Disciplined Agile Consortium 48
  49. 49. Rolling Up Metrics © Disciplined Agile Consortium 49 Portfolio Team •  Measure against organizational goals •  Overall indicator or category indicators •  Organizational goals evolve slowly (~annually) Program •  Measure against program goals •  Category indicators per sub-team •  Program goals evolve quarterly (or thereabouts) •  Measure against team goals •  Specific metrics per team •  Metrics evolve regularly as the situation evolves •  Unique dashboard per team * 0..1 * 11 0..*
  50. 50. © Disciplined Agile Consortium 50
  51. 51. Metrics Principles •  There is no easy answer •  Every metric has strengths and weaknesses •  Compete against yourself •  Measure to improve •  You get what you measure •  Measure outcomes at the team level •  Each team needs a unique set of metrics •  Team use metrics to self organize •  Trust but verify •  Adopt common metrics categories across teams •  Don’t manage to the metrics •  Automate wherever possible •  Prefer trends over scalars •  Prefer leading over trailing metrics •  Prefer pull over push © Disciplined Agile Consortium 51
  52. 52. Goal Question Metric © Disciplined Agile Consortium 52
  53. 53. Some Potential Metrics •  Iteration burndown •  Velocity and Acceleration •  Release burndown/up •  Points/Value delivered •  Code quality •  Defect trends (by severity, by source) •  Test coverage •  Build status •  Net promoter score (NPS) •  Resource utilization •  Team skill level •  DA certifications •  Lead and cycle time •  Cost of delay •  Cumulative flow diagram © Disciplined Agile Consortium 53
  54. 54. Important Resources © Disciplined Agile Consortium 54 Disciplined Agile IT Governance Webinar: February 2017 Chapter 20: Governing Disciplined Agile Teams
  55. 55. © Disciplined Agile Consortium 55
  56. 56. Got Discipline? © Disciplined Agile Consortium 56
  57. 57. Shuhari and Disciplined Agile Certification At the shu stage you are beginning to learn the techniques and philosophies of disciplined agile development. Your goal is to build a strong foundation from which to build upon. At the ha stage you reflect upon and question why disciplined agile strategies work, seeking to understand the range of strategies available to you and when they are best applied. At the ri stage you seek to extend and improve upon disciplined agile techniques, sharing your learnings with others. © Disciplined Agile Consortium 57
  58. 58. Scott Ambler + Associates is the thought leader behind the Disciplined Agile Delivery (DAD) framework and its application. We are an IT management consulting firm that advises organizations to be more effective applying disciplined agile and lean processes within the context of your business. Our website is We can help © Disciplined Agile Consortium 58