Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quality Process KPIs Metrics


Published on

Published in: Business, Technology

Quality Process KPIs Metrics

  1. 1. Douglas Gabel Director, Quality Management Office "Implementing a Successful Goal-Driven KPI Approach in Quality"
  2. 2. Agenda <ul><li>cover the formula for SUCCESSFUL </li></ul><ul><li>personal choice KPI’s / Metrics </li></ul>
  3. 3. “ Count what is countable, measure what is measurable and what is not measurable, make measurable.” - Galileo, (1564 – 1642)
  4. 4. Why do we measure quality? The first answers that pop into your head might be: <ul><li>you can’t manage what you don’t measure </li></ul><ul><li>what you measure gets done </li></ul><ul><li>we have to be accountable </li></ul><ul><li>they have to be held accountable </li></ul><ul><li>they told us to </li></ul>
  5. 5. Today <ul><li>many in management still feel quality testing processes are fluff, overhead, or added process </li></ul><ul><li>quality teams need to give management more visibility into the value of quality testing </li></ul><ul><li>the challenge has always been determining what to measure and what KPI’s to use </li></ul><ul><li>we need to align quality testing with business outcomes to determine if the testing processes really contribute to the improvement of business objectives </li></ul>
  6. 6. Going Forward <ul><li>we need to determine: </li></ul><ul><ul><li>what to measure </li></ul></ul><ul><ul><li>what metrics to use </li></ul></ul><ul><ul><li>what is the best process to follow </li></ul></ul><ul><ul><li>what KPI’s are important to the business </li></ul></ul><ul><ul><li>what metrics are really needed to learn or understand the business </li></ul></ul><ul><ul><li>what can drive improvement </li></ul></ul>
  7. 7. Basis for presentation <ul><li>For those who have been delivering quality testing for some time, this basic approach is a derivation from the Mercury Quality Model. It has been proven successful and is a logical and simple series of steps that can benefit any Quality organization as a measurement strategy. </li></ul>To read more, Reference “Optimize Quality For Business Outcomes”, A Practical Approach to Software Testing by Andreas Golze, Charlie Li, Shel Prince
  8. 8. Formula for SUCCESSFUL <ul><li>S et business goals </li></ul><ul><li>U nderstand the impact of departments on those business goals </li></ul><ul><li>C hoose supporting business processes </li></ul><ul><li>C reate business-process goals </li></ul><ul><li>E xamine what to measure </li></ul><ul><li>S tandardize measurements across departments </li></ul><ul><li>S cope data source and integration needs </li></ul><ul><li>F ormalize indicators and thresholds </li></ul><ul><li>U tilize and execute appropriate action plans </li></ul><ul><li>L ay the groundwork and baseline </li></ul>
  9. 9. Set business goals <ul><li>establish high-level (specific) goals to measure against – not simple objectives </li></ul><ul><li>document them – write them down – communicate them </li></ul><ul><li>for each initiative you should have at least 3 or so goals </li></ul><ul><li>be sure to identify milestones / checkpoints </li></ul>
  10. 10. Understand the impact of each department/group on business goals <ul><li>groups having direct impact on the goal like Development, QA, Operations, Support, etc…should be evaluated </li></ul><ul><li>leveling resources, adjusting budgets, narrowing scope may be necessary activities to ensure success </li></ul><ul><li>evaluate non-IT demand departments for consequential impact </li></ul>
  11. 11. Choose supporting business processes <ul><li>Evaluate the various attributes of each business process that affects IT costs (staff, software, hardware, training, etc…) </li></ul><ul><li>challenge where streamlining can occur </li></ul><ul><li>value in stepping back and strategize at a high level where tactical changes can be made to the overall business model </li></ul>
  12. 12. Create business process goals <ul><li>these goals are sub-goals that assist with accomplishing a quality outcome </li></ul><ul><li>keep these goals to a limited set for tracking and reporting purposes </li></ul>
  13. 13. Examine what to measure <ul><li>establish a quantifiable set of questions to ask for each sub-goal (it’s easy to identify what metrics to use or measure when we ask the right questions – of the right people) </li></ul><ul><li>continue to ask why we measure, and take the accuracy of what you are measuring seriously </li></ul>
  14. 14. “ Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.” Albert Einstein
  15. 15. Standardize measurements across departments <ul><li>avoid using metrics that only one department will benefit from </li></ul><ul><li>keep metrics simple to avoid mis-interpretation </li></ul><ul><li>build consistency into any measures you utilize for greatest acceptance </li></ul><ul><li>find ways to interpret results with accuracy and be able to properly compare results across departments </li></ul>
  16. 16. Scope data source and integration needs <ul><li>always weigh the feasibility and ease of getting the data when creating metrics </li></ul><ul><li>allocate enough resources and time to set up the technology required to capture the data you want </li></ul>
  17. 17. <ul><li>data / result thresholds must be agreed upon so anyone looking at the data / results will interpret it the same way </li></ul><ul><li>thresholds help with the structure of reports and dashboards that management can understand and use </li></ul><ul><li>avoid representing every bit of data / results in a report / dashboard – this only clutters and confuses the message / findings </li></ul>Formalize indicators and thresholds
  18. 18. Utilize and execute appropriate action plans <ul><li>all KPI’s / thresholds aligned with business goals build the foundation for action plans </li></ul><ul><li>be prepared to act on pre-defined thresholds and ready to respond to dynamic business changes when thresholds are met </li></ul>
  19. 19. Lay the groundwork and baseline <ul><li>understand your environment, the state of your business </li></ul><ul><li>take a snapshot of the initial state of the business and use this as a baseline for comparison in the future </li></ul><ul><li>communicate, communicate, communicate your adherence and actions taken </li></ul>
  20. 20. In Summary <ul><li>SUCCESSFUL is a goal-driven approach with a logical and simple series of steps to follow to create a measurement strategy </li></ul><ul><li>this is an iterative process – revisit when goals are met or need to be realigned </li></ul>
  21. 21. Useful Measures / Metrics / KPI’s / Baselines
  22. 22. What does a good set of metrics look like? <ul><li>A good system of metrics provides a basis for the following: </li></ul><ul><li>Summary charts, showing the application related defect counts over time, their open and closure rates, and the progress towards delivery, policy compliance and risk assessment goals </li></ul><ul><li>the numbers necessary to answer management’s questions about “Is the application contributing to our business goals?”, “How robust and/or secure is the application?” or “Does this application put us at risk in meeting our business objectives?” </li></ul><ul><li>application/security defect density (that is, the average number of application/security defects per unit of code is being monitored and the rate is going in the right direction: Down!) </li></ul>
  23. 23. Useful References <ul><li>The following are a number of useful KPI’s / Metrics for most organizations: </li></ul><ul><li>Defect Removal Efficiency (DRE) </li></ul><ul><ul><li># defects in test phase / (# defects in test phase + # defects in prod 1 st 30 days) </li></ul></ul><ul><li>Test Case Effectiveness (TCE) </li></ul><ul><ul><li># defects not mapped to test cases / # defects mapped to test cases </li></ul></ul><ul><li>Traceability / Code Coverage </li></ul><ul><ul><li>% of test cases that map directly back to requirements and % of requirements that map directly to test cases </li></ul></ul><ul><li>Planned vs. Actual Execution </li></ul><ul><ul><li>line chart test cases planned vs. executed daily </li></ul></ul>
  24. 24. Useful References <ul><li>The following are a number of useful KPI’s / Metrics for most organizations: </li></ul><ul><li>Defects Distribution by Severity/Status </li></ul><ul><ul><li>bar chart defects by Severity (by project/cycle?) </li></ul></ul><ul><ul><li>bar chart defects by Status (by project/cycle?) </li></ul></ul><ul><li>Defects Distribution by Root Cause </li></ul><ul><ul><li>bar chart defects by Root Cause (by project/cycle?) </li></ul></ul><ul><li>Testing Effort </li></ul><ul><ul><li>actual testing effort (person hours) / total overall project effort (person hours) </li></ul></ul>
  25. 25. Useful References <ul><li>Suggested Baselines to start: </li></ul><ul><li>Defect Removal Efficiency (DRE) </li></ul><ul><ul><li>90% defects removed </li></ul></ul><ul><li>Test Case Effectiveness (TCE) </li></ul><ul><ul><li>10% not mapped </li></ul></ul><ul><li>Traceability / Code Coverage </li></ul><ul><ul><li>98% </li></ul></ul><ul><li>Planned vs. Actual Execution </li></ul><ul><ul><li>8% deviation +/- from plan </li></ul></ul><ul><li>Defects Distribution by Severity/Status </li></ul><ul><ul><li>bar chart actuals </li></ul></ul><ul><li>Defects Distribution by Root Cause </li></ul><ul><ul><li>bar chart actuals </li></ul></ul><ul><li>Testing Effort </li></ul><ul><ul><li>25 to 30% </li></ul></ul>