Successfully reported this slideshow.
Your SlideShare is downloading. ×

Software Quality Metrics Do's and Don'ts - QAI-Quest 1 Hour Presentation

Loading in …3

Check these out next

1 of 62 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (20)


Similar to Software Quality Metrics Do's and Don'ts - QAI-Quest 1 Hour Presentation (20)

More from XBOSoft (20)


Recently uploaded (20)

Software Quality Metrics Do's and Don'ts - QAI-Quest 1 Hour Presentation

  1. 1. Quest  2014   1  
  2. 2. "Software Quality Metrics Do’s and Don’ts" Philip Lew
  3. 3. Meet Your Instructor •  Team Lead, Data warehousing product development •  Software product manager, BI product •  COO, large IT services company •  CEO, XBOSoft, software qa and testing services •  Relevant specialties –  Software quality process improvement –  Software quality evaluation and measurement –  Software quality in use / UX design –  Mobile User Experience and usability Quest  2014   3  
  4. 4. Let’s  Meet  Each  Other   •  Name   •  Your  Company/Role   •  InteresAng  Adbit  or  fact   •  ObjecAve  in  being  here   ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   4  
  5. 5. Some  InformaAon   •  Used  1  hour  efficiently  and  covered  the   material.  It  was  interacAve  too.   •  Presenter  was  comfortable  and  engaging.   Content  was  good.   •  Very  interacAve  and  demanding,  commands   great  aWenAon.   •  Best  content.  Provided  great  brainstorming   opportuniAes  and  gave  examples.  Great   presenter.   Quest  2014   5  
  6. 6. More  informaAon   •  Slides  too  low  where  people’s  heads  are.  Can’t  see.   •  How  do  we  measure  quality?  You  didn’t  answer  that   at  all.  Very  disappointed.   •  I  realize  interacAon  is  good,  but  found  the   discussions  to  be  distracAng.   •  It  was  too  hard  to  hear  what  was  said.   •  Instructor  only  focused  on  people  in  front  who’s   hands  were  up.   •  Too  large  a  group.   •  Slides  were  different.   Quest  2014   6  
  7. 7. Session Spirit and Expectations •  Interactive given time and other context •  I won’t read the slides… – Slides for you as a take-away – Slightly different than your handouts – Some new examples and ideas – Write me and I’ll send these to you •  I’ll repeat questions (if I can remember) 7  Quest  2014  
  8. 8. Quest  2014   8  
  9. 9. Motivation Why is QA Measurement Important •  Quality  Assurance  measurement  provides  informaAon  to   answer  quesAons  about  your  tesAng,  your  team,  and  your   company.     –  Is  your  QA  team  improving?   –  Have  you  reached  the  highest  state  of  efficacy?     –  Is  your  tesAng  as  thorough  as  it  can  be?     –  Is  soMware  ready  for  release?  Can  we  stop  tesAng  now?   –  Are  you  able  to  do  everything  you  want  within  budget?   –  What  parts  of  the  soMware  are  at  most  risk  ?   •  If  the  answer  to  any  one  of  those  quesAons  is  “no”,  or  “I  don’t   know”,  then  you  are  in  the  right  place.     ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   9  
  10. 10. Meal of Metrics Schedule   variance   Effort   variance   Cost   variance   Defect   removal   effecAveness   ProducAvity   Defect   aging   CriAcal   Defect   rate   Test   cases   executed   Test   coverage   Pass/Fail   Rate   ROI   AutomaAon   coverage   10   Value   Provided   %   Defects/ hour   Customer   saAsfacAon   Avg.   Time  to   Fix   Recurrence   Rate   Code   complexity   Test   Density   Defect   Find  Rate   Defect   Arrival   Rate   Require ments   VolaAlity   Require ments   Ambiguity   Planned /Actual   OverAme   Rate   CompleAon   Rate   Velocity   ExecuAon   Rate   ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.  
  11. 11. Don’t – Calculate metrics that don’t help answer specific questions •  Are you collecting measurements and calculating metrics without thinking what answers will I get after collecting this information? Quest  2014   11  
  12. 12. POLL: How many metrics are you collecting on a regular basis within your organization?A.  1-5 B.  6-10 C.  11-15 D.  0
  13. 13. Metrics - Benefits •  Understand  how  QA,  tesAng,  and  its  processes   and  where  the  problems  are   •  Evaluate  the  process  and  the  product  in  the  right   context   •  Predict  and  control  process  and  product  qualiAes   •  Reuse  successful  experiences   –  Feed  back  experience  to  current  and  future  projects   •  Monitor  how  something  is  performing   •  Analyze  the  informaAon  provided  to  drive   improvements     13  Quest  2014  
  14. 14. How can measurement help us (YOU) •  Create a organizational memory – baselines of current practices-situation •  Determine strengths and weaknesses of the current process and product –  What types of errors are most common? •  Develop reasoning for adopting/refining techniques –  What techniques will minimize the problems? •  Assess the impact of techniques –  Does more/less manual functional testing versus automation reduce defects? •  Evaluate the quality of the process/product –  Are we applying inspections appropriately? –  What is the reliability of the product before/after delivery? 14  Quest  2014  
  15. 15. Boil it Down-Understand, Evaluate and Improve •  To do this… We need metrics Can you think of other examples in our lives where this applies? Where do you use metrics to evaluate and improve? 15  Quest  2014   Understand   Evaluate   Improve  
  16. 16. Group Exercise •  How does your organization define software quality? •  Come up with a definition of software quality that you can agree on. – Can be one sentence to a short paragraph. ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   16   If the goal is to improve software quality, then what is IT?
  17. 17. Metrics in real life Food  Eaten   Weight   Performance   Race  Results   17   •  Calories •  Fat •  Carbohydrates •  Protein •  Time of day •  Vitamins •  … •  Blood pressure •  Cholesterol •  Blood glucose •  Red cell count •  White cell count •  Hematocrit •  Hemoglobin •  Body fat % •  … •  Placing •  … •  Effort/Power •  Heart rate/Watts •  Speed •  Time Intelligence Finesse Context •  Training •  Sleep Quest  2014  
  18. 18. DO Think about the process you are measuring and measure at each step in the process.
  19. 19. Software Quality Components ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   19   ISO 25010CMMI
  20. 20. Evolution of SW Quality ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   20   Software processes Code Process quality Software Quality (internal) Software Quality (external) Software Quality (in use) Product Usability CMMI assessment model white box testing black box testing Usability testing Usability heuristic Usability logging CMMI Assessment Company Programmer Tester End User Type of quality What is measured How measured? Who measures? ISO 9000 ISO 9241ISO 9126 ISO 25010
  21. 21. Total Quality View (organizational) ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   21  
  22. 22. For  Those  in  Agile   Waterfall   •  Speed   •  Quality   •  Cost   Agile   •  Speed   •  Quality   •  Cost   Quest  2014   22   Where’s the beef?
  23. 23. Be-­‐Do-­‐Have   Be   Do   Have   Quest  2014   23  
  24. 24. Be-­‐Do-­‐Have   Be   Do   Have   Quest  2014   24   Process •  Iterative (sprints) •  Daily standups •  Face to face communication •  Post mortem – end of sprint •  Delivery meeting – end of sprint •  Planning meeting – before sprint •  Self organizing People •  Communicative •  Collaborative/Cooperative •  Flexible and willing •  Knowledgeable-multi •  Initiative/responsible •  Responsive Results •  Speed •  Quality Velocity Defects Adherence to plan (many) Overtime Emails Defect Aging Defects not fixed rate
  25. 25. Measuring Quality •  Within each phase, we model to understand and evaluate quality at that phase •  At each phase, define quality and develop quality measures •  The lower level measurements will aggregate to higher cumulative measurements •  Aggregation can be done with various weighting schemes with flexibility according to the business of the organization ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   25  
  26. 26. Don’t – Measure the wrong thing •  Make sure your metrics help you determine if meeting a goal •  Some sample metrics to review: –  Test Coverage = Number of units (KLOC/FP) tested / total size of the system –  Test Density-Number of tests per unit size = Number of test cases per KLOC/FP –  Test cases executed –  Defects per size = Defects detected / system size –  Defects found per tester –  Test cases written per day Quest  2014   26   Don’t measure something just because you can
  27. 27. “Not everything that can be counted counts, and not everything that counts can be counted.” Albert Einstein Don’t measure something just because you can.
  28. 28. DO Be clear about WHY you are measuring.
  29. 29. Why we need to measure ? •  Our bosses want us to… •  They want someone to point fingers at •  They want to fire some people and save money •  They need to report to their managers •  They  want  some  basis  on  which  to  evaluate  us   and  give  us  a  raise!   •  We  need  to  figure  out  a  way  to  do  beWer!   •  We  want  to  improve  our  work  and  improve   soMware  quality   29  Quest  2014  
  30. 30. The Metrics Conundrum •  QA and Testing Language – Defects – Execution status – Test cases – Pass/fail rates – DRE… •  Business  Language   –  Cost  effecAve   –  ROI   –  Cost  of  ownership   –  Cost  of  poor  quality   –  ProducAvity   –  Calls  to  help  desk   –  Customer   saAsfacAon   –  Customer  retenAon   30  Quest  2014  
  31. 31. In your organization… •  What measurements do you take in your organization and why? •  Who uses them and for what? 31  Quest  2014  
  32. 32. The Metric Reality •  Measurement and metrics are like dinner. It takes 2-3 hours to make dinner, and 15 minutes to consume… •  But… many metrics are never reviewed or analyzed (consumed) •  WHY? 32  Quest  2014  
  33. 33. The Metric Conundrum •  Test leads and test managers rarely have the right metrics to show or quantify value •  Metric collection and reporting are a drag •  QA metrics usually focus only on test execution •  Test tools don’t have most of the metrics we want •  Reports generated by QA are only rarely reviewed •  Metrics are not connected to anything of value/ meaningful for ________. 33  Quest  2014  
  34. 34. Don’t – Forget to differentiate between quality, testing and defects •  Metric becomes the goal •  Organizations concentrated on “the metrics”, forget to understand the metric’s relationship to the goal or objective. •  Defect counts need to be incorporated into an overall valuation because Quality is ultimately measured in the eyes of the end user. Quest  2014   34  
  35. 35. Software Quality Evaluation Frameworks Goal Question Metric Why and What Metrics Understanding Software Quality ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   35   Business&Product Processes/Lifecycles Software Dev Processes TesAng   Metrics   Software Testing Processes Software Quality Models Software Quality Processes
  36. 36. Don’t – Forget about context •  Metrics don’t have consistent context so they are unreliable – Context needs to be defined and then maintained for measurements to be meaningful. •  Difficult in today’s environment with changing test platforms and other contextual factors. Quest  2014   36  
  37. 37. What contextual factors could there be? •  Release complexity •  Development methodology •  Software maturity •  Development team maturity and expertise •  Development team and QA integration •  Resources available •  User base Quest  2014   37   All metrics need to be normalized for proper interpretation
  38. 38. Metrics need context to tell the whole story •  Normalized per function point (or per LOC) •  At product delivery (first X months or first year of operation) •  Ongoing (per year of operation) •  By level of severity – Gross numbers don’t tell much •  By category or cause, e.g.: requirements defect, design defect, code defect, documentation/on- line help defect, defect introduced by fixes, etc. – Absolute and Total numbers tell 0 – Too granular also tell 0Quest  2014   38  
  39. 39. Don’t – Be sporadic or irregular •  Measurements used are not consistent – Just as context needs to be consistent, so do the measurements, methods, and time intervals that you collect the measurements and calculate the metrics. •  Just as in weighing yourself, it doesn’t make sense to drink 2 gallons one day and weigh in, and go jogging 10 miles the next day and weigh in. Quest  2014   39  
  40. 40. Measurements Quality •  Two  facets  of  measurement  quality   are  reliability  and  validity.   •  Reliability:  means  how  repeatable  the  results  are  and   low  reliability  indicates  that  many  random  errors  are   occurring  during  measurement.     •  Validity:  means  the  metric  measures  what  it  intended   to  measure.     –  Errors  in  validity,  called  systemaAc  errors,  are  usually   experienced  because  not  all  factors  were  taken  into   account  when  the  measurement  was  designed.   ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   40  
  41. 41. Metrics Abandoned for Many Reasons •  How many of you have had these problems? – That measurement is not valid because… – That measurement is not reliable because… ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   41   If no one believes (and therefore takes improvement actions) your metrics, then why bother?
  42. 42. Do Understand Metric Terms •  What is a metric? •  What is a measurement? •  What is an indicator? •  What’s the difference? 42  ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.  
  43. 43. Measures Versus Metrics Measures   •  To  be  executed   •  Executed   •  Passed   •  Failed   •  Re-­‐executed   •  Total  ExecuAons   •  Total  Passes   •  Total  Failures   ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   43   Metrics •  TC % complete •  TC % passed •  % failures •  % defects corrected •  % bad fixes •  % defect discovery
  44. 44. Don’t – Collect measurements that no one wants •  Metrics have no audience – As a corollary to the previous factor, if there is no question to be answered, then there will also be no audience for the metric. •  Metrics need to have an audience in order to have meaning. •  How many of the metrics and reports that you generate are read? Quest  2014   44  
  45. 45. Poll: How many of you collect metrics that you don’t need or use?
  46. 46. Group  Exercise   Who  are  your  stakeholders?   •  CEO/CTO/CIO   •  Development  manager/VP   •  QA  manager   •  Others?   Quest  2014   46  
  47. 47. Do - Collect what “they” want •  Ratios and percentages rather than absolutes •  Comparisons over time, or by release •  Report on what leads to improvement in: – Costs – Time – Quality Quest  2014   47  
  48. 48. Do - Collect what they want Costs  (Some  potenAal  metrics  include):     •  Business  losses  per  defect  that  occurs  during  operaAon   •  Business  interrupAon  costs   •  Costs  of  work-­‐arounds   •  Costs  of  reviews,  inspecAons  and  prevenAve  measures   •  Costs  of  test  planning  and  preparaAon   •  Costs  of  test  execuAon,  defect  tracking,  version  and  change  control   •  Costs  of  test  tools  and  tool  support   •  Costs  of  test  case  library  maintenance   •  Costs  of  tesAng  &  QA  educaAon  associated  with  the  product   •  Re-­‐work  effort  (hours,  as  a  percentage  of  the  original  coding  hours)   •  Lost  sales  or  goodwill   •  Annual  QA  and  tesAng  cost  (per  funcAon  point)   Quest  2014   48  
  49. 49. Do - Collect what they want Time-­‐Resources  (Some  potenAal  metrics   include):   •  Labor  hours/defect  fix   •  Turnaround  Ame  for  defect  fixes,  by  level  of   severity   •  Time  for  minor  vs.  major  enhancements   – actual  vs.  planned  elapsed  Ame   •  Effort  for  minor  vs.  major  enhancements   •  actual  vs.  planned  effort  hours   Quest  2014   49  
  50. 50. Do - Collect what they want Quality  (Some  potenAal  metrics  include): •  Survey before, after (and ongoing) product delivery •  # system enhancement requests per year •  # maintenance fix requests per year •  User problems: call volume to customer service/Tech support •  User Satisfaction –  training time per new user, time to reach task time of x –  # errors per new user •  # product recalls or fix/patch releases/year •  # production re-runs •  Availability  (Ame  system  is  available/  Ame  the  system  is  needed  to  be   available) Quest  2014   50  
  51. 51. Do Collect what they want •  Show them in combination and relative to each other – Cost vs. quality – Cost vs. time – Quality vs. time Quest  2014   51  
  52. 52. Don’t – Make the collection effort the end game •  Measurements are too hard to get – If you end up designing the right metric to answer the right question, it doesn’t matter if it takes several man days to get the data and do the calculations. •  Unless the value and decisions made from these metrics have considerable value, they’ll soon be abandoned. Quest  2014   52  
  53. 53. Poll: How many of you started to collect metrics but then found it was too difficult or time consuming and quit?
  54. 54. Don’t – Forget indicators •  Metrics have no indicators so cannot evaluate – You collect mounds of data but then what? – How do you determine what is ‘good’ or ‘bad’? – Before you get started collecting and calculating you need to put together a way to evaluate the numbers you get with meaningful indicators that can be used as benchmarks as your metrics program matures. Quest  2014   54  
  55. 55. Indicators   •  Indicators  change  numbers  into  meaning   •  What  does  68%  mean?   ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   55  
  56. 56. Don’t Mix Up Causality •  The  cause  and  effect  relaAonship  has  three  requirements:   –  cause  precedes  effect  in  Ame  or  by  logic.   –  shown  cause  and  effect  relaAonship   –  direct  constant  relaAonship   •  When  the  relaAonship  between  variables  can  be  graphed  and  a  line  of  best  fit  can  be  found   •  Be  careful  of  extreme  and  inaccurate  data  values.   •  When  defining  a  casual  relaAonship  be  sure  to  watch  for  misguided  interpreted   relaAonships.            CASE  1                  Z  causes  changes  in  X                  Z  causes  changes  in  Y                  X  is  perceived  to  cause  changes  in  Y  or  vice  versa  and  Z  is  overlooked            CASE  2                  X  causes  changes  in  Z                  Z  causes  changes  in  Y                  X  is  perceived  to  cause  changes  in  Y  and  Z  is  overlooked   ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   56  
  57. 57. Indictors and Interpretation ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   57  
  58. 58. Conclusions •  Designing and implementing a software quality metrics program requires more than defect analysis. •  Think - who and what questions that you want to answer or goals of using metrics. – Many refer to this as the goal-question-metric paradigm (GQM) – In simple terms, what are you going to do with the numbers once you get them? •  Most of the “Don’ts” are related to not thinking about the objectives of the metrics and actions you will take based on them.Quest  2014   58  
  59. 59. Solutions •  Develop a stakeholders list and their goals- objectives •  Develop a list of questions that, if answered, would determine that the goals are met •  Develop a catalogue of metrics (that answer the questions) that can mix and match to apply to the goals depending on the stakeholder •  Develop and collect metrics that accompany each part of the development process, not just testing – There are many “defects” not directly in dev. Quest  2014   59  
  60. 60. Measurement   Framework   Improvement   Decisions   Stakeholders   60  Quest  2014   Do Develop a Metrics Framework
  61. 61. SoMware  Quality  Dashboard   (model)   Process SoMware   Quality   Product UsersPeople Quality of the people Quality of the stakeholders goals and objectives (GQM) Quality Processes to implement the goals And the technology to support them Viewpoint and measurement of quality from the end user If the ‘people’ are connected well to end users, this should also be high quality; with incorporation of context. High quality people, with high quality processes should produce repeatable high quality product From both internal and external quality views ©  2014  XBOSoM,  Inc.-­‐  All  Rights  Reserved.   61  
  62. 62. Thanks Q&A @xbosoft 408-350-0508 Philip Lew @philiplew White Papers: /xbosoft Blog: /xbosoft