Software Quality Metrics For Testers
Why and What to Measure
Meet Your Instructor
•  Phil Lew
–  Telecommunications consultant and network designer
–  Architect, Data warehousing prod...
Session Spirit and
Expectations
•  Interactive
•  I won’t read the slides…
•  Slides for you as a take-away
•  Different t...
Understand, Evaluate and
Improve
•  If our end goal is improvement in what?
•  To improve, we need to evaluate
•  In order...
Metrics in real life
Food	
  Eaten	
   Weight	
   Performance	
   Race	
  Results	
  
STARWEST	
  2013	
  
5	
  
•  Calori...
Metrics - Benefits
•  Understand how QA, testing, and its processes
and where the problems are
•  Evaluate the process and...
How can measurement help us (YOU)
•  Create a organizational memory – baselines of current
practices-situation
•  Determin...
All this is good
But in reality…
Why we need to measure ?
•  Our bosses want us to…
•  They want someone to point fingers at
•  They want to fire some peop...
The Metric Conundrum
•  QA and Testing
Language
–  Defects
–  Execution status
–  Test cases
–  Pass/fail rates
–  DRE…
• ...
In your organization…?
•  What measurements do you take in your
organization and why?
•  Who uses them and for what?
STARW...
The Metric Reality
•  Measurement is like dinner. It takes 2-3 hours to
make dinner, and 15 minutes to consume…
•  But… ma...
The Metric Conundrum (cont.)
•  Test leads and test managers rarely have the
right metrics to show or quantify value
•  Me...
Let’s solve this!
Outline
•  Choose the right metrics
•  Design a measurement framework
•  Report what is meaningful
•  Use metrics not only...
Guiding principles about
metrics
•  Metrics can represent a snapshot
•  Metrics can change over time
•  Metrics can be sim...
Metrics can represent a
snapshot
•  Not only at a particular moment, like this week,
but for a process
STARWEST	
  2013	
 ...
Metric Basics
•  What is a metric?
•  What is a measurement?
•  What is an indicator?
STARWEST	
  2013	
   18	
  
Some Metrics – Let’s get
started
•  How many of
these do you
know/use?
•  What others do
you use?
•  What are they
used fo...
Metrics Exercise
Work in teams of 2-3
List the metrics you use, what you use them
for, who wants them, what decisions they...
Developing a Measurement
Framework
Stakeholders	
  
Measurement	
  
IdenRficaRon	
  
Measurement	
  
Report	
  Design	
  
M...
Stakeholders
•  Who are they?
•  What are their objectives?
•  What questions do they have?
•  What answers can metrics pr...
Possible Stakeholders
Stakeholder	
   Objec0ve	
   Ques0on	
   Answer	
  
CIO	
  
CEO	
  
VP	
  –	
  QA	
  –	
  
Developme...
Metric Q&A Worksheet
Category	
   Ques0on	
   Metric	
   Tool	
  to	
  Use	
   Needed	
  
Customer	
  
SaRsfacRon	
  
Are	...
Stakeholder Workshop (15
minutes)
•  Divide up into teams of 2-3
•  List out stakeholders
•  Brainstorm stakeholder questi...
Organizing our thoughts
•  Putting metrics into a framework
•  Helps us to make sure we have all functions in
our organiza...
Identify Metrics to Answer
Questions
Customer	
  and	
  
Management	
  
Metrics	
  
OperaRons	
  
OpRmizaRon	
  
Metric	
 ...
Measurement and Reporting
•  Measure, build metrics AND develop reports that
answer the questions of your stakeholders
Pla...
Test Measurement and
Reporting
•  Measure, build metrics AND develop reports that
answer the questions of your stakeholder...
Test Measurement and
Reporting
•  Measure, build metrics AND develop reports that
answer the questions of your stakeholder...
Test Measurement and
Reporting
•  Measure, build metrics AND develop reports that
answer the questions of your stakeholder...
Test Measurement and
Reporting
•  Measure, build metrics AND develop reports that
answer the questions of your stakeholder...
What’s Missing?
STARWEST	
  2013	
   33	
  
Requirem
ents	
  
stability	
  
Internal	
  
Quality	
  
Defects	
  
Automa
Ro...
Data and Reporting
OperaRons	
  
OpRmizaRon	
  
Metric	
  
CalculaRon	
  
Customer	
  and	
  
Management	
  
Metrics	
  
D...
Exercise: Develop Your Own
Framework
STARWEST	
  2013	
   35	
  
Work backwards or forwards
Work in small groups 2-3
Reporting
•  Work with stakeholders to report
information so that they can make
decisions
•  Metric is meaningless if not ...
Reporting Root Causes and
Trends
•  The most important metrics you can report
–  Root causes
•  Sources of defects
•  Sour...
Reporting – Show Value
•  Answer Questions
–  Are we delivering a quality product?
–  Are our customers satisfied?
–  Are ...
Show Value – Connect the dots
–  Did we understand
requirements correctly and
timely?
–  What are the primary causes of
de...
Getting Started with
Measurement
•  What questions-answers are important to your
organization?
– Develop a model/determine...
Set up an action plan
•  Define a metrics improvement plan – What are
your goals?
•  Conduct stakeholder workshops – Invol...
What’s Next
•  Choose a project – start small and build up
for early wins
•  Organize to support the change
– Committed pe...
Software Testing Metrics
Take Aways
•  Quality is different for each organization
–  No one right answer for what metrics
...
Putting It All Together
Measurement	
  
Framework	
   Improvement	
  
Decisions	
  
Stakeholders	
  
STARWEST	
  2013	
   ...
Thanks
Please fill out an evaluation form
and drop it in the collection basket
located at the back of the room.
www.xbosof...
Upcoming SlideShare
Loading in...5
×

Software Quality Metrics for Testers - StarWest 2013

1,546

Published on

Presentation by Phil Lew at StarWest 2013.
When implementing software quality metrics, we need to first understand the purpose of the metrics and who will be using them. Will the metric be used to measure people or the process, to illustrate the level of quality in software products, or to drive toward a specific objective? QA managers typically want to deliver productivity metrics to management but management may want to see metrics that describe customer or user satisfaction. Philip Lew believes that software quality metrics without actionable objectives toward increasing customer satisfaction are a waste of time. Learn how to connect each metric with potential actions based on evaluating the metric. Metrics for the sake of information may be helpful but often just end up in spreadsheets of interest to no one. Take home methods to identify metrics that support actionable objectives. Once the metrics and their objectives have been established, learn how to define and use metrics for real improvement.

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,546
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
90
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Software Quality Metrics for Testers - StarWest 2013

  1. 1. Software Quality Metrics For Testers Why and What to Measure
  2. 2. Meet Your Instructor •  Phil Lew –  Telecommunications consultant and network designer –  Architect, Data warehousing product development –  Software product manager, BI product –  COO, large IT services company –  CEO, XBOSoft, software qa and testing services •  Relevant specialties –  Software quality process improvement –  Software quality evaluation and measurement –  Software quality in use / UX design STARWEST  2013   2  
  3. 3. Session Spirit and Expectations •  Interactive •  I won’t read the slides… •  Slides for you as a take-away •  Different than your handouts – Always thinking of new examples and ideas, you can write me and I’ll send these to you STARWEST  2013   3  
  4. 4. Understand, Evaluate and Improve •  If our end goal is improvement in what? •  To improve, we need to evaluate •  In order to evaluate, we must understand what we are evaluating •  To do this… We need metrics Can you think of other examples in our lives where this applies? Where do you use metrics to evaluate and improve? STARWEST  2013   4  
  5. 5. Metrics in real life Food  Eaten   Weight   Performance   Race  Results   STARWEST  2013   5   •  Calories •  Fat •  Carbohydrates •  Protein •  Time of day •  Vitamins •  … •  Blood pressure •  Cholesterol •  Blood glucose •  Red cell count •  White cell count •  Hematocrit •  Hemoglobin •  Body fat % •  … •  Placing •  … •  Effort/Power •  Heart rate/Watts •  Speed •  Time Intelligence Finesse Context •  Training •  Sleep
  6. 6. Metrics - Benefits •  Understand how QA, testing, and its processes and where the problems are •  Evaluate the process and the product in the right context •  Predict and control process and product qualities •  Reuse successful experiences –  Feed back experience to current and future projects •  Monitor how something is performing •  Analyze the information provided to drive improvements STARWEST  2013   6  
  7. 7. How can measurement help us (YOU) •  Create a organizational memory – baselines of current practices-situation •  Determine strengths and weaknesses of the current process and product –  What types of errors are most common? •  Develop reasoning for adopting/refining techniques –  What techniques will minimize the problems? •  Assess the impact of techniques –  Does more/less functional testing reduce defects? •  Evaluate the quality of the process/product –  Are we applying inspections appropriately? –  What is the reliability of the product before/after delivery? STARWEST  2013   7  
  8. 8. All this is good But in reality…
  9. 9. Why we need to measure ? •  Our bosses want us to… •  They want someone to point fingers at •  They want to fire some people and save money •  They need to report to their managers •  They  want  some  basis  on  which  to  evaluate  us   and  give  us  a  raise!   •  We  need  to  figure  out  a  way  to  do  beHer!   •  We  want  to  improve  our  work  and  improve   soKware  quality   STARWEST  2013   9  
  10. 10. The Metric Conundrum •  QA and Testing Language –  Defects –  Execution status –  Test cases –  Pass/fail rates –  DRE… •  Business  Language   –  Cost  effecRve   –  ROI   –  Cost  of  ownership   –  Cost  of  poor  quality   –  ProducRvity   –  Calls  to  help  desk   –  Customer   saRsfacRon   –  Customer  retenRon   STARWEST  2013   10  
  11. 11. In your organization…? •  What measurements do you take in your organization and why? •  Who uses them and for what? STARWEST  2013   11  
  12. 12. The Metric Reality •  Measurement is like dinner. It takes 2-3 hours to make dinner, and 15 minutes to consume… •  But… many metrics are never reviewed or analyzed (consumed) •  WHY? STARWEST  2013   12  
  13. 13. The Metric Conundrum (cont.) •  Test leads and test managers rarely have the right metrics to show or quantify value •  Metric collection and reporting are a drag •  QA metrics usually focus only on test execution •  Test tools don’t have most of the metrics we want •  Reports generated by QA are only rarely reviewed •  Metrics are not connected to anything of value/ meaningful for ________. STARWEST  2013   13  
  14. 14. Let’s solve this!
  15. 15. Outline •  Choose the right metrics •  Design a measurement framework •  Report what is meaningful •  Use metrics not only to manage testing, but as a Quality barometer •  Show value STARWEST  2013   15  
  16. 16. Guiding principles about metrics •  Metrics can represent a snapshot •  Metrics can change over time •  Metrics can be simple – don’t have to be complicated •  Metrics can assist us in making decisions today •  Metrics can assist us in predicting the future •  Some metrics are critical for some while useless for others STARWEST  2013   16  
  17. 17. Metrics can represent a snapshot •  Not only at a particular moment, like this week, but for a process STARWEST  2013   17   Code   quality   Internal   Quality   Tested   in  a   lab   External   Quality   •  What  the   end  user   sees   •  What  the   end  user   experiences   In-­‐Use   Quality   At what stage in the process do you take measurements?
  18. 18. Metric Basics •  What is a metric? •  What is a measurement? •  What is an indicator? STARWEST  2013   18  
  19. 19. Some Metrics – Let’s get started •  How many of these do you know/use? •  What others do you use? •  What are they used for? •  By who? Schedule   variance   Effort   variance   Cost   variance   Defect   removal   effecRveness   ProducRvity   Defect   aging   CriRcal   Defect   rate   Test   cases   executed   Test   coverage   Pass/Fail   Rate   ROI   AutomaRon   coverage   STARWEST  2013   19  
  20. 20. Metrics Exercise Work in teams of 2-3 List the metrics you use, what you use them for, who wants them, what decisions they make based on them STARWEST 2013 20
  21. 21. Developing a Measurement Framework Stakeholders   Measurement   IdenRficaRon   Measurement   Report  Design   Measurement  Framework   Data  CollecRon  Methods  and  Sources   STARWEST  2013   21  
  22. 22. Stakeholders •  Who are they? •  What are their objectives? •  What questions do they have? •  What answers can metrics provide? STARWEST  2013   22  
  23. 23. Possible Stakeholders Stakeholder   Objec0ve   Ques0on   Answer   CIO   CEO   VP  –  QA  –   Development   Director  -­‐   Development   Owner   Auditor   Test  Lead   Partner   ?   STARWEST  2013   23  
  24. 24. Metric Q&A Worksheet Category   Ques0on   Metric   Tool  to  Use   Needed   Customer   SaRsfacRon   Are  we  meeRng   schedule?   Schedule   Variance   Project   Management   Tool   None   SoKware   TesRng   Efficiency   SoKware   Quality   What  kinds  of   defects  are  most   common?   Usability   Are  end  users   gedng  lost,   understand  our  UI   ?   ?   •  QuesRons  are  the  fundamental  building  block   •  Answering  quesRons  that  lead  to  decisions  à  VALUE  24  
  25. 25. Stakeholder Workshop (15 minutes) •  Divide up into teams of 2-3 •  List out stakeholders •  Brainstorm stakeholder questions (2 per stakeholder) •  Determine answers and metrics to that question •  Group your metrics into categories •  Present and discuss to the group STARWEST  2013   25  
  26. 26. Organizing our thoughts •  Putting metrics into a framework •  Helps us to make sure we have all functions in our organization and process covered •  Helps us to develop relationships between different elements in our process STARWEST  2013   26  
  27. 27. Identify Metrics to Answer Questions Customer  and   Management   Metrics   OperaRons   OpRmizaRon   Metric   CalculaRon   Departmental   Metrics   Project/Build   Metrics   Planning   Quality   Status   Data   CollecRon   Metric Identification Req’ts. & Questions Reporting Efficiency  and   EffecRveness   Schedule  and   Budget   Defect  types  and   quanRRes   Resource   uRlizaRon   Snapshots  and   trends   STARWEST  2013   27  
  28. 28. Measurement and Reporting •  Measure, build metrics AND develop reports that answer the questions of your stakeholders Planning   •  Requirements  coverage  %   •  Test  case  prioriRes  and   complexity/atomicity   •  Requirements  traceability   •  Test  ExecuRon  prioriRzaRon   •  Requirements  aging   •  Resource  and  budget  variance   STARWEST  2013   28  
  29. 29. Test Measurement and Reporting •  Measure, build metrics AND develop reports that answer the questions of your stakeholders Quality   •  Test  pass/fail  rate   •  #  open  defects   •  #  priority  1  defects  open   •  #  priority  1  defects  found   •  Re-­‐opened  defects   •  Defect  density   •  Defects  found  in  operaRon   •  Defects  found/test  case/script   •  Test  execuRon  progress   STARWEST  2013   29   •  Customer  saRsfacRon   •  Help  desk  calls  
  30. 30. Test Measurement and Reporting •  Measure, build metrics AND develop reports that answer the questions of your stakeholders OperaRons   •  Time  spent  gedng  environment   ready   •  Time  to  close  defects  –  aging   •  Effort  per  test  case   •  TesRng  schedule  variance   •  Defect  arrival  and  closure  rate   •  Test  schedule  variance   STARWEST  2013   30  
  31. 31. Test Measurement and Reporting •  Measure, build metrics AND develop reports that answer the questions of your stakeholders OpRmizaRon   •  AutomaRon  script  execuRon   Rme   •  Regression  Rme   •  AutomaRon  coverage   •  Resource  efficiency   •  Resource  effecRveness   •  Test  cycle  Rme   STARWEST  2013   31  
  32. 32. Test Measurement and Reporting •  Measure, build metrics AND develop reports that answer the questions of your stakeholders Status   •  Test  execuRon  %  complete   •  Test  case  prioriRes  and  complexity/ atomicity   •  Requirements  status/trending   •  Test  ExecuRon  EffecRveness   •  Over  Rme  (trending)  versus  snapshot   STARWEST  2013   32  
  33. 33. What’s Missing? STARWEST  2013   33   Requirem ents   stability   Internal   Quality   Defects   Automa Ron   coverage   External   Quality   Customer   retenRon   Help  desk  call   rate   In-­‐Use   Quality   Can you think of where other metrics we have listed out before, and where they would fit?
  34. 34. Data and Reporting OperaRons   OpRmizaRon   Metric   CalculaRon   Customer  and   Management   Metrics   Departmental   Metrics   Project/Build   Metrics   Planning   Quality   Status   Automated   Data   CollecRon   All for nothing unless you have supporting data Config  mgmt   Rqmt  mgmt   Time   tracking   Test  mgmt   Project   planning   Other?   STARWEST  2013   34  
  35. 35. Exercise: Develop Your Own Framework STARWEST  2013   35   Work backwards or forwards Work in small groups 2-3
  36. 36. Reporting •  Work with stakeholders to report information so that they can make decisions •  Metric is meaningless if not understood or misunderstood •  Develop a report catalog – Metric library – Canned reports usually have limited impact STARWEST  2013   36  
  37. 37. Reporting Root Causes and Trends •  The most important metrics you can report –  Root causes •  Sources of defects •  Sources of mistakes •  Sources of delay or inefficiency •  Sources of improvement –  Trending •  Showing metrics over time, pre-set intervals •  Showing change •  Make sure to differentiate Snapshot versus trend STARWEST  2013   37  
  38. 38. Reporting – Show Value •  Answer Questions –  Are we delivering a quality product? –  Are our customers satisfied? –  Are we meeting timelines? –  Did we understand requirements correctly and timely? –  How much is QA costing? –  Are we meeting budget? –  How have we cut costs? –  How have we improved efficiency? –  What are the primary causes of delay? –  What are the causes of poor quality? –  Is quality improving? –  What is the status? – Current versus goal? –  Have we gotten better? STARWEST  2013   38  
  39. 39. Show Value – Connect the dots –  Did we understand requirements correctly and timely? –  What are the primary causes of delay? –  What are the causes of poor quality? STARWEST  2013   39   Requirem ents   stability   Internal   Quality   Defects   Automa Ron   coverage   External   Quality   Customer   retenRon   Help  desk  call   rate   In-­‐Use   Quality   QA and Testing Language Business Language –  Is quality improving? –  What is the status? – Current versus goal? –  Are we delivering a quality product? –  Are our customers satisfied? –  Are we meeting timelines? –  How much is QA costing? –  Are we meeting budget and schedule? –  How have we improved efficiency? –  What are the primary causes of delay? –  What are the causes of poor quality?
  40. 40. Getting Started with Measurement •  What questions-answers are important to your organization? – Develop a model/determine what to measure – Define improvement objectives •  Based on intuition/available data •  Look at high payoff areas, problem areas •  What data can you collect/Which technique can you use – Maybe some elements of the model drop out- can’t be measured that easily STARWEST  2013   40  
  41. 41. Set up an action plan •  Define a metrics improvement plan – What are your goals? •  Conduct stakeholder workshops – Involve all •  Build your measurement framework with phases in the process and categories – Q&A catalog – Metric catalog – Report catalog •  Set up the infrastructure and operationalize •  Start collecting and developing benchmarks STARWEST  2013   41  
  42. 42. What’s Next •  Choose a project – start small and build up for early wins •  Organize to support the change – Committed people – Recognize the new processes – Assign roles and resources STARWEST  2013   42  
  43. 43. Software Testing Metrics Take Aways •  Quality is different for each organization –  No one right answer for what metrics –  Different metrics have different meanings for different people – SET UP FRAMEWORK AND PLAN FOR YOUR Company •  Management wants metrics –  Know what key questions need to be answered –  REPORTING - Metrics programs often fail due to poor articulation •  Measurement and improvement is a long term effort, just like a diet. STARWEST  2013   43  
  44. 44. Putting It All Together Measurement   Framework   Improvement   Decisions   Stakeholders   STARWEST  2013   44   Questions and Answers
  45. 45. Thanks Please fill out an evaluation form and drop it in the collection basket located at the back of the room. www.xbosoft.com @xbosoft 408-350-0508 Philip Lew @philiplew philip.lew@xbosoft.com Some resources: http://www.xbosoft.com/knowledge_center/ www.facebook.com/xbosoft
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×