How Metrics Programs Can Destroy Your Soul


Published on

Testers are often evaluated by metrics that don’t really quantify the value of their work. Metrics such as tests planned, tests executed, coverage achieved, and defects reported all measure effort rather than results. Since people generally want to meet metrics goals, measurements that focus on activity rather than effectiveness often encourage unintended behaviors. Since the true value of testers lies in their ability to analyze and communicate risks and impacts, we must change the focus of metrics from numbers to insights. Scott Barber shares what stakeholders are really looking for when they request specific metrics, how the metrics they request frequently fail, and how to help your organization create metrics that do provide real insight. Discover the tools you need to explain what can be measured, what those measurements mean, and how to combine measurements into metrics that tell insightful stories about your testing.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

How Metrics Programs Can Destroy Your Soul

  1. 1.       nt Session    Presented by:  Scott        Brought to you by:      340 Corporate Way, Suite   Orange Park, FL 32073  888‐2 W8  Concurre 4/9/2014    12:45 PM          “How Metrics Programs Can Destroy Your Soul”      Barber SmartBear              300, 68‐8770 ∙ 904‐278‐0524 ∙ ∙ 
  2. 2. Scott Barber SmartBear   Chief performance evangelist for SmartBear Scott Barber is a respected leader in the advancement of software testing practices, an industry activist, and load testing celebrity of sorts. Scott authored several books―Performance Testing Guidance for Web Applications, Beautiful Testing, How to Reduce the Cost of Testing, and Web Load Testing for Dummies―and more than 100 articles and blog posts. Founder/president of PerfTestPlus, Scott co-founded the WOPR, served as director of the AST and CMG, and is a founding member of ISST. His industry writing, speaking, and activism focus on improving the effectiveness and business alignment of software development practices. Learn more about Scott Barber.
  3. 3. 2/25/2014 1 How Metrics Programs Can Destroy Your Soul Created for: How Metrics Programs  Can Destroy Your Soul By: Scott Barber Product Owner – Load Testing Tools SmartBear Software, Inc.
  4. 4. 2/25/2014 2 Product Owner – Load Testing Tools SmartBear Software,  Inc.  scott barber@smartbear @sbarber Co‐Founder: Workshop On Performance and Reliability www.performance‐ Co‐Author:Author: Contributing Author: Books: About me: Have valid needs to:  • Assess & compare performance & quality of: di id l Managers & Execs • Individuals • Teams • Projects • Products • Compute costs • Identify risks & trends
  5. 5. 2/25/2014 3 Commonly satisfy those needs via:  • Instinct • Not Scalable Managers & Execs • Not Scalable • Non‐transferable • Qualitative Metrics • “Group Gut” • Inconsistent i i i• Quantitative Metrics • Scalable • Transferable Commonly satisfy those needs via:  • Instinct • Not Scalable Managers & Execs • Tied to individuals C ’ b i d• Not Scalable • Non‐transferable • Qualitative Metrics • “Group Gut” • Inconsistent i i i • Can’t be trained • Predicated on trust  (not data) • Virtually impossible to  defend • Generally reserved for  “hands on” managers • Quantitative Metrics • Scalable • Transferable hands‐on  managers  with one project or  team
  6. 6. 2/25/2014 4 Commonly satisfy those needs via:  • Instinct • Not Scalable Managers & Execs • Very difficult to do  ll• Not Scalable • Non‐transferable • Qualitative Metrics • “Group Gut” • Inconsistent i i i well • Expensive • Time consuming • Not good for detailed  comparisons • Not good for  estimation • Quantitative Metrics • Scalable • Transferable estimation • Few managers/execs  trained in using them Commonly satisfy those needs via:  • Instinct • Not Scalable Managers & Execs • Scale well P id i li i &• Not Scalable • Non‐transferable • Qualitative Metrics • “Group Gut” • Inconsistent i i i • Provide simplicity &  consistency • Easy to use • Good for detailed  comparisons • Good for estimation • Good for trends • Quantitative Metrics • Scalable • Transferable • Good for trends • Most managers/Execs  trained  • Validated in many  areas over long  periods of time
  7. 7. 2/25/2014 5 Are based on measurements that are:  • Verifiable St d di d Effective Metrics • Standardized Are explicitly linked to goals  • Correlated/causal • Compliance F t d b l tFavor trends over absolutes Evolve Are not used in isolation  “Most metrics are created, collected, and reported to satisfy a leader’s request. The leader’s role is to supply clarity and direction by providing the proper questions. Middle management’s role is to th ti M t i ff f idi th What is a Metric? answer the questions. Metrics offer a means of providing the answers so that all involved can have faith in them. Unfortunately, leaders often don’t know exactly what they want. Chances are you have played the Guessing Game with a leader, where the data you provided wasn’t what he needed, so he asked for different data, figuring he would know the right data when he saw it Despite repeated failures you continued to chase data as ifsaw it. Despite repeated failures, you continued to chase data as if all the effort invested in collecting the wrong data would eventually prove worth your perseverance.” From: “Do-It-Yourself Metrics” by Martin Klubeck, Michael Langthorne, and Don Padgett
  8. 8. 2/25/2014 6 Provide insight into the health of:  • Individuals • Teams Metrics Should • Teams • Projects • Products Help to: • Assess efficiency & progress • Inform business decisions  • Enhance communication • Maintain focus  However; Even in well studied &  historically validated areas,  quantitative metrics are…quantitative metrics are…
  9. 9. 2/25/2014 7 Quantitative Metrics:  • Obscure Details ll & i Far From Perfect • Roll‐ups & summaries • Focus on “big picture” • Risk Measurement Dysfunction • People tend to optimize metrics • Metrics optimization invites “bad stuff” • Are “invitations to conversations” • Don’t tell the whole story in isolation • Can be wildly misleading … and… Common quantitativeCommon quantitative  management metrics  models are not a good fit  for software development
  10. 10. 2/25/2014 8 … and… Within software  development, testing is the  *worst* fit for common  quantitative managementquantitative management  metrics models Inconsistent Units: • Test/Test Cases Why? • Size/Importance of defects Unknown Unknowns: • How many defects are there? • How many tests would it take to…? Quality is Subjective: • How good is “good enough”? • What do Pass/Fail *mean*?
  11. 11. 2/25/2014 9 Test/Test Cases Planned? Test/Test Cases Executed? These metrics indicate… ? Defects Found? Defect Removal Rate? Coverage? Test/Test Cases Planned? Test/Test Cases Executed? These metrics encourage… ? Defects Found? Defect Removal Rate? Coverage?
  12. 12. 2/25/2014 10 Communicate impacts to revenue: • Help the CEO get a bigger yacht, sooner. What do we *want* to do? Identify risks to the business: • Generally revenue centric. Identify risks to the buyers/users: • If they don’t want to buy, we lose. Streamline Development: • Get to “good enough” faster. Do your test metrics help with… • Risks to revenue? Test Metrics Effectiveness • Risks to the business? • Risks to the buyers/users? • Increasing delivery velocity?
  13. 13. 2/25/2014 11 Change Your Perspective Risk as a common language Risk Security FunctionalCompliance Whether explicitly or implicitly, all forms of testing revolve around the reduction and PerformanceUsability reduction and management of risk. © 2011 PerfTestPlus, Inc. All rights reserved. The Secret to Managing Risk To effectively manage risk, you must effectively manage knowledge. © 2011 PerfTestPlus, Inc. All rights reserved.
  14. 14. 2/25/2014 12 Control Model Testing Control Model Testing is a business-a business aligned approach to software testing that derives “test cases” from knowledge models of the system © 2011 PerfTestPlus, Inc. All rights reserved. based on a risk-based taxonomy. Basic Entities © 2011 PerfTestPlus, Inc. All rights reserved.
  15. 15. 2/25/2014 13 Risk Assessment Framework The Open Group ( Risk Taxonomy Technical Standard - Testing For Threats & Risk Controls prevent or mitigate risk whichmitigate risk which may impact business objectives. Control Model Testing helps identify © 2011 PerfTestPlus, Inc. All rights reserved. Testing helps identify and assess these controls.
  16. 16. 2/25/2014 14 Possibly Better Test Metrics Risks/Threats Identified? Risk/Threat Tests Planned/Executed? Risks/Threats Dismissed? Risks/Threats Uncontrolled/Unmitigated? Risks/Threats Controlled/Mitigated? Questions?
  17. 17. 2/25/2014 15 Contact Info about me/scott Product Owner – Load Testing Tools SmartBear Softwear, Inc. E-mail: Scott barber@smartbear com Web Site: Blog: Twitter: @sbarber