Download as PDF, PPTX
Uploaded byXBOSoft
Software Quality Metrics Do's and Don'ts - QAI-Quest 1 Hour Presentation
This document outlines an instructor-led presentation on software quality metrics. It introduces the instructor and his relevant experience. It then provides information from attendee feedback on previous sessions, including both positive and negative comments. The presentation agenda is then outlined, with topics like how measurement can help organizations understand, evaluate, and improve their processes. Group exercises are included to discuss defining software quality and examples of metrics used in real life. The presentation also covers best practices for using metrics, different models of software quality, and measuring quality within agile development.
More Related Content
Similar to Software Quality Metrics Do's and Don'ts - QAI-Quest 1 Hour Presentation
Software Quality Metrics Do's and Don'ts - QAI-Quest 1 Hour Presentation
- 1.
- 2. "Software Quality Metrics Do’sand Don’ts" Philip Lew
- 3. Meet Your Instructor • Team Lead, Data warehousing product development • Software product manager, BI product • COO, large IT services company • CEO, XBOSoft, software qa and testing services • Relevant specialties – Software quality process improvement – Software quality evaluation and measurement – Software quality in use / UX design – Mobile User Experience and usability Quest 2014 3
- 4. Let’s Meet Each Other • Name • Your Company/Role • InteresAng Adbit or fact • ObjecAve in being here © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 4
- 5. Some InformaAon • Used 1 hour efficiently and covered the material. It was interacAve too. • Presenter was comfortable and engaging. Content was good. • Very interacAve and demanding, commands great aWenAon. • Best content. Provided great brainstorming opportuniAes and gave examples. Great presenter. Quest 2014 5
- 6. More informaAon • Slides too low where people’s heads are. Can’t see. • How do we measure quality? You didn’t answer that at all. Very disappointed. • I realize interacAon is good, but found the discussions to be distracAng. • It was too hard to hear what was said. • Instructor only focused on people in front who’s hands were up. • Too large a group. • Slides were different. Quest 2014 6
- 7. Session Spirit andExpectations • Interactive given time and other context • I won’t read the slides… – Slides for you as a take-away – Slightly different than your handouts – Some new examples and ideas – Write me and I’ll send these to you • I’ll repeat questions (if I can remember) 7 Quest 2014
- 8.
- 9. Motivation Why is QAMeasurement Important • Quality Assurance measurement provides informaAon to answer quesAons about your tesAng, your team, and your company. – Is your QA team improving? – Have you reached the highest state of efficacy? – Is your tesAng as thorough as it can be? – Is soMware ready for release? Can we stop tesAng now? – Are you able to do everything you want within budget? – What parts of the soMware are at most risk ? • If the answer to any one of those quesAons is “no”, or “I don’t know”, then you are in the right place. © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 9
- 10. Meal of Metrics Schedule variance Effort variance Cost variance Defect removal effecAveness ProducAvity Defect aging CriAcal Defect rate Test cases executed Test coverage Pass/Fail Rate ROI AutomaAon coverage 10 Value Provided % Defects/ hour Customer saAsfacAon Avg. Time to Fix Recurrence Rate Code complexity Test Density Defect Find Rate Defect Arrival Rate Require ments VolaAlity Require ments Ambiguity Planned /Actual OverAme Rate CompleAon Rate Velocity ExecuAon Rate © 2014 XBOSoM, Inc.-‐ All Rights Reserved.
- 11. Don’t – Calculatemetrics that don’t help answer specific questions • Are you collecting measurements and calculating metrics without thinking what answers will I get after collecting this information? Quest 2014 11
- 12. POLL: How manymetrics are you collecting on a regular basis within your organization?A. 1-5 B. 6-10 C. 11-15 D. 0
- 13. Metrics - Benefits • Understand how QA, tesAng, and its processes and where the problems are • Evaluate the process and the product in the right context • Predict and control process and product qualiAes • Reuse successful experiences – Feed back experience to current and future projects • Monitor how something is performing • Analyze the informaAon provided to drive improvements 13 Quest 2014
- 14. How can measurementhelp us (YOU) • Create a organizational memory – baselines of current practices-situation • Determine strengths and weaknesses of the current process and product – What types of errors are most common? • Develop reasoning for adopting/refining techniques – What techniques will minimize the problems? • Assess the impact of techniques – Does more/less manual functional testing versus automation reduce defects? • Evaluate the quality of the process/product – Are we applying inspections appropriately? – What is the reliability of the product before/after delivery? 14 Quest 2014
- 15. Boil it Down-Understand, Evaluateand Improve • To do this… We need metrics Can you think of other examples in our lives where this applies? Where do you use metrics to evaluate and improve? 15 Quest 2014 Understand Evaluate Improve
- 16. Group Exercise • Howdoes your organization define software quality? • Come up with a definition of software quality that you can agree on. – Can be one sentence to a short paragraph. © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 16 If the goal is to improve software quality, then what is IT?
- 17. Metrics in reallife Food Eaten Weight Performance Race Results 17 • Calories • Fat • Carbohydrates • Protein • Time of day • Vitamins • … • Blood pressure • Cholesterol • Blood glucose • Red cell count • White cell count • Hematocrit • Hemoglobin • Body fat % • … • Placing • … • Effort/Power • Heart rate/Watts • Speed • Time Intelligence Finesse Context • Training • Sleep Quest 2014
- 18. DO Think about theprocess you are measuring and measure at each step in the process.
- 19. Software Quality Components © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 19 ISO 25010CMMI
- 20. Evolution of SWQuality © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 20 Software processes Code Process quality Software Quality (internal) Software Quality (external) Software Quality (in use) Product Usability CMMI assessment model white box testing black box testing Usability testing Usability heuristic Usability logging CMMI Assessment Company Programmer Tester End User Type of quality What is measured How measured? Who measures? ISO 9000 ISO 9241ISO 9126 ISO 25010
- 21. Total Quality View (organizational) © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 21
- 22. For Those in Agile Waterfall • Speed • Quality • Cost Agile • Speed • Quality • Cost Quest 2014 22 Where’s the beef?
- 23. Be-‐Do-‐Have Be Do Have Quest 2014 23
- 24. Be-‐Do-‐Have Be Do Have Quest 2014 24 Process • Iterative (sprints) • Daily standups • Face to face communication • Post mortem – end of sprint • Delivery meeting – end of sprint • Planning meeting – before sprint • Self organizing People • Communicative • Collaborative/Cooperative • Flexible and willing • Knowledgeable-multi • Initiative/responsible • Responsive Results • Speed • Quality Velocity Defects Adherence to plan (many) Overtime Emails Defect Aging Defects not fixed rate
- 25. Measuring Quality • Withineach phase, we model to understand and evaluate quality at that phase • At each phase, define quality and develop quality measures • The lower level measurements will aggregate to higher cumulative measurements • Aggregation can be done with various weighting schemes with flexibility according to the business of the organization © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 25
- 26. Don’t – Measurethe wrong thing • Make sure your metrics help you determine if meeting a goal • Some sample metrics to review: – Test Coverage = Number of units (KLOC/FP) tested / total size of the system – Test Density-Number of tests per unit size = Number of test cases per KLOC/FP – Test cases executed – Defects per size = Defects detected / system size – Defects found per tester – Test cases written per day Quest 2014 26 Don’t measure something just because you can
- 27. “Not everything thatcan be counted counts, and not everything that counts can be counted.” Albert Einstein Don’t measure something just because you can.
- 28. DO Be clear aboutWHY you are measuring.
- 29. Why we needto measure ? • Our bosses want us to… • They want someone to point fingers at • They want to fire some people and save money • They need to report to their managers • They want some basis on which to evaluate us and give us a raise! • We need to figure out a way to do beWer! • We want to improve our work and improve soMware quality 29 Quest 2014
- 30. The Metrics Conundrum • QA and Testing Language – Defects – Execution status – Test cases – Pass/fail rates – DRE… • Business Language – Cost effecAve – ROI – Cost of ownership – Cost of poor quality – ProducAvity – Calls to help desk – Customer saAsfacAon – Customer retenAon 30 Quest 2014
- 31. In your organization… • What measurements do you take in your organization and why? • Who uses them and for what? 31 Quest 2014
- 32. The Metric Reality • Measurement and metrics are like dinner. It takes 2-3 hours to make dinner, and 15 minutes to consume… • But… many metrics are never reviewed or analyzed (consumed) • WHY? 32 Quest 2014
- 33. The Metric Conundrum • Test leads and test managers rarely have the right metrics to show or quantify value • Metric collection and reporting are a drag • QA metrics usually focus only on test execution • Test tools don’t have most of the metrics we want • Reports generated by QA are only rarely reviewed • Metrics are not connected to anything of value/ meaningful for ________. 33 Quest 2014
- 34. Don’t – Forgetto differentiate between quality, testing and defects • Metric becomes the goal • Organizations concentrated on “the metrics”, forget to understand the metric’s relationship to the goal or objective. • Defect counts need to be incorporated into an overall valuation because Quality is ultimately measured in the eyes of the end user. Quest 2014 34
- 35. Software Quality Evaluation Frameworks Goal Question Metric Why and What Metrics UnderstandingSoftware Quality © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 35 Business&Product Processes/Lifecycles Software Dev Processes TesAng Metrics Software Testing Processes Software Quality Models Software Quality Processes
- 36. Don’t – Forgetabout context • Metrics don’t have consistent context so they are unreliable – Context needs to be defined and then maintained for measurements to be meaningful. • Difficult in today’s environment with changing test platforms and other contextual factors. Quest 2014 36
- 37. What contextual factorscould there be? • Release complexity • Development methodology • Software maturity • Development team maturity and expertise • Development team and QA integration • Resources available • User base Quest 2014 37 All metrics need to be normalized for proper interpretation
- 38. Metrics need contextto tell the whole story • Normalized per function point (or per LOC) • At product delivery (first X months or first year of operation) • Ongoing (per year of operation) • By level of severity – Gross numbers don’t tell much • By category or cause, e.g.: requirements defect, design defect, code defect, documentation/on- line help defect, defect introduced by fixes, etc. – Absolute and Total numbers tell 0 – Too granular also tell 0Quest 2014 38
- 39. Don’t – Besporadic or irregular • Measurements used are not consistent – Just as context needs to be consistent, so do the measurements, methods, and time intervals that you collect the measurements and calculate the metrics. • Just as in weighing yourself, it doesn’t make sense to drink 2 gallons one day and weigh in, and go jogging 10 miles the next day and weigh in. Quest 2014 39
- 40. Measurements Quality • Two facets of measurement quality are reliability and validity. • Reliability: means how repeatable the results are and low reliability indicates that many random errors are occurring during measurement. • Validity: means the metric measures what it intended to measure. – Errors in validity, called systemaAc errors, are usually experienced because not all factors were taken into account when the measurement was designed. © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 40
- 41. Metrics Abandoned forMany Reasons • How many of you have had these problems? – That measurement is not valid because… – That measurement is not reliable because… © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 41 If no one believes (and therefore takes improvement actions) your metrics, then why bother?
- 42. Do Understand MetricTerms • What is a metric? • What is a measurement? • What is an indicator? • What’s the difference? 42 © 2014 XBOSoM, Inc.-‐ All Rights Reserved.
- 43. Measures Versus Metrics Measures • To be executed • Executed • Passed • Failed • Re-‐executed • Total ExecuAons • Total Passes • Total Failures © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 43 Metrics • TC % complete • TC % passed • % failures • % defects corrected • % bad fixes • % defect discovery
- 44. Don’t – Collectmeasurements that no one wants • Metrics have no audience – As a corollary to the previous factor, if there is no question to be answered, then there will also be no audience for the metric. • Metrics need to have an audience in order to have meaning. • How many of the metrics and reports that you generate are read? Quest 2014 44
- 45. Poll: How manyof you collect metrics that you don’t need or use?
- 46. Group Exercise Who are your stakeholders? • CEO/CTO/CIO • Development manager/VP • QA manager • Others? Quest 2014 46
- 47. Do - Collectwhat “they” want • Ratios and percentages rather than absolutes • Comparisons over time, or by release • Report on what leads to improvement in: – Costs – Time – Quality Quest 2014 47
- 48. Do - Collectwhat they want Costs (Some potenAal metrics include): • Business losses per defect that occurs during operaAon • Business interrupAon costs • Costs of work-‐arounds • Costs of reviews, inspecAons and prevenAve measures • Costs of test planning and preparaAon • Costs of test execuAon, defect tracking, version and change control • Costs of test tools and tool support • Costs of test case library maintenance • Costs of tesAng & QA educaAon associated with the product • Re-‐work effort (hours, as a percentage of the original coding hours) • Lost sales or goodwill • Annual QA and tesAng cost (per funcAon point) Quest 2014 48
- 49. Do - Collectwhat they want Time-‐Resources (Some potenAal metrics include): • Labor hours/defect fix • Turnaround Ame for defect fixes, by level of severity • Time for minor vs. major enhancements – actual vs. planned elapsed Ame • Effort for minor vs. major enhancements • actual vs. planned effort hours Quest 2014 49
- 50. Do - Collectwhat they want Quality (Some potenAal metrics include): • Survey before, after (and ongoing) product delivery • # system enhancement requests per year • # maintenance fix requests per year • User problems: call volume to customer service/Tech support • User Satisfaction – training time per new user, time to reach task time of x – # errors per new user • # product recalls or fix/patch releases/year • # production re-runs • Availability (Ame system is available/ Ame the system is needed to be available) Quest 2014 50
- 51. Do Collect whatthey want • Show them in combination and relative to each other – Cost vs. quality – Cost vs. time – Quality vs. time Quest 2014 51
- 52. Don’t – Makethe collection effort the end game • Measurements are too hard to get – If you end up designing the right metric to answer the right question, it doesn’t matter if it takes several man days to get the data and do the calculations. • Unless the value and decisions made from these metrics have considerable value, they’ll soon be abandoned. Quest 2014 52
- 53. Poll: How manyof you started to collect metrics but then found it was too difficult or time consuming and quit?
- 54. Don’t – Forgetindicators • Metrics have no indicators so cannot evaluate – You collect mounds of data but then what? – How do you determine what is ‘good’ or ‘bad’? – Before you get started collecting and calculating you need to put together a way to evaluate the numbers you get with meaningful indicators that can be used as benchmarks as your metrics program matures. Quest 2014 54
- 55. Indicators • Indicators change numbers into meaning • What does 68% mean? © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 55
- 56. Don’t Mix UpCausality • The cause and effect relaAonship has three requirements: – cause precedes effect in Ame or by logic. – shown cause and effect relaAonship – direct constant relaAonship • When the relaAonship between variables can be graphed and a line of best fit can be found • Be careful of extreme and inaccurate data values. • When defining a casual relaAonship be sure to watch for misguided interpreted relaAonships. CASE 1 Z causes changes in X Z causes changes in Y X is perceived to cause changes in Y or vice versa and Z is overlooked CASE 2 X causes changes in Z Z causes changes in Y X is perceived to cause changes in Y and Z is overlooked © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 56
- 57. Indictors and Interpretation © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 57
- 58. Conclusions • Designing andimplementing a software quality metrics program requires more than defect analysis. • Think - who and what questions that you want to answer or goals of using metrics. – Many refer to this as the goal-question-metric paradigm (GQM) – In simple terms, what are you going to do with the numbers once you get them? • Most of the “Don’ts” are related to not thinking about the objectives of the metrics and actions you will take based on them.Quest 2014 58
- 59. Solutions • Develop astakeholders list and their goals- objectives • Develop a list of questions that, if answered, would determine that the goals are met • Develop a catalogue of metrics (that answer the questions) that can mix and match to apply to the goals depending on the stakeholder • Develop and collect metrics that accompany each part of the development process, not just testing – There are many “defects” not directly in dev. Quest 2014 59
- 60. Measurement Framework Improvement Decisions Stakeholders 60 Quest 2014 Do Develop a Metrics Framework
- 61. SoMware Quality Dashboard (model) Process SoMware Quality Product UsersPeople Quality of the people Quality of the stakeholders goals and objectives (GQM) Quality Processes to implement the goals And the technology to support them Viewpoint and measurement of quality from the end user If the ‘people’ are connected well to end users, this should also be high quality; with incorporation of context. High quality people, with high quality processes should produce repeatable high quality product From both internal and external quality views © 2014 XBOSoM, Inc.-‐ All Rights Reserved. 61
- 62. Thanks Q&A www.xbosoft.com @xbosoft 408-350-0508 Philip Lew @philiplew philip.lew@xbosoft.com White Papers:http://www.xbosoft.com/knowledge_center/ /xbosoft Blog: http://blog.xbosoft.com/ /xbosoft