Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Project quality (and test process) metrics

2,488 views

Published on

Presentation from lecture on Poznań University of Technology for IT students.

Published in: Technology
  • Be the first to comment

Project quality (and test process) metrics

  1. 1. Project quality and test process metrics Zbigniew Moćkun, Tomasz Rękawek © 2011 Cognifide Limited. In commercial confidence only.
  2. 2. Who are we? © 2011 Cognifide Limited. In commercial confidence only.
  3. 3. Agenda • Introduction to metrics • Code quality metrics − Metrics − Sonar as example • Project quality and testing process metrics − Metrics − Example quality report © 2011 Cognifide Limited. In commercial confidence only.
  4. 4. Metrics What is it about? © 2011 Cognifide Limited. In commercial confidence only.
  5. 5. Why do we use metrics? • • • • • • Project overview Control a process Control risks Force the use of good practices Project audits Projects / methodologies / ... comparison Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  6. 6. I realized that metrics... Encourage me to ask questions © 2011 Cognifide Limited. In commercial confidence only.
  7. 7. Be careful with metrics!! Processes Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore People © 2011 Cognifide Limited. In commercial confidence only.
  8. 8. Type of measurements • Trend • KPI (Key Performance Indicator) − Lower threshold − Higher threshold − Value • How to set correct KPI? Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  9. 9. Not only about numbers Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  10. 10. Qualitative vs. Quantitative - case study Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  11. 11. Level of details • • • • You cann’t measure everything Choose the most important metrics Monitor and ask questions Looking for details when you found a problem Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  12. 12. Metrics gathering • • • • Analysis of live data (on request) Analisis based on daily values (can be run as cron job) Data vary in time manner Froze data for analysis © 2011 Cognifide Limited. In commercial confidence only.
  13. 13. Visualize metrics • • • • Use charts if possible Use colors as status (green, amber, red) Add description Remember about recommendation (your feeling) − Subjective assessment is important too Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore © 2011 Cognifide Limited. In commercial confidence only.
  14. 14. Project Quality metrics Quality is everywhere © 2011 Cognifide Limited. In commercial confidence only.
  15. 15. Code quality metrics Code quaility metrics are covered by Tomek Rękawek presentation which can be found here: http://www.slideshare.net/TomaszRkawek/code-metrics © 2011 Cognifide Limited. In commercial confidence only.
  16. 16. QA vs. code quality metrics • Part of assurance • Define KPI with Technical Lead • Understand what you measure and why • Discuss trends, exception with TL • Motivate developers to keep high coding standards © 2011 Cognifide Limited. In commercial confidence only.
  17. 17. Project quality metrics © 2011 Cognifide Limited. In commercial confidence only.
  18. 18. Why do we test? Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test © 2011 Cognifide Limited. In commercial confidence only.
  19. 19. Testing = measuring Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test Source: wikipedia © 2011 Cognifide Limited. In commercial confidence only.
  20. 20. Dev – QA cooperation © 2011 Cognifide Limited. In commercial confidence only.
  21. 21. Metric by actions © 2011 Cognifide Limited. In commercial confidence only.
  22. 22. One measurement, two metrics • Acceptance rate / Internal Quality − Acceptance actions/all qa actions − All QA actions = accept or reject − Measure quality of software send to QA • QA velocity − Sum off all actions − Divide by number of QA Engineers assigned to project − QA velocity depend on software quality that comes to qa − Why testing takes so long? © 2011 Cognifide Limited. In commercial confidence only.
  23. 23. Test Case Execution metrics • Planned test cases vs. written test case (Preparation, UAT) • Number / percentage of executed test cases (Acceptance, regression, UAT) • Number of passed/failed test cases (Acceptance, regression, UAT) • Test Case metrics mean nothing © 2011 Cognifide Limited. In commercial confidence only.
  24. 24. Requirement coverage • Requirement coverage − How many requirements have test cases? − How many requirements were tested? − How many requirements passed? − Can we measure coverage if we use exploratory approach? • Requirement traceability − Defect – test case - requirement © 2011 Cognifide Limited. In commercial confidence only.
  25. 25. Bugs metrics – S curve • S Curve • Open against Resolved • Wages − Blocker: 10 − Critical: 8 − Major: 6 • No Major or higher rule − 5 Trivials = 1 Minor − 5 Minors = 1 Major , ... © 2011 Cognifide Limited. In commercial confidence only.
  26. 26. External quality © 2011 Cognifide Limited. In commercial confidence only.
  27. 27. The importance of metrics • Example A − − − − − − − Story comes to QA QA found 1 major issue and reject (1 day) Dev fix it (1 day) QA test it again, 1 major issue found and reject(1 day) Dev fix it (1 day) QA accept (1 day) Sum: 5 days, 2, rejects, 2 issues only • Example B − − − − − Story comes to QA QA found 1 critical, 1 major and 3 minor issues and reject story (1 day) Dev fix it (2 day) QA test it again and accept (1 day) Sum: 4 days, 1 reject, 5 issues • Acceptance rate against raised issues metric © 2011 Cognifide Limited. In commercial confidence only.
  28. 28. Bug Taxonomy © 2011 Cognifide Limited. In commercial confidence only.
  29. 29. Bug Taxonomy • Phases − Project: Discovery, Development, UAT, Live • Test Case lifecycle − Requirements/Documentation, Test Scenario, Test Execution • Risk areas • Application modules/components • Functional/non-functional • Product specific areas − CMS: author, publish, frontend, backend © 2011 Cognifide Limited. In commercial confidence only.
  30. 30. Defect Density - components © 2011 Cognifide Limited. In commercial confidence only.
  31. 31. Defect Density - Acceptance vs. Regression (automation vs. manual) © 2011 Cognifide Limited. In commercial confidence only.
  32. 32. Defect Density Application specific © 2011 Cognifide Limited. In commercial confidence only.
  33. 33. Defect Density – won’t fix Do we test in right way? © 2011 Cognifide Limited. In commercial confidence only.
  34. 34. Automation - coverage • Defects found (manual vs. automated) • Coverage − Manual vs. automated test cases − High priority − Application specific  Author vs. publish © 2011 Cognifide Limited. In commercial confidence only.
  35. 35. Other metrics • Performance monitoring trend as example • Client side grades (webpagetest.org as example) • QA queue • Agile metrics − Lead Time (Open to Resolved time) − Idle Time (time spend in queues) © 2011 Cognifide Limited. In commercial confidence only.
  36. 36. Qualitative metrics • Client feeling − Surveys − Demo − Just try to talk • Tester / team subjective opinion − Recommendation − Feeling − Usability © 2011 Cognifide Limited. In commercial confidence only.
  37. 37. Agile vs. Waterfall • Quality metrics are independent against: − Methodology − Approach • Two main test approach can be used in both − Exploratory vs. scripting • Agile put more emphasis on qualitative metrics © 2011 Cognifide Limited. In commercial confidence only.
  38. 38. Example report © 2011 Cognifide Limited. In commercial confidence only.
  39. 39. References • http://www.kaner.com/pdfs/BugTaxonomies.pdf • ISTQB Advance Test Manager Syllabus − http://www.istqb.org/downloads/viewcategory/46.html • Michael Bolton blog post − http://www.developsense.com/blog/2009/01/meaningful-metrics/ • Douglas Hoffman − http://www.softwarequalitymethods.com/Papers/DarkMets%20Pape r.pdf © 2011 Cognifide Limited. In commercial confidence only.
  40. 40. Poznań Testing and Quality Group • Local group • Next meeting: 28th of November − Bogdan Bereza-Jarociński: Quality in Agile – quality at all − Jakub Bryl: How to organize security testing © 2011 Cognifide Limited. In commercial confidence only.
  41. 41. Q&A © 2011 Cognifide Limited. In commercial confidence only.

×