Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
QA Metrics in Agile
Look at the Quality from different angels. Real appliance.
By Volodymyr Prymakov
www.qaexperts.pro
About Me
Volodymyr Prymakov, Ukraine
Senior QA Manager at Ciklum, TCoE
Head of Performance QA Unit at TCoE
Interim Head of...
What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
req...
What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and effi...
Filters/Bases
for
QA Metrics
Measuring Product
Quality
www.qaexperts.pro
Definition of Done (DoD) defines
Which quality Product should
have?
Which Way and How
thoroughly a product or
functionalit...
Definition of Done for Stories, e.g.
• Coded and integrated to main branch,
• Unit tests are written and passing,
• API co...
Definition of Done for Sprint, e.g.
• Stories that meet DoD are considered DONE, the rest goes to
next sprint.
• It should...
Definition of Done for RELEASE, e.g.
• All Stories, which were merged to Main branch, meets DoD for stories.
• Risky or bu...
Definition of Done
dictates you
which Activities and QA metrics
to perform and monitor!
www.qaexperts.pro
QAMetrics
www.qaexperts.pro
Types of Metrics
Quantative - Information that is directly measured as a
number
Qualitative - Information that results fro...
Types of Metrics
1. Absolute metrics
2. Relative metrics
3. Derivative metrics
Purpose of Metrics.
1. Goal Metrics: To mea...
Stages at which QA and other metrics are needed
Implementation period
• Daily Monitoring of
Quality, product
implementatio...
QA Metrics
for
Daily
Monitoring
www.qaexperts.pro
Goals of Daily Quality Monitoring
1. What is the current quality of the
product?
2. How much work is left?
3. Do we have e...
!!!Important:
Breakdown Defect Metrics
Always breakdown your metrics to the following groups:
All Bugs! – shows general pr...
Defects reported by Date
Critical and Blocker bugs Major and lower priority bugs
Jira “Recently Created Chart” Gadget
Allo...
Open Defects reported Today
Jira “Filter Results” Gadget
Allows you to quickly review new bugs
www.qaexperts.pro
Open Defects By Component/Platform
Major and lower priority bugs
Jira “Pie Chart” Gadget
Critical and Blocker bugs
Shows w...
Defects By Component/Platform for Current
Affected Version (any Status)
Major and lower priority bugs
Jira “Pie Chart” Gad...
Critical Open Bugs by Status
Jira “Issue Statistics” Gadget
Allows you to quickly get Status of Open Critical Bugs and und...
Open Bugs by Priority
Jira “Issue Statistics” Gadget
Shows general quality of the product
www.qaexperts.pro
!!!NOT-assigned Open Critical Defects
Jira “Issue Statistics” Gadget
To Fix Version
To Team
(for big teams only)
To Sprint...
www.qaexperts.pro
Workload:
Assigned Open Tasks and Bugs for Developers/QAs
Jira “Issue Statistics” Gadget
By teams By Dev...
Release scope status by developers/QAs/teams
If by the end of sprint or before stabilization period at release a lot of st...
Stories/Bug Status By Fix Version
created >= -365d
Stories Bugs
created >= -180d
Allows to understand ability to finish/fi...
Current Fix Version Progress
Shows general release/sprint ProgressJira “Release Report”
www.qaexperts.pro
Team Burndown Chart
www.qaexperts.pro
Kanban board for All Open Critical and Blocker Bugs
(For big teams)
Allows to
accumulate
and track all
open Critical
and B...
Regression testing
Recommended Statuses:
• Failed with Critical and Blocker bugs
• Failed with Major and lower priority bu...
Regression Testing Coverage and Results
For Release/Milestone
– General Results
By Cycle/Stages
- Every cycle should has c...
Regression Testing Coverage and Results
By Product Component
• by platform/device
• by test priority:
• Priority 1
• Prior...
QA Metrics
For Regular
Quality Data
Reports
(Snapshots)
www.qaexperts.pro
Sprint Report
1. Story Delivery rate % = Actual Stories Completed vs. Committed Stories, e.g.: 90%
2. Amount of Stories me...
Regression testing reports, daily or by cycles
Reports about Regression testing, cross platform testing, 3d-party integrat...
Release Readiness testing Report (by necessity)
1. Specify functional testing accomplished and its status:
Regression and ...
Regression and Release Readiness Report example
www.qaexperts.pro
Reminder Reports (Be necessity)
1. Critical and Blocker Bugs in open state.
2. A Lot of Open Major bugs
3. A lot of Bugs a...
Trend-based
QA Metrics
for
for process
and product
quality
analysis www.qaexperts.pro
Milestones for trends
• AffectedVersion – AffectedVersion field in jira
• Time Span - e.g. period between start and end
da...
Found on Environment
Critical
and
Blocker
bugs
Shows leaks of
Critical and
Blocker bugs to
later
development
stages
Pivot ...
Found on Environment
All bugs
Shows leaks of
bugs to later
development
stages
Pivot Report
www.qaexperts.pro
Bugs By Priority – Any Status
All Bugs
by
Aff. Version
Shows absolute
release quality
history
– decrease in
number of all
...
Bugs By Priority – OPEN (not-closed)
All Bugs
by
Aff. Version
Shows still open
bugs for
releases
– Ideally No
critical and...
Bugs by RootCause
Typical Root causes of Bugs:
FrontendOthers
Backend
Frontend
+ Backend
Environme
nt
issues
3d party
comp...
Bugs by RootCause
Critical
and Blocker
bugs
Shows Root cause of all bugs
Pivot Report
All Bugs
Shows Root cause of the mos...
Created Vs Resolved - Cumulative
Critical
and Blocker
Bugs
Created should be less than resolved
Jira “Created vs. Resolved...
Created Vs Resolved - Cumulative
All Bugs
Difference should be minimal
and If it is big and no positive
trend is shown the...
Defects Dept
Critical and Blocker Bugs
Note: Should Be always close to Zero for Critical bugs
All Bugs
Spreadsheets + adva...
Bugs Opened vs Stories Delivered
There should be correlation between the metrics - more stories you deliver more bugs may ...
Bugs per Story Point
Critical
and Blocker Bugs
If number
decreases in
time it means
that we
improve QA
process Note: This ...
Bugs per Story Point
Other Variations:
1. Bugs per requirement
2. Bugs per developer
3. Bugs per 1000/Code
4. Time lost fo...
Fix Rate
Should be close to 100% or higher
Spreadsheets + Advanced Jira period based queries
Critical and Blocker Bugs
All...
“Average Age” and “Time in status”
For Critical
and Blocker
Bugs
Jira “Average Age” and “Time in Status” Gadgets
Average A...
Cumulative Flow Chart
Jira Agile Reports
To find process bottlenecks and predict project close time
www.qaexperts.pro
Bugs by Components/Platforms
Pivot or other spreadsheet Report
Bugs trend By Component – Critical and Blocker / All Bugs
S...
Sprint and Business Metrics
Estimates Accuracy % = Estimated amount of
effort/Actual amount of effort-1
Story Delivery Rat...
User Satisfaction
Number of hot fixes had to be made in this or that release.
Shows leaks of Critical bugs on prod.
Bugs f...
Qualitative Metrics about the project state
If you want to gather more metrics about project processes, approaches,
techno...
QA Metrics
for
Team and
Individual
Efficiency
analysis
www.qaexperts.pro
Developers and Dev Team efficiency
REOPENED:
Reopened Bugs Ratio % = Reopened bugs vs Resolved bugs
Reopened Bugs Ratio % ...
QAs and QA team Efficiency
REJECTED
Absolute Number of rejected bugs:
All resolutions,
Not a bug,
Duplicate
Cannot reprodu...
QAs and QA team Efficiency
WORK DONE:
 Stories Tested
 Created bugs Critical and Blocker (not rejected)
 Created bugs M...
QA Metrics for
Test
Automation
www.qaexperts.pro
 Unit test coverage & number of test
 API testing coverage & number of test
for product in general
by component
 GUI Au...
Automation Efficiency
AUTOMATION SPEED:
 Automation tests Frequency
 Duration of tests Running
 Velocity of script crea...
QA Metrics
TOOLS
www.qaexperts.pro
Tools
 Jira + Special time-
related JQL
 Confluence + Jira and
other Macros in it
 Spreadsheet processors:
Googlesheets...
JiraReportingInGsheets Tool
Link to test version:
https://chrome.google.com/webstore/detail/jirareportingingsheets/jmjoohl...
Thank you
Darya Alymova and Sergey Sosnitskiy
for their contribution to certain metrics definition and
implementation
www....
QA Experts.Pro Workshops
2 days workshop
November 2017
http://qaexperts.pro/#signup
www.qaexperts.pro
Thank you
www.qaexperts.pro
Upcoming SlideShare
Loading in …5
×

QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение

421 views

Published on

Что такое качество продукта и процесса разработки. Как его измерять. Какие метрики гарантируют качество продукта, а какие важны для принятии решения о готовности продукта к релизу. Тренды качества, их польза в понимании улучшения качества продукты и процесса разработки. Абсолютные и относительные метрики. Инструменты.

Published in: Education
  • Be the first to comment

QA Fest 2017. Владимир Примаков. QA метрики. Взгляд на качество с разных сторон. Реальное применение

  1. 1. QA Metrics in Agile Look at the Quality from different angels. Real appliance. By Volodymyr Prymakov www.qaexperts.pro
  2. 2. About Me Volodymyr Prymakov, Ukraine Senior QA Manager at Ciklum, TCoE Head of Performance QA Unit at TCoE Interim Head of TCoE at Thomas Cook •14 years in QA • >45 projects experience • Certified: ISTQB Advanced Test Analyst & Manager, ICAgile CP, SAFE /in/vladimirprimakov/ www.qaexperts.pro
  3. 3. What is quality of an application Requirements fit business needs Requirements Has high quality Product corresponds to requirements Product No Outstanding Defects Product satisfies real life conditions Product Meets user needs Minimal QA Influence Medium QA Influence High QA Influence High QA Influence High QA Influence Guessing Verification Verification Verification Validation Validation Minimal QA Influence www.qaexperts.pro
  4. 4. What is quality of Process Quality Process brings much Business Value with keeping project or program engine safe and efficient 1. Deliver new functionality often 2. Deliver more functionality 3. Delivery quality software 4. Delivery what is planed 5. Deliver with minimal risks 6. Business trusts dev teams, processes, and produced results www.qaexperts.pro 1. Project vision, goals, roadmap, and other plans are clear to everybody 2. Process is transparent and strait-forward 3. Product quality is always transparent 4. Technical dept is being eliminated 5. Efficient Communication and Interdependency solving 6. Shared decisions making and responsibility 7. Lessons learned and continuous improvement are in place 8. High team sprit and satisfaction
  5. 5. Filters/Bases for QA Metrics Measuring Product Quality www.qaexperts.pro
  6. 6. Definition of Done (DoD) defines Which quality Product should have? Which Way and How thoroughly a product or functionality should be tested? Min Sufficient Max Allowed www.qaexperts.pro
  7. 7. Definition of Done for Stories, e.g. • Coded and integrated to main branch, • Unit tests are written and passing, • API component tests are written and passing, • Code reviewed and conforms to coding standards Passed Dev Testing: • Code deployed to QA environment. Automatic API-integration and EndToEnd smoke testing tests are passing, • Test Design is completed, • Positive and negative Acceptance Tests are (automated and) passed on platforms of priority 1. Exploratory tests are passed on platform of priority 2. • Selective Regression is carried out by a necessity and passed. Passed QA testing: No (P1/P2) bugs outstanding for STORY and neighbor functionality Story meets INVEST criteria Story is Signed-Off by Product Owner www.qaexperts.pro
  8. 8. Definition of Done for Sprint, e.g. • Stories that meet DoD are considered DONE, the rest goes to next sprint. • It should be avoided to merge Risky or buggy code to main branch at the end of sprint! Stories meets DoD • Regression testing is passed with P1 test cases on at least 1 top priority platform/device • Smoke or exploratory testing is passed on platforms/devices of priority 1 and 2 Cover Regression Test • Main Integration tests for live 3d-party services are passed on STAGE or other environments3d party Integration Test No (P1/P2) bugs outstanding for product and neighbor products. Sprint is Signed-Off by Chief Product Owner www.qaexperts.pro
  9. 9. Definition of Done for RELEASE, e.g. • All Stories, which were merged to Main branch, meets DoD for stories. • Risky or buggy code is not merged to main branch or is toggled off or is reverted! Stories meets DoD • Regression testing is passed for all priority 1 platforms/devices. • Exploratory testing is passed for priority 2 platforms/devices. • Smoke testing is passed for priority 3 platforms. Full Regression Test • All Integration tests for live 3d-party services are passed on STAGE or other environments • There is no other 3rd party release ongoing that may impact release negatively 3d party Integration Test and Releases • UAT is passed on STAGE environment UAT • Performance Testing successfully passed • Security Testing successfully passed • and other types of testing and preparations to release Other Types of Testing No (P1/P2) bugs outstanding for product and neighbor products Release GO Aproval from by Chief Product Owner, QA team, Release Manager and respective Business representatives All other unresolved bugs are tracked, reviewed, and planned when to be fixed www.qaexperts.pro
  10. 10. Definition of Done dictates you which Activities and QA metrics to perform and monitor! www.qaexperts.pro
  11. 11. QAMetrics www.qaexperts.pro
  12. 12. Types of Metrics Quantative - Information that is directly measured as a number Qualitative - Information that results from human observation and analysis www.qaexperts.pro
  13. 13. Types of Metrics 1. Absolute metrics 2. Relative metrics 3. Derivative metrics Purpose of Metrics. 1. Goal Metrics: To measure results/goals 2. Actionable metrics: to make decision 3. Predictive metrics: to predict future www.qaexperts.pro
  14. 14. Stages at which QA and other metrics are needed Implementation period • Daily Monitoring of Quality, product implementation, and test progress Key sprint and release points • QA Report Before sprint Demo • QA Reports During stabilization / regression / UAT period • QA report Before Release • QA Reminder report when product has critical quality state or low product implementation or testing progress After Release • Release quality and process analysis • Quality and process Trends and analysis • Team efficiency analysis www.qaexperts.pro
  15. 15. QA Metrics for Daily Monitoring www.qaexperts.pro
  16. 16. Goals of Daily Quality Monitoring 1. What is the current quality of the product? 2. How much work is left? 3. Do we have enough time to finish and stabilize the product? www.qaexperts.pro
  17. 17. !!!Important: Breakdown Defect Metrics Always breakdown your metrics to the following groups: All Bugs! – shows general product quality state Critical and Blocker bugs – Shows whether we meet Readiness criteria! Lower Priority bugs– Shows the rest of bugs, general amount of which can negatively influence on the end-user’s impression about the product. www.qaexperts.pro
  18. 18. Defects reported by Date Critical and Blocker bugs Major and lower priority bugs Jira “Recently Created Chart” Gadget Allows you track new bugs and immediately react to them! www.qaexperts.pro
  19. 19. Open Defects reported Today Jira “Filter Results” Gadget Allows you to quickly review new bugs www.qaexperts.pro
  20. 20. Open Defects By Component/Platform Major and lower priority bugs Jira “Pie Chart” Gadget Critical and Blocker bugs Shows which component/platform is not ready for release or requires bug fixes www.qaexperts.pro
  21. 21. Defects By Component/Platform for Current Affected Version (any Status) Major and lower priority bugs Jira “Pie Chart” Gadget Critical and Blocker bugs Shows which component/platform require more thorough testing, e.g. (Selective) Regression testing www.qaexperts.pro
  22. 22. Critical Open Bugs by Status Jira “Issue Statistics” Gadget Allows you to quickly get Status of Open Critical Bugs and understand whether we have enough time to fix them www.qaexperts.pro
  23. 23. Open Bugs by Priority Jira “Issue Statistics” Gadget Shows general quality of the product www.qaexperts.pro
  24. 24. !!!NOT-assigned Open Critical Defects Jira “Issue Statistics” Gadget To Fix Version To Team (for big teams only) To Sprint To Developer Shows probably lost or forgotten Critical open bugs, which should be processed or escalated immediately! www.qaexperts.pro
  25. 25. www.qaexperts.pro Workload: Assigned Open Tasks and Bugs for Developers/QAs Jira “Issue Statistics” Gadget By teams By Developers For current Fix Version or sprint By QAs Stories By teams By Developers By QAs Bugs To assure that there is no bottlenecks and overloads. To Remind about work assigned! To spread load equally between teams, Developers and QAs!
  26. 26. Release scope status by developers/QAs/teams If by the end of sprint or before stabilization period at release a lot of stories are in open status, it can bring risk of late merge status, not enough time for testing, not ability to finish functionality etc. Action: Try to spread load equally by teams, QAs and Dev Specialists. Remind about work Not Done. Postpone certain functionality to next Release! Stories Bugs Jira “Two Dimensional Filter Statistics” Gadget www.qaexperts.pro
  27. 27. Stories/Bug Status By Fix Version created >= -365d Stories Bugs created >= -180d Allows to understand ability to finish/fix all tasks or bugs in time for a certain sprint/release Jira “Two Dimensional Filter Statistics” Gadget www.qaexperts.pro
  28. 28. Current Fix Version Progress Shows general release/sprint ProgressJira “Release Report” www.qaexperts.pro
  29. 29. Team Burndown Chart www.qaexperts.pro
  30. 30. Kanban board for All Open Critical and Blocker Bugs (For big teams) Allows to accumulate and track all open Critical and Blocker bugs in 1 place www.qaexperts.pro
  31. 31. Regression testing Recommended Statuses: • Failed with Critical and Blocker bugs • Failed with Major and lower priority bugs • Failed by automated tests • Blocked • Not tested • In Progress • Passed www.qaexperts.pro
  32. 32. Regression Testing Coverage and Results For Release/Milestone – General Results By Cycle/Stages - Every cycle should has certain purpose, e.g: 1. Initial Feedback about quality, 2. Exploratory testing, 3. Confirmation testing 4. 3rd Party services testing. www.qaexperts.pro
  33. 33. Regression Testing Coverage and Results By Product Component • by platform/device • by test priority: • Priority 1 • Priority 2 • Priority 3 By Type of Test: • Smoke test • Regression test, • UAT test www.qaexperts.pro
  34. 34. QA Metrics For Regular Quality Data Reports (Snapshots) www.qaexperts.pro
  35. 35. Sprint Report 1. Story Delivery rate % = Actual Stories Completed vs. Committed Stories, e.g.: 90% 2. Amount of Stories met DoD, e.g.: 25 3. Amount of Stories Moved to a new sprint, e.g.: 4 Because of 1 not implemented, 1 Reopen because of Critical Bugs, 2 Not Tested Specifies the reasons, risks and recommendations. 4. Regression testing coverage and results: Graph 5. Cross platform testing coverage and results: Graph 6. 3d-party integration testing results: Graph or written explanation. 7. Testing and testing activities NOT done – describe reasons, connected risks, and recommendations. 8. Open Bugs: All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link to jira) Open Major bugs opened during the sprint, e.g: 40 Major bugs open during sprint. (link to jira) 9. Point to the buggiest product component and platform: Graph and explanation 10. General Conclusions www.qaexperts.pro
  36. 36. Regression testing reports, daily or by cycles Reports about Regression testing, cross platform testing, 3d-party integration testing, 3d party releases testing, etc. 1. Name of the Regression testing stage, e.g.: Regression Cycle 1 (Preliminary feedback) 2. Coverage and results: Graph 3. Open Bugs New (created today) and All Open Critical and Blocker bugs: New 4 Critical&Blocker Bugs, (All 5 Critical&Blocker Bugs) (list or links to jira) New (Created today) and All Open Major bugs opened during the release: New 10 Major bugs, (All Major bugs created for current release) (links to jira) 4. Specify Time needed to finish regression testing and/or necessity in additional regression and other testing. 5. Point to the buggiest product component, platform, or functionality if needed. 6. Recommendations www.qaexperts.pro
  37. 37. Release Readiness testing Report (by necessity) 1. Specify functional testing accomplished and its status: Regression and cross-platform testing, 3dparty integration testing, UAT e.g.: Written explanations and Links to Test management system, or Graphs of testing coverage and results. 2. Testing and testing activities NOT done – describe reasons, connected risks and recommendations. 3. Specify Non-functional testing status, like performance and security testing. Whether the product is ready or not ready from non-functional point of view. Describe connected risks and recommendations. 4. Open Bugs. All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link) Open Major bugs opened during the release and All Major bugs, e.g: 40 Major bugs open during release., 70 Major bugs are Open in general (list or links to jira) 5. Functionality Not ready for release, etc.: (list or link)+ written explanation 6. General Conclusions! www.qaexperts.pro
  38. 38. Regression and Release Readiness Report example www.qaexperts.pro
  39. 39. Reminder Reports (Be necessity) 1. Critical and Blocker Bugs in open state. 2. A Lot of Open Major bugs 3. A lot of Bugs and stories Not Done in the middle or later stage of sprints: In development, in Test, etc. 4. Availability of not tested stories during stabilization period. 5. Inability to finish testing in time because of certain issues. Reminder emails or chat messages about: www.qaexperts.pro
  40. 40. Trend-based QA Metrics for for process and product quality analysis www.qaexperts.pro
  41. 41. Milestones for trends • AffectedVersion – AffectedVersion field in jira • Time Span - e.g. period between start and end date of sprint Time based conditions in jira queries: a) createdDate >= '2017-08-30' and createdDate <='2017-09-12 ‘ b) status changed TO 'READY TO TEST' after '2017-08-30' before '2017-09-12‘ c) assignee was in (‘user’) • Sprint – Sprint Field in jira, etc. www.qaexperts.pro
  42. 42. Found on Environment Critical and Blocker bugs Shows leaks of Critical and Blocker bugs to later development stages Pivot Report www.qaexperts.pro
  43. 43. Found on Environment All bugs Shows leaks of bugs to later development stages Pivot Report www.qaexperts.pro
  44. 44. Bugs By Priority – Any Status All Bugs by Aff. Version Shows absolute release quality history – decrease in number of all and critical bugs shows process improvements Pivot Report Note: Should be considered carefully when the team size and product increment increase. Note: Blank version is bad, and it shows that the Affected Version is not specified for certain bugs and thus they can be lost www.qaexperts.pro
  45. 45. Bugs By Priority – OPEN (not-closed) All Bugs by Aff. Version Shows still open bugs for releases – Ideally No critical and blocker bugs should be here Pivot Report Note: If critical bugs are shown here for previous releases, they might be lost, forgotten or blocked. www.qaexperts.pro
  46. 46. Bugs by RootCause Typical Root causes of Bugs: FrontendOthers Backend Frontend + Backend Environme nt issues 3d party components www.qaexperts.pro
  47. 47. Bugs by RootCause Critical and Blocker bugs Shows Root cause of all bugs Pivot Report All Bugs Shows Root cause of the most of important bugs www.qaexperts.pro
  48. 48. Created Vs Resolved - Cumulative Critical and Blocker Bugs Created should be less than resolved Jira “Created vs. Resolved Chart” Gadget Bug found vs. fixed chart is a defect analysis line chart that lets us see defect removal process patterns and understand defect management effectiveness www.qaexperts.pro
  49. 49. Created Vs Resolved - Cumulative All Bugs Difference should be minimal and If it is big and no positive trend is shown then probably Bagathon sprint should be planned Jira “Created vs. Resolved Chart” Gadget www.qaexperts.pro
  50. 50. Defects Dept Critical and Blocker Bugs Note: Should Be always close to Zero for Critical bugs All Bugs Spreadsheets + advanced Jira time period based queries Defects dept is a cumulative difference between Resolved and Open bugs = (Open in sprint N - Resolved in sprint N)+ (Open in sprint N+1- Resolved in sprint N+1 )+… Note: Shows accumulation of bugs in time www.qaexperts.pro
  51. 51. Bugs Opened vs Stories Delivered There should be correlation between the metrics - more stories you deliver more bugs may appear. Ideal trend is to deliver more user stories with less bugs. !!!Pay attention to anomalies and investigate them deeper. Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
  52. 52. Bugs per Story Point Critical and Blocker Bugs If number decreases in time it means that we improve QA process Note: This metric is very good for cases when team size or product increment is variable or change (increase) All Bugs Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
  53. 53. Bugs per Story Point Other Variations: 1. Bugs per requirement 2. Bugs per developer 3. Bugs per 1000/Code 4. Time lost for bugs h - Time spent for Bugs vs time spent for stories Note: Very useful to for making future project estimations. www.qaexperts.pro
  54. 54. Fix Rate Should be close to 100% or higher Spreadsheets + Advanced Jira period based queries Critical and Blocker Bugs All Bugs Fix Rate = Closed bugs / Open bugs Ideally should be close to 100% www.qaexperts.pro
  55. 55. “Average Age” and “Time in status” For Critical and Blocker Bugs Jira “Average Age” and “Time in Status” Gadgets Average Age - Displays the average number of days issues have been unresolved. For critical and Blocker bugs this time should be very short. If process improves it should decrease in time. Average Time in status, e.g. Open status - Displays the average number of days issues have spent in status. For critical and Blocker bugs this time should be very short. If process improves it should decrease in time. www.qaexperts.pro
  56. 56. Cumulative Flow Chart Jira Agile Reports To find process bottlenecks and predict project close time www.qaexperts.pro
  57. 57. Bugs by Components/Platforms Pivot or other spreadsheet Report Bugs trend By Component – Critical and Blocker / All Bugs Shows weakest component of the product, shows where additional testing should be performed, shows where technical Improvements should be made, Shows whether component quality improves Bugs trend By Platform – Critical and Blocker / All Bugs. Shows amount of cross-platform bugs, shows the weakest platform requiring more testing and stabilization. . www.qaexperts.pro
  58. 58. Sprint and Business Metrics Estimates Accuracy % = Estimated amount of effort/Actual amount of effort-1 Story Delivery Rate % = Actual Stories Completed / Committed Stories % of Change Requests = Number of change requests / Number of Initial Backlog stories Change Requests Costs = Time spent of change requests * Average Rate of developers on the project % Costs of Defect = Average Time for defect fix * Average developer rate on the project  Trends  For Sprint  General value www.qaexperts.pro
  59. 59. User Satisfaction Number of hot fixes had to be made in this or that release. Shows leaks of Critical bugs on prod. Bugs found by end-users Critical/Blocker (Shows leaks of Critical bugs on prod.) Major and Low priority Customer Complaints and Feedbacks reports. (Net Promoter Score) A/B Testing results Shows which functionality is more preferred Other Marketing investigations and surveys All these answer the following questions: a) What are the weakest points of the product? b) What functionality is most used ? c) Which focus in product development and testing should be made? www.qaexperts.pro
  60. 60. Qualitative Metrics about the project state If you want to gather more metrics about project processes, approaches, technology, communication, mood in teams, etc. Just ask people to provide evaluation and feedback Experts evaluation and/or Audit Surveys Retrospectives 1-2-1 and other meeting Rumors/small talks at Smoking room/kitchen  www.qaexperts.pro
  61. 61. QA Metrics for Team and Individual Efficiency analysis www.qaexperts.pro
  62. 62. Developers and Dev Team efficiency REOPENED: Reopened Bugs Ratio % = Reopened bugs vs Resolved bugs Reopened Bugs Ratio % = Reopened Stories vs Stories delivered to testing STORY POINTS DELIVERED STORY DEFECTS ISSUED STORY DEFECTs/STORY POINTs rate % QUALITATIVE METRICS: Gather qualitative metrics about developers: approach to coding, complexity of code, contribution to innovation, communication efficiency, quick reaction to problems, etc.  Per PROJECT in general  Per Team  Per specific Developer www.qaexperts.pro
  63. 63. QAs and QA team Efficiency REJECTED Absolute Number of rejected bugs: All resolutions, Not a bug, Duplicate Cannot reproduce Incomplete Rejected Rate % = rejected vs created BUGs LEAKS: to PROD other later stages of development. Break down by Per product part Per functionality So you can correlate it with a certain QA or group of QAs Separately:  For critical and blocker  For Lower priority defects  Per PROJECT in general  Per QA Team  Per specific QA www.qaexperts.pro
  64. 64. QAs and QA team Efficiency WORK DONE:  Stories Tested  Created bugs Critical and Blocker (not rejected)  Created bugs Major and lower priority (not Rejected)  Test cases Created (passed DoD). Test cases rejected.  Test cases Executed.  Automation scripts Created (passed DoD). Test cases rejected.  Advanced test artifacts created and activities accomplished: test suites, plans, release support, etc. QUALITATIVE METRICS: Gather qualitative metrics about QAs: Complexity of tasks, Communication efficiency, Proactivity, Contribution to innovation, Test management skils, Conflict solving, etc.  Per PROJECT in general  Per QA team  Per specific QA  For Sprint (Velocity)  Trends  General value www.qaexperts.pro
  65. 65. QA Metrics for Test Automation www.qaexperts.pro
  66. 66.  Unit test coverage & number of test  API testing coverage & number of test for product in general by component  GUI Automation testing coverage & number of test  for product in general  by product part  by platform/device  by type of test end-to-end test Smoke test Regression test UAT test  by test priority: Priority 1 Priority 2 Priority 3 Automation Coverage & Results www.qaexperts.pro
  67. 67. Automation Efficiency AUTOMATION SPEED:  Automation tests Frequency  Duration of tests Running  Velocity of script creation and maintenance  Time for script creation and maintenance AUTOMATION EFFICIENCY: Stability of automation False Positives Root Cause for failure Bugs found by Automation Critical and Blocker Other bugs Bugs missed with automation. Critical and Blocker Other bugs ROI – Time automation economy, etc. Cost per automated test  Per PROJECT in general  Per QA team  Per specific QA  For Sprint (Velocity)  Trends  General value www.qaexperts.pro
  68. 68. QA Metrics TOOLS www.qaexperts.pro
  69. 69. Tools  Jira + Special time- related JQL  Confluence + Jira and other Macros in it  Spreadsheet processors: Googlesheets, Excel  TestTrail www.qaexperts.pro
  70. 70. JiraReportingInGsheets Tool Link to test version: https://chrome.google.com/webstore/detail/jirareportingingsheets/jmjoohlbcigmigokjldgcgnfofcagock JiraReportingInGSheets add-on for GoogleSheets allows to automatically build custom jira reports in google sheets Created by QAExperts.pro www.qaexperts.pro
  71. 71. Thank you Darya Alymova and Sergey Sosnitskiy for their contribution to certain metrics definition and implementation www.qaexperts.pro
  72. 72. QA Experts.Pro Workshops 2 days workshop November 2017 http://qaexperts.pro/#signup www.qaexperts.pro
  73. 73. Thank you www.qaexperts.pro

×