This document discusses quality assurance metrics and trends in practice. It begins by providing examples of costly technology failures from bugs in various systems. It then defines various known QA metrics that can be classified as product, process, or QA metrics. The document discusses representing metrics and their dependencies using a graph theory approach to analyze and prioritize metrics based on their calculated weights and dependencies. It provides an example of applying this approach to define and analyze key metrics for a test project. The analysis reveals that certain metrics should be prioritized for improvement based on the calculated weights.

1. ©2015 InfoStretch Corporation. All rights reserved.
Liana Gevorgyan | May 6, 2015
Measuring
Quality
QA Metrics and Trends in Practice
US - UK - India

2. ©2015 InfoStretch Corporation. All rights reserved.
SECTION 1: TECHNOLOGY IN LIFE
Bugs Are Costly

3. 1999
Mars Climate Orbiter Crash
Instead of using the provided
metric system for navigation,
the contractor carried out
measurements using empirical
units and the space craft
crashed into Mars.
COST
$135 Million

4. 1996
ARIANE Failure
Ariane 5 rocket exploded 36.7
seconds after take off. The
engine of this satellite was
much faster than that of the
previous models, but it had a
software bug that went
unnoticed.
COST
>$370 Million

5. 2003
EDS Fails Child Support
EDS created an IT system
for a Child Support
Agency in the UK that had
many software
incompatibility errors.
COST
$1.1 Billion

6. 2013
NASDAQ Trading Shutdown
August 22, 2013 NASDAQ
Stock Market Shut down
trading for three hours
because of a computer
error.
COST
$2 Billion

7. 1985-1987
Therac-25 Medical Accelerator
A software failure caused
wrong dosages of x-rays.
These dosages were
hundreds or thousands of
times greater than
normal, resulting in death
or serious injury.
COST
5 Human Lives

8. Technology In Our Daily Life
Average usage of electronic systems in developed countries:
 One PC or desktop in each home.
 80% of people are using mobile phones
 40% of people are driving cars with various electronic systems
 People are traveling via train, plane on an average once a year
 Dozens of other embedded systems in our homes
 Dozens of software programs in our work place, service systems
Quality of all mentioned systems are equal to the Quality of life!

9. SECTION 2: DEFINING THE “WHAT”
Known QA Metrics & Trends

10. Defining “What”
10
 Metrics and Trends
 Measure to Understand
 Understand to Control
 Control to Improve

11. Several Known QA Metrics and Trends
11
 Manual & automation time ratio during
regression cycle
 Scripts maintenance time during delivery
iteration
 Daily test cases manual execution
 Automation effectiveness for issues
identification
 Issues found per area during regression
 Areas impacted after new features integration
 Issues identification behavior based on major
refactoring.
 Software process timetable metrics
 Delivery process productivity metric
 Software system availability metrics
 Test cases coverage
 Automation coverage
 Defined issues based on gap analysis
 Ambiguities per requirement
 Identified issues by criticality
 Identified issues by area separation
 Issues resolution turnaround time
 Backlog growth speed
 Release patching tendency and costs
 Customer escalations by
Blocker/Critical/Major issues per release
 QA engineer performance
 Continuous integration efficiency

12. Metrics Classification
12
PRODUCT METRICS
PROCESS METRICS
QA METRICS

13. Metrics Examples by Classification
13
 Delivery process productivity metrics
 Continuous integration efficiency
 Release patching tendency and costs
 Backlog growth speed
 QA engineer performance
 Software process timetable metrics
 Software system stability metrics
 Identified issues by criticality
 Identified issues by area separation
 Customer escalations by Blocker/Critical/Major
issues per release
 Ambiguities per requirement
 Backlog growth speed
PRODUCT METRICSPROCESS METRICS

14. Sample Metrics Visual
14
55%20%
15%
7% 3%
Automated UI and BE
Automated UI
In Progress
Pending Automation
Not Feasible
AUTOMATION COVERAGE BUGS BY SEVERITY
3 % 5 %
12 %
34 %
45 %
Blocker Critical High Medium Low

15. Visual Depiction Of Sample Trends
15
Blocker
High
Low
0
10
20
30
40
Blocker
Critical
High
Medium
Low
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
%
ISSUE ESCALATIONS BY CRITICALITY - MONTHLY TREND
1 2 3 4 5
6 WEEKS
REJECTED BUGS % PER WEEK

16. Expectations
Smooth releases
Predefined risks with mitigation plans
Nice feedback and appreciation
Top notch and innovative products
16

17. Real Life
 Delivery not always ideal
 We are familiar what is patching the release
 Lack of process tracking data for analysis
 Experimental delivery models not exactly the Best practice
models
17

18. SECTION 3: DELIVERY
PROCESSES & METRICS
Waterfall & Agile

19. Waterfall Process
REQUIREMENTS
VALIDATION
ARCHITECTURE
VERIFICATION
MODULE DESIGN
VERIFICATION
IMPLEMENTATION
SYSTEM TEST
OPERATIONS AND
MAINTENANCE
REVALIDATION

20. Agile Process
20
TESTING/VALIDATION/VERIFICATION
Product Backlog
 Client prioritized
product features
Sprint Backlog
 Features assigned
to Sprint
 Estimated by team
 Team Commitment
Working Code Ready For
DEPLOYMENT
Time-Boxed
Test/Develop
PRODUCT BACKLOG BACKLOG TASKS
Scrum Meetings
Every 24 hours

21. Agile Process Metrics
21
SCRUM TEAM SPRINT METRICS
Scrum Team’s Understanding
of Sprint Scope and Goal
Scrum Team’s Adherence to
Scrum Rules & Engineering
Practices
Scrum Team’s
Communication
Retrospective Process
Improvement
Team Enthusiasm
Quality Delivered to
Customer
Team Velocity
Technical Debt
Management
Actual Stories
Completed vs. Planned

22. Processes Are Not Always Best Practices
22
 Unique way of Agile
 Transition from Waterfall to Agile
 Transition from Agile to Kanban

23. Metrics Set Definition for Your Project
23
 Process
 Technology
 Iterations
 Project/Team Size
 Goal

24. SECTION 4: WEIGHT BASED ANALYSIS
FOR QA METRICS & MEASUREMENTS
Mapping With Graph Theory

25. Metric and Trends for your Project
25
You are watching Metrics/Trends set, are they the right ones?
Trends are in an acceptable range, but the products quality is
not improving?
Trying to improve one metric and another is going down?
How do you analyze and fix it?

26. Mapping QA Metrics Info Graph Theory
26
Process Metrics > A= Metric 1, B=Metric 2…
Actions/Data set that takes effect in Metrics > A1, A2…
Metric dependencies of specific action
Product Metrics > C= Metric 3, D=Metric 4…

27. Preconditions & Definitions For Metrics &
Actions mapped model
27
 Node’s initial weight is predefined and has value from 1-10
 Edge’s weight is predefined and has value from 1-10
 Connections between Nodes is defined based on dependencies of Metrics
from each other and from Actions
 All Actions have fixed 1 weight

28. Initial Metrics Model & Dependencies
28
Assume
Current Metric set is:
2 Process Metrics -> M1, M2
2 Product
Metrics -> M3, M4
Where :
M1 has dependency on M3
M1 has dependency on M4
M2 has dependency on M3
There are 3 Actions or Data sets that have effect on
some of the Metrics. Those are A1, A2, A3
Where :
M1 has dependency on A1 and A2
M4 has dependency on A3
Initial Priority
Initial Priority based on
Best Practices
W(M1) = 5
W(M2) = 4
W(M3) = 3
W(M4) = 2

29. Metrics Visualization via Graph
29
M2
M3
M1
4
3
M4
5
2
A2
A1
A3
Process Metrics > A= Metric 1, B=Metric 2…
Actions/Data set that takes effect in Metrics > A1, A2…
Metric dependencies of specific action
Product Metrics > C= Metric 3, D=Metric 4…

30. Weight Assignment On Undirected Graph
30
M2
M3
M1
4
3
M4
5
2
A2
A1
A3
2
3
5
1
1
6
1
1 1
Process Metrics > A= Metric 1, B=Metric 2…
Actions/Data set that takes effect in Metrics > A1, A2…
Metric dependencies of specific action
Product Metrics > C= Metric 3, D=Metric 4…

31. Calculation Formula for Metrics New Priority
31
 Priority of the node is calculated the following way:
where
 - initial priority of the node
 - node weight assigned by user
 - cumulative weight of each node's edges
𝑾(𝑨)

32. New Priority Calculations For One Metric
32
M2
M3
2
4
3
A2
A1
1
11
1
Initial Priority
M2 = W(M2)=4
New Priority
M2 = W(M2) * (W(M2-A1) + W(M2-A2) + W(M2-M3))
M2 = 4 * (1+1+2) = 4*4 = 16

33. New Priority Calculations For Graph
33
M1 M2 M3 M4 A1 A2 A3
W 5 4 3 2 1 1 1
M1 5 3 5 40
M2 4 2 1 1 16
M3 3 3 2 6 33
M4 2 5 10
CA LCULATIO NS
Metrics
New
Priority
M1
M3
M2
M4
Metrics
Initial
Priority
M1
M2
M3
M4

34. Metrics Priorities: Current Vs. Calculated
34
Initial Priority Based on
Best Practices
M1
M2
M3
M4
Project Dependent
Calculated Priority
M1
M3
M2
M4
INITIAL PRIORITY NEW PRIORITY

35. SECTION 5: METRICS WEIGHT
BASED ANALYSIS IN PRACTICE
Defining “How”

36. Metrics Definition For Test Project
36
 Process – Agile with Area ownership
 Technology – SAAS Based Enterprise Web & Mobile App
 Iteration – 2 weeks
 Project Size – 5 Scrum Teams
 Goal – Customer Satisfaction, No Blocker, Critical Issues Escalation
by Customer

37. Key Metrics and Dependencies
37
Metrics
M1 - Customer Escalations per defect severity – Product Metric
M2 – Opened Valid Defects per Area – Product Metric
M3 – Rejected Defects – Process Metric
M4 - Test cases Coverage – Process Metric
M5 - Automation Coverage – Process Metric
M6 - Defect fixes per Criticality – Product Metric
Actions and Data Sets
A1 – Customer types per investment and escalations per severity
A2 – Most Buggy areas

38. Metrics Initial Priority
Weight assignment and dependency analysis
38
Metric Name Predefined
Node
Weight
Metrics By
Initial
Priority
M1 - Customer Escalations per
defect severity
8 M1
M2 – Opened Valid Defects per
Area
5 M6
M3 – Rejected Defects 4 M2
M4 - Test cases Coverage 3 M3
M5 - Automation Coverage 2 M4
M6 - Defect fixes per Criticality
per Team
6 M5
Node
Weight
M1 M2 M3 M4 M5 M6 A1 A2
M1 = 8 2 6 4 5 2
M2 = 5 2 1 3
M3 = 4 1 3
M4 = 3 6 3 3 2
M5 = 2 2
M6 = 6 4 2

39. Graph Creation
39
3
M1
M3
M2
6
8
4
M4
5
A2
A1 2
5
1
1
3
M6
M5 26
4
3
1
2
2
2

40. Calculations and Metrics Prioritization
40
Node Weight M1 M2 M3 M4 M5 M6 A1 A2 Calculated
Priority
M1 = 8 2 6 4 5 2 152
M2 = 5 2 1 3 30
M3 = 4 1 3 16
M4 = 3 6 3 3 2 42
M5 = 2 2 4
M6 = 6 4 2 36
Initial
Priority
M1
M6
M2
M3
M4
M5
Calculated
Priority
M1
M3
M6
M2
M4
M5

41. Key Metric Changes & Improvement Plans
41
Metrics by Calculated Priority
M1 - Customer Escalations
per defect severity
M3 – Rejected Defects
M6 - Defect fixes per
Criticality per Team
M2 – Opened Valid Defects
per Area
M4 - Test cases Coverage
M5 - Automation Coverage
 Group Defect by Severity and per Customer investment to understand
real picture. 1000 Minor issues can cost more than 1 High severity issue.
 Proceed Trainings to low defect rejection, so developers will not spend
more time on analysis of invalid issues
 Make sure Defect fixes are going in Parallel with new feature
development for each sprint
 Continuously update Test case after each new issue, to make sure you
have good coverage
 Automate as much as possible to cut the costs and increase the coverage

42. Monitoring of Trend Based Priority Metrics
Based on Process Changes
42
60
70 68
75
20
25
29
34
70
80 82
78
0
10
20
30
40
50
60
70
80
90
Jan Feb March April
M1 M3 M6

43. Let the Challenge Begin… & Have FUN
43

44. Thank You
Global Footprint
About Us
A leading provider of next-gen mobile application
lifecycle services ranging from design and
development to testing and sustenance.
Locations
Corporate HQ: Silicon Valley
Offices: Conshohocken (PA), Ahmedabad (India),
Pune (India), London (UK)
InfoStretch Corporation

45. References
 Narsingh Deo, Graph Theory with Applications to Engineering and Computer Science, Prentice Hall 1974.
 A.A. Shariff K, M.A. Hussain, and S. Kumar, Leveraging un- structured data into intelligent information – analysis and
evaluation, Int. Conf. Information and Network Technology, IPCSIT, vol. 4, IACSIT press, Singapore, pp. 153-157, 2011.
 http://en.wikipedia.org/wiki/List_of_software_bugs
 http://www.starshipmodeler.com/real/vh_ari52.htm
 http://news.nationalgeographic.com/news/2011/11/pictures/111123-mars-nasa-rover-curiosity-russia-phobos-lost-curse-
space-pictures/
 http://www.bloomberg.com/news/articles/2013-08-22/nasdaq-shuts-trading-for-three-hours-in-latest-computer-error

46. ©2015 InfoStretch Corporation. All rights reserved. 46
Q & A
Liana Gevorgyan
Sr. QA Manager
InfoStretch Corporation Inc.
liana.gevorgyan@infostretch.com
www.linkedin.com/in/lianag/en
Info@infostretch.com
www.infostretch.com