It is quite possible that you are spending a considerable amount of your time as a QA manager making sense of the multitude of metrics reported by your teams, connecting the facts, understanding the underlying reality, and articulating it to your peers and leadership. Still, others in the organization may not interpret the message correctly, rendering most of your efforts futile. Nirav Patel and Sutharson Veeravalli share insights to help you resolve this challenge through a composite measure called Quality Index. By aligning metrics to business outcomes and using Quality Index as a tool of articulation, disparate interpretation of data can be eliminated and a cohesive message delivered to stakeholders. Learn how QA can acquire a voice across the senior forums by articulating succinct, contextual, and actionable information to speed up executive decisions in the course of programs and projects.

1. ©2015, Cognizant Technology solution 1
Quality Index
A Composite Metric for the Voice
of Testing
October 1, 2015
Nirav Patel - Walgreens Boots
Alliance
nirav.x.patel@walgreens.com
Sutharson Veeravalli – Cognizant
Technology Solutions
Sutharson.S.V@cognizant.com

2. ©2015, Cognizant Technology solution 22©2015, Cognizant Technology solution
Speaker(s) Bio
A director at Cognizant’s Quality Engineering & Assurance (QE&A) practice, Sutharson
Veeravalli manages large delivery organizations and specializes in implementing
transformational initiatives that enhance the delivery ecosystem. Sutharson has twenty years
of experience spanning banking and retail domains, process and domain consulting, large
system delivery, product management, and QA transformation for clients. He brings a diverse
experience in leading multiple transformational initiatives that have delivered multimillion
dollar savings to clients
Nirav Patel, the QA manager of QAE for pharmacy, retail, and health and wellness divisions at
Walgreens, has been in QA for more than fifteen years. Specializing in metrics-driven
governance and financial control, Nirav drives the concept-to-implementation process of the
transformational initiatives across his portfolio. He is an active advocate of quality
engineering to drive consistency and efficiency within the practice.
Nirav
Sutharson

3. ©2015, Cognizant Technology solution 33
The Curious case of information overload...
GUARD [on screen]: I've sealed the compartment. It can't get out, that lock's got a billion combinations.
DOCTOR: A Dalek's a genius. It can calculate a thousand billion combinations in one second flat.
DOCTOR: It's not just energy. That Dalek just absorbed the entire internet. It knows everything
Doctor Who - 2005 Series, Episode 6
Blessed are the ones who have Daleks as their project managers… rest of us have to still deal with the information overload in managing projects
How many times have you felt…???
Overwhelmed at the amount of data & metrics
produces by the teams. and yet not getting a
holistic view on project health ?
Unsure if in-progress & completed
programs are on track to deliver the
intended business outcomes?
Limited in conveying the project/program
status to non-technical business & senior
management stakeholder? Challenged in articulating the quality
assurance view to external stakeholders?
©2015, Cognizant Technology solution

4. ©2015, Cognizant Technology solution 44
Why will traditional metrics & articulation not solve our challenges?
Business
Expectation
Simple non technical view on
program/project status
Standardized, contextual &
actionable information
Information aligned to outcomes
to be delivered
QA challenges
Delivery metrics requires some
level of understanding QA
domain
Plethora of metrics , too difficult
to summarize
Voice of QA is limited and mostly
overridden by program
pressures
Lack of business outcome view.
Focus is more on tactical
delivery
QA should restrict to test
execution and achieving end
date
Solution
Synergize bottom up metrics to
create a multi dimensional
view that uses a simple
composite metric to succinctly
summarize status against
expected outcomes,
strengthening the “Voice of
QA”
Standardized, business outcomes aligned metrics
framework that uses “Quality Index” as voice of QA
©2015, Cognizant Technology solution

5. ©2015, Cognizant Technology solution 55
Multi-dimensional Metrics Framework
• QUALITY INDEX based reporting
• A unified view of quality depicting
the quality level against
benchmark
Business outcome aligned
• At highest level metrics are
aligned to business outcome
• Execution predictability
• Product stability
• Learning & skills
• Financial control
Broader Measures
• Measures not restricted to single
phase.
• Specific measure (called quality gauge)
defined for each phase of SDLC
Contextual insights
• Metrics collated contextual to stakeholder
• Level 1: for QA stakeholders
• Level 2: SDLC stakeholders
• Level 3: Business stakeholders
Objective :
To create a unified metrics framework to
• Standardize metrics, data sources, tools and process to capture data
• Unified reporting for multi-level (L1, L2 and L3), contextual management status
• Facilitate single view of Quality through ‘Quality index (QI)’ (Project, Program and release level) & Centralized dashboard
Single view of quality
©2015, Cognizant Technology solution

6. ©2015, Cognizant Technology solution 66
Build maturity Requirements maturity QA efficiency QA Quality Upstream quality
Broader Measure & Contextual Insights
Requirements Analysis Test Design Test Execution & ClosureDesign & Build
Measure
SDLC
Stakeholders
QA
Stakeholders
 Process adherence
 Automation/tooling
percentage
 Improvement
defects found in
DIT
 Build success rate
 Deployment time
 Requirements traceability
 Test design productivity
 Rework effort (due to QA)
 Review Efficiency
 Test design productivity
(Auto)
 Performance test design
productivity
 Build to QA spend ratio
 Requirements
Implementation index
(based on static
testing defects
 Test execution
productivity
 Test execution
productivity
 Performance test
execution
productivity
 Test Efficiency
=(1-defect
rejection)
 Requirement stability
index (RSI)
 Defect reopen rate
 Environment
downtime
 Schedule Variance
 Business time to
market
 QA committed
schedule
©2015, Cognizant Technology solution

7. ©2015, Cognizant Technology solution 77
Business Outcome aligned metrics with QI
Benefits quantified
through Single unit of
measure
“Y-o-Y Improvement
on Cost/TCP”
Quality
Index
Quality
Index
Quality
Index
Quality
Index
Execution
Predictability
Learning & Skills
Production Stability
Financial Control
• Requirements implementation index
• Requirement stability index (RSI)
• Defect reopen rate
• Test environment downtime
• Schedule Variance
• Effort variation (due to QA)
• Risk Aligned test coverage
• Maturity of build Organization
• Team Competency
Index
• Review Efficiency
• Innovation Index
• Requirements Traceability
• UT/DIT coverage
• UT/DIT test results
• Defect leakage to production
• Application defect density
• Test Design productivity (manual,
automation, performance)
• Test Execution productivity (manual,
automation, performance)
• Rework Effort(Due to QA)
• Test Effectiveness
• Build to QA ratio
• % CAPEX reduction through virtualization
• Automation Coverage
What is the improvement ?
©2015, Cognizant Technology solution

8. ©2015, Cognizant Technology solution 8©2015, Cognizant Technology solution 8
Quality Index – Voice of the QA
Quality Index (QI) is the composite metric used to
summarize the quality of each measure
A QI of 1 indicates the measure is in alignment with the
expected benchmark value.
How is QI Calculated ?
A measure (e.g. Execution predictability) is linked to influencing metrics
Every metric is measured against the expected benchmark
The QI index is calculated as 1, if the metric value is equal to the expected benchmark level
Based on QI value a RAG status is assigned:
QI >1 is Green, > 0.8 and < 1 is Amber and < 0.8 is Red
The individual metric level QI is rolled up to measure level by assigning weightages to each
metric
How is QI rolled up ?
Quality index is calculated at project level and rolled up through
business as below
How is QI represented?
QI is represented as dial for current value and also trend graph
Project 1 Project 2 Project nProject 3
Release 1 Release 2 Release n
LOB 1 LOB 2 Program
Project 4

9. ©2015, Cognizant Technology solution 99
Weightage
15%
Benchmark
5%
Weightage
25%
Benchmar
k 8%
Weightage
10%
Benchmark
1%
Weightage
25%
Benchmar
k 5%
Weightage
15%
Benchmark
5%
Example of Calculating Execution Predictability
% Rework Effort due to change in
requirements - Measure of the effort going
on in rework of test artifacts
(Total Rework Effort
(across all phases)) /
(Actual Effort expended
for the project )*100
Tool #1
Tool #2
%
Execution Predictability - This view helps to analyze, evaluate & infer the execution predictability of the Project from QAE point of view. It helps to infer if the
execution measures/metrics are on track & to help to change course correction based on deviations, if any. The metrics that are captured as part of this view are as
listed below
Defect Reopen rate - Measures the
quality
(No of defects reopened
post retest/no of valid
defects logged)
Tool #1 %
Test Environment Downtime -
Measures availability of test
environments to test teams
(# of person days lost
due to environment
downtime/ (total planned
test execution person
days + planned test data
setup person days))*100
Tool #1 %
Schedule Variance - Provides a
measure of deviation of the actual
schedule against the planned schedule
Actual end date – Planned end
date) / Planned end date –
Planned start date) * 100
Tool #2 %
Requirement Stability Index -
Measure of the stability of requirements
(Number of Original
Requirements + Number of
requirements changed + Number
of requirements added + Number
of requirements deleted)/
(Number of Original
Requirements)
Tool #1 #
Effort Variance - Provides a measure
of the deviation of the actual effort
expended against the estimated effort
(Actual Effort -- Estimated
Effort) / Estimated Effort) *
100
Tool #2 %
Weightage
15%
Metric value is compared against the
benchmark. Meaning if “% Rework”
value is “9%” and Benchmark was
“4%” then QI is (4% / 9%) *
100% = 44%
Value from Step 1 * Weightage %
= 44% * 15% = 6.6%
Execution predictability =
SUMMATION of all Metrics Computed
Proportionally Weightage Index
Step1
Step2
Step3
Compute Base
Quality Index
Compute Proportionally
Weightage Index
Compute Execution
Predictability
Benchmark
4%
©2015, Cognizant Technology solution

10. ©2015, Cognizant Technology solution 1010
Typical QI based dashboard
©2015, Cognizant Technology solution

11. ©2015, Cognizant Technology solution 1111
In Summary
Provide contextual, actionable and relevant information to
stakeholders
Align messaging to ‘business outcomes’ as against ‘delivery status”
Have single view of quality using “Quality Index” like composite
metric
Scale the framework outside delivery into all QA goals
Business Value of QA can be demonstrated by
Quality Dashboard
Outcome Alignment
Multi-dimensional
Metrics Framework
Quality Index
©2015, Cognizant Technology solution

12. ©2015, Cognizant Technology solution 1212©2015, Cognizant Technology solution

13. ©2015, Cognizant Technology solution 13©2015, Cognizant Technology solution

14. ©2015, Cognizant Technology solution 1414
Outcome Based View
Expected outcome Measure Influencing metrics
Programs and projects needs to be
monitored for ON TIME delivery
Execution Predictability
% Rework Effort due to change in requirements
Defect reopen rate
Test environment downtime
Schedule Variance
Requirement stability index (RSI)
% Effort Variation
Measure quality of product under
test by BSS QAE
Measure quality of product
deployed post BSS QAE phase
Product Stability - Inflight
Product Stability – Post QA
Requirements traceability
Application Defect Density
UAT Defect Leakage (%)
Production Defect Leakage (%)
©2015, Cognizant Technology solution

15. ©2015, Cognizant Technology solution 1515
QI – Calculation sample
Measure Influencing Metrics Release 1 Release 2 Release 3 Release 4 Weightage Benchmark
Metric level
Quality Index
Measure level
Quality Index
Execution
predictability
% Rework Effort due to
change in requirements
0.3% 1.0% 9.0% 1.0% 15% 4% 150.0% 1.04
Defect reopen rate - 10.0% 12.0% 11.0% 20% 5% 0.0%
Test environment
downtime
12.0% 9.0% 9.0% 3.0% 25% 8% 66.7%
Schedule Variance 3.0% 10.0% - - 15% 5% 150.0%
Requirement stability
index (RSI)
- 1.10 1.07 1.30 10% 1 0.0%
% Effort Variation -0.4% 56.0% -6.0% -17.0% 15% 5% 122.0%
Production
Stability –
Inflight
Requirements
traceability
100.0% 100.0% 100.0% 100.0% 5% 100% 100.0% 1.21
Application Defect
Density
14.0% 12.0% 8.0% 11.0% 20% 1% 70.0%
Production
Stability - Post
QA
UAT Defect Leakage (%) 1.0% 0.2% 1.0% 0.1% 35% 3% 150.0%
Production Defect
Leakage (%)
2.0% 0.8% 1.0% - 40% 2% 100.0%
©2015, Cognizant Technology solution

16. ©2015, Cognizant Technology solution 1616
Financial
Control
Production
Stability
Execution
Predictability
Learning &
Skills
A View for Agile
 Requirements implementation index
 Requirement stability index (RSI)
 Defect reopen rate
 Test environment downtime
 Schedule Variance
 Effort variation
 Risk Aligned test coverage
 Maturity of Dev Organization
 Defect Density (Agile)
 % Scope Creep
 Testing Efficiency at the Sprint level  Test Design productivity (manual,
automation, performance)
 Test Execution productivity (manual,
automation, performance)
 Rework Effort(Due to QA)
 Test Effectiveness
 Build to QA ratio
 Offshore, Near shore leverage
 % CAPEX reduction through virtualization
 Velocity of the sprint
 Velocity rate
 Story point Productivity
 Requirements Traceability
 UT/DIT coverage
 UT/DIT test results
 Defect leakage to production
 Application defect density
 Testing Efficiency at the Sprint level
 Team Competency Index
 Review Efficiency
 Innovation Index
©2015, Cognizant Technology solution