This document discusses using agile metrics to measure and improve different dimensions of an agile engagement, including cost, time, and quality. It outlines the need for defining KPIs and metrics, preparing for measurement, collecting and analyzing measurement data, and using the analysis to set improvement targets. Examples of potential metrics are provided, such as defect removal efficiency, productivity, and effort variance. The document also shows an example of normalizing sprint data and using regression analysis to establish a baseline and improvement goals for defect removal efficiency.

1. Together. Free your energies
Agile Metrics
Measuring & Improving Different Dimensions
Niteen Kumar
Capgemini Netherlands

2. 2AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
IT?
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
STAKE HOLDER
MANAGEMENT TRANSPARENCY
COMMON
UNDERSTANDING
DECISION
MAKING

3. 3AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
SPECIFIC MEASURABLE ATTAINABLE RELEVANT TIME BOUND
S M A R T
CANT IT BE MEASURED?
DOES IT COST TOO MUCH?
BY WHEN?
WHAT STORY WILL IT TELL?
ARE MEASUREMENT TARGET ORIENTED?
COST TIME QUALITY

4. 4AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 % SLA COMPLIANCE
 PRODUCTIVITY
 COST PER INCIDENT
% FIRST TIME RIGHT
 % INCIDENT REDUCTION
 DFECT DENSITY
 SCHEDULE VARIANCE
 CYCLE TIME
 DEFECT REMOVAL EFFECIENCY
 % ACCEPTANCE OF USE CASES
 DFECT DENSITY
 CONTRIBUTION MARGIN
 PRODUCTIVITY (AET)
 COST PER INCIDENT
 CONTRIBUTION MARGIN
 PRODUCTIVITY
 EFFORT VARIANCE
 % COMPLETION STORY POINT
 DEFECT REMOVAL EFFECIENCY
 % ACCEPTANCE OF USE CASES
 CONTRIBUTION MARGIN
 VELOCITY
 EFFORT VARIANCE
DEVELOPMENT
AGILE
COST TIME
MAINTENANCE
QUALITY

5. 5AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
TAILORING
 PROCESS
 WORK BREAK DOWN STRUCTURE
 DEFECT REGISTRATION
 METRICS SHEET
 TOOLS – CLARITY | TEAMFORGE
WBS  REQUIREMENT ANALYSIS
 DESIGN
 CODING
 TESTING
 REVIEW
 REWORK
DEFECT
 TYPE
 SEVERITY
 PHASE INJECTED
 ROOT CAUSE

6. 6AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
WHAT WOULD YOU LIKE TO KNOW?
RELATIONSHIP
DISTRIBUTION
COMPARISON
 AMONG ITEMS?
 OVER TIME?
 TWO VARIABLE
 THREE VARIABLE
 SINGLE VARIABLE
 TWO VARIABLE
 THREE VARIABLE

7. 7AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 MEASUREMENT USED
Following KPI’s and related sub processes are used on monthly basis to monitor Cost,
Time and Quality aspects of the engagement.
“Y” FACTOR OR LAGGING INDICATORS USL UCL LCL LSL
DEFECT REMOVAL EFFECIENCY
“X” FACTOR OR LEADING INDICATORS USL UCL LCL LSL
SCOPE EFFORT
TESTING EFFORT
DESIGN EFFORT
MODELING EFFORT
% TEST CASES EXECUTED AGAINST PLANNED

8. 8AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 RAW DATA
2015 SPRINT LEADING AND LAGGING INDICATOR
COST QUALITY DETAILS QUALITY
SPRINT
NUMBER
i
Total # of
Features /
Use Case/
Stories
Planned
i
Estimated
Size
(Story
Points)
i
Actual
Completed
Size
(Story
Points) or
Sprint
Velocity
i
Total Planned
Effort
(P.Hrs) or
Total
Capacity
(P.Hrs)
i
ACTUAL EFFORT For Below Activities
(Person Hours)
P.Hrs
Total
Actual
Effort
(P.Hrs)
i
Total # of
Features /
Use Case/
Stories
COMPLETE
D
i
Total # of
Features /
Use Case/
Stories
ACCEPTED
i
%
Acceptance
of Use Case
Overall
Effort
Variance in
%
i
Total
Number Of
Planned
Test Cases
i
Total
Number Of
Test Cases
Executed
i
Total
Number Of
INTERNAL
Defects
i
Total
Number Of
EXTERNAL
Defects
i
Total
Defects
DRE
Defect
Density
(Defects/
Actual Size)
i
Resource
Utilization %
% of Story
Completion
Productivity
TOTAL AT ENGAGEMENT LEVEL g 452,00 360,50 2508,00 744,00 232,00 4296,50 639,00 480,00 330 292 216 182
i i i i h SCOPE DSGN MODL TST-E
SCRUM
MASTE
R
h h h i h h h h
9,00 20 112 88 240,00 24,00 21,00 144,00 40,00 12,00 241,00 20,00 20,00 100% 0% 26 61 87 30% 0,99 100% 100% 2,74
10,00 17 63 55 240,00 24,00 22,50 143,00 32,00 12,00 233,50 10,00 3,00 30% -3% 11 1 12 92% 0,22 97% 18% 4,25
11,00 20 156 99 240,00 24,00 11,00 140,00 40,00 12,00 227,00 16,00 5,00 31% -5% 10 5 15 67% 0,15 95% 25% 2,29
12,00 43 121 97 240,00 24,00 28,00 140,00 32,00 12,00 236,00 38,00 10,00 26% -2% 12 7 19 63% 0,20 98% 23% 2
13,00 27 144 123 240,00 23,00 22,50 144,00 40,00 13,00 242,50 26,00 10,00 38% 1% 5 5 10 50% 0,08 101% 37% 1,97
14,00 42 110 101 220,00 23,00 24,00 112,00 40,00 13,00 212,00 40,00 20,00 50% -4% 12 12 24 50% 0,24 96% 48% 2
15,00 46 147 137 220,00 23,00 7,00 144,00 40,00 13,00 227,00 41,00 28,00 68% 3% 18 15 33 55% 0,24 103% 61% 1,66
16,00 18 144 103 240,00 23,00 23,00 144,00 40,00 13,00 243,00 14,00 13,00 93% 1% 10 47 57 18% 0,55 101% 72% 2
20,00 109 150 157 185,00 24,00 24,00 85,00 40,00 12,00 185,00 108,00 99,00 92% 0% 9 2 11 82% 0,07 100% 91% 1,18
21,00 19 148 148 240,00 24,00 30,50 144,00 40,00 12,00 250,50 19,00 16,00 84% 4% 1 0 1 100% 0,01 104% 84% 2
22,00 57 140 140 240,00 24,00 30,00 136,00 40,00 12,00 242,00 57,00 41,00 72% 1% 35 29 3 0 3 100% 0,02 101% 72% 1,73
23,00 49 42 116 240,00 24,00 15,00 144,00 40,00 12,00 235,00 48,00 42,00 88% -2% 24 24 9 3 12 75% 0,10 98% 86% 2
24,00 33 105 125 240,00 24,00 20,00 88,00 40,00 12,00 184,00 33,00 28,00 85% -23% 41 37 9 0 9 100% 0,07 77% 85% 1,47
25,00 45 83 140 240,00 24,00 20,00 158,00 40,00 12,00 254,00 33,00 28,00 85% 6% 32 29 9 0 9 100% 0,06 106% 62% 2
26,00 29 170 130 240,00 24,00 16,00 140,00 40,00 12,00 232,00 25,00 22,00 88% -3% 44 44 14 1 15 93% 0,12 97% 76% 1,78
27,00 20 95 60 240,00 24,00 14,00 124,00 40,00 12,00 214,00 20,00 18,00 90% -11% 36 36 31 7 38 82% 0,63 89% 90% 4
29,00 28 115 112 220,00 24,00 8,00 122,00 40,00 12,00 206,00 26,00 17,00 65% -6% 41 25 5 1 6 83% 0,05 94% 61% 1,84
30,00 31 118 64 220,00 24,00 12,00 140,00 40,00 12,00 228,00 24,00 24,00 100% 4% 40 38 16 8 24 67% 0,38 104% 77% 4
31,00 47 122 96 220,00 24,00 12,00 116,00 40,00 12,00 204,00 41,00 36,00 88% -7% 37 30 6 7 13 46% 0,14 93% 77% 2,13

9. 9AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 NORMALISED DATA
Sprint 9-21 data has been used to arrive at the Defect Removal Efficiency (DRE) PPM.
 Y = DEFECT REMOVAL EFFICIENCY (DRE)
 X = Scope Effort/Size & Test Effort / Size
DRE Cycle Time / Size Scope / Size Design / Size Model / Size Test Effort / Size
0.30 0.07 0.27 0.24 1.64 0.32
0.92 0.11 0.44 0.41 2.60 0.58
0.67 0.06 0.24 0.11 1.41 0.40
0.63 0.06 0.25 0.29 1.44 0.33
0.50 0.19 0.19 0.18 1.17 0.33
0.50 0.06 0.23 0.24 1.11 0.40
0.55 0.15 0.17 0.05 1.05 0.29
0.18 0.06 0.19 0.22 1.40 0.19
0.82 0.04 0.55 0.15 0.54 0.33
1.00 0.04 0.56 0.21 0.97 0.44

10. 10AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 CONTROL CHART FOR DRE “Y” AND “X” FACTORS - SPRINT 9-21
Regression Analysis: DRE versus Scope/Size, Test Effort/Size
The regression equation is : DRE = - 0.118 + 0.941 Scope/size +
1.20 Test ff/size
Predictor Coef SE Coef T P
Constant -0.1179 0.1627 -0.72 0.492
Scope/Size 0.9414 0.3449 2.73 0.029
Test Effort/Size 1.1994 0.5003 2.40 0.048
S = 0.134968 R-Sq = 79.0% R-Sq(adj) = 73.0%

11. 11AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 DEFECT REMOVAL EFFICIENCY – (DRE) PPM Baseline & Improvement Goal for Next Sprints
1.0 Defect Removal Efficiency (DRE) = - 0.118 + 0.941 Scope/Size + 1.20 Test Effort/Size
ENGAGEMENT XXX DRE PPM -IT 1 DASH BOARD – SPRINT 1 TO 21
DRE (SPRINT 1-21)
PPM
- 0.118 + 0.941 Scope/Size + 1.20 Test Effort/Size
SPRINT 1-21 BASELINE UNIT UCL AVG LCL STD
DEFECT REMOVAL EFFECIENCY % 1.27 0.6 -0.06 0.22
SCOPE EFFORT HRS/SIZE 0.58 0.3 0.03 0.09
TEST EFFORT HRS/SIZE 0.66 0.36 0.05 0.10
IMP TARGET USL TARGET LSL STD
DEFECT REMOVAL EFFECIENCY % 1.15 0.85 0.55 0.1
IMPROVEMENT TARGET USL AVG LSL STD
SCOPE EFFORT HRS/SIZE 0.55 0.4 0.25 0.05
TEST EFFORT HRS/SIZE 0.65 0.5 0.35 0.05
WHAT IF UCL 1.18
WHAT IF AVG 0.86
WHAT IF LCL 0.54

12. 12AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 SPRINT 22-31 NORMALISED DATA FOR “Y” AND “X” FACTORS
 Y = DRE
 X = Scope/Size & Test effort / Size
DRE Cycle Time / Size Scope / Size Design / Size Model / Size Test Effort / Size
1.00 0.04 0.17 0.21 0.97 0.29
0.75 0.03 0.21 0.13 1.24 0.34
1.00 0.05 0.19 0.16 0.70 0.32
1.00 0.04 0.17 0.14 1.13 0.29
0.93 0.05 0.18 0.12 1.08 0.31
0.82 0.01 0.40 0.23 2.07 0.67
0.83 0.03 0.21 0.07 1.09 0.36
0.67 0.09 0.38 0.19 2.19 0.63
0.48 0.06 0.25 0.13 1.21 0.42

13. 13AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 CONTROL CHART FOR “Y” AND “X” FACTORS (SPRINT 22-31)

14. 14AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 STAGED CONTROL CHART & 2 SAMPLE t TEST FOR “Y” DRE
Two-Sample T-Test and CI: Sprint 9-21, Sprint 22-31
Two-sample T for Sprint 9-21, Sprint 22-31
N Mean StDev SE Mean
DRE Sprint 9-21 10 0.607 0.259 0.082
DRE Sprint 22-31 9 0.832 0.177 0.059
Difference = mu (DRE Sprint 9-21) - mu (DRE Sprint 22-31)
Estimate for difference: -0.225
95% CI for difference: (-0.440, -0.010)
T-Test of difference = 0 (vs not =): T-Value = -2.23 P-Value = 0.042 DF = 15

15. 15AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 STAGED CONTROL CHART & 2 SAMPLE t TEST FOR “Y” DRE
% DEFECTS REPORTED BY CUSTOMER = 58% % DEFECTS REPORTED BY CUSTOMER = 21%

16. 16AGILE ENGAGEMENT - SPRINT QUALITY IMPROVEMENT NITEEN KUMAR
WHY DO WE NEED
THEM
DESIGNING
KPI’S
PREPARING FOR
MEASUREMENT
MEASUREMENT &
ANALYSIS
 STAGED CONTROL CHART & 2 SAMPLE t TEST FOR % ACCEPTANCE OF
USE CASE BY CUSTOMER
% USE CASE ACCEPTANCE 61% % USE CASE ACCEPTANCE 84%
Two-sample T for % Acceptance of Use Case vs %
Acceptance of Use Case_1
N Mean StDev SE Mean
% Acceptance of Use Case 10 0.613 0.293 0.093
% Acceptance of Use Case 9 0.845 0.102 0.034
Difference = mu (% Acceptance of Use Case) - mu (%
Acceptance of Use Case_1)
Estimate for difference: -0.2317
95% CI for difference: (-0.4491, -0.0144)
T-Test of difference = 0 (vs not =): T-Value = -2.35 P-
Value = 0.039 DF = 11

17. The information contained in this presentation is proprietary.
© 2012 Capgemini. All rights reserved.
www.capgemini.com
About Capgemini
With more than 120,000 people in 40 countries, Capgemini
is one of the world's foremost providers of consulting,
technology and outsourcing services. The Group reported
2011 global revenues of EUR 9.7 billion.
Together with its clients, Capgemini creates and delivers
business and technology solutions that fit their needs and
drive the results they want. A deeply multicultural
organization, Capgemini has developed its own way of
working, the Collaborative Business ExperienceTM, and
draws on Rightshore ®, its worldwide delivery model.
Rightshore® is a trademark belonging to Capgemini