Project Management Office
Impact Measures Program
Sample 1

Agenda
A d
Which Measures Collected
Sample Reporting
Project L
P j Level, Organizational L
l O i i l Level
l
Perception versus Facts
Trends
By Year, MAP Impact, Preliminary Offshore Results
Estimation Validation Example
Next S
Steps Planned
l d
Global productivity measures
Consolidation and Automation in EPIC
2

Impact Measures Activities
I tM A ti iti
10 Project Impact M
P j I Measures
Projects Sample over 1,000 Effort Hours
Sampling = 20% of BUNA CIO Group Spend
excluding Infrastructure
Reported after Implementation
Also Available upon Request
Project Estimation Validation
Application Support Productivity – Support FTE
per function installed
3

Impact Measures Collected
I tM C ll t d
Impact Measure Calculation / Raw Data
Estimating Accuracy - (actual labor hours - estimated) ÷
1 Effort estimated
Estimating Accuracy - (actual calendar months -
2 Schedule estimated) ÷ estimated
*
function points produced ÷
3 Productivity FTE month
* 4 Unit Cost project cost ÷ function points
* 5 System Delivery Rate function points ÷ calendar months
(requirements added, changed,
deleted) ÷
6 Requirements Volatility total requirements in design
7 Client Satisfaction ratings by client area
defects found in system test ÷
8 System Test Effectiveness total project defects
defects found in the first 30 days
* 9 Production Defect Density of production ÷ function points
total development costs ÷ actual
10 Cost per Hour labor hours
* Measures using functionality count
4

Impact Measures
Project Level Reporting
October 30, 2006 5

INDIVIDUAL Project Scorecard
Sample – 1 print out
Adobe Acrobat
Document
6

INDIVIDUAL Project Scorecard Sample
7

Perceptions vs Fact
October 30, 2006 8

Print copy only – don’t present
py y p
Use actual printouts below
Single release funded by 2 clients
g y
Overlapped some development / testing
Similar results
Different client perception
ff
Adobe Acrobat Adobe Acrobat
Document
Document
October 30, 2006 9

Individual Project Scorecard –
Sample 2
10

Individual Project Scorecard –
Sample 3
11

Perception vs Facts
Here are Projects (>30,000 Hrs)
BTM Back Office
B k Offi
Core Teller / BTM Front Office
eCommerce 2005
C
Fin’l Fusion Upgrade (OLB)
HELOC
Trade Based Accounting
Wires Ph1
Wi
Wires Ph 2 & 3
Which performed the “best”?
12

Perception vs Fact – Hours?
p
Projects sorted by Size in Hours Hrs Est *
Wires Ph 2 & 3 75K
Trade Based Accounting 55k
HELOC 41K 34K
eCommerce 2005 38k 36K
Core Teller / BTM Front Office 35k 27K
BTM Back Office 30k
Wires Ph1 28k
Fin l
Fin’l Fusion Upgrade (OLB) 27k
* Binding estimate at the end of design
13

Perception vs Fact – Functionality ?
p y
Projects sorted by Size in FPs Hrs FPs
Wires Ph 2 & 3 75K 13536
eCommerce 2005 38K 7587
HELOC 41K 5680
Core Teller / BTM Front Office 35K 5224
Trade Based Accounting 55K 3124
Fin’l Fusion Upgrade (OLB) 27K 2112
BTM Back Office 30K 1212
Proportion of Size in Hours NOT proportional to Size in FPs
P ti f Si i H ti l t Si i FP
14

Perception vs Fact – Productivity?
p y
Productivity
Projects sorted by Productivity Hrs Size-FP (FPs/FTE month)
eCommerce 2005
C 38K 7587 FP
FPs 29.0
29 0
Wires Ph 2 & 3 ** 75K 13536 26.1
Core Teller / BTM Front Office** 35K 5224 21.7
HELOC 41K 5680 20.3
20 3
Fin’l Fusion Upgrade (OLB) 27K 2112 11.3
Trade Based Accounting ** 55K 3124 8.3
BTM Back Office** 30K 1212 5.9
Wires Ph1** 28K 1114 5.7
Industry avg = 16 The Organizationavg = 23 - 34
y g g g
** New Dev or Vendor pkg -- Expect below avg productivity
15

Organization Level Reporting
October 30, 2006 16

ORGANIZATIONAL TRENDS
* Improved since 2002
• Estimates – +50%
• Reqts Volatility – +60%
• Production Defects - +40%
2008
• Cost per hour – +15%
Global?
X Decline since 2002
• Satisfaction – -22% 2005
MAP
& Offshore
PreMap
Before
2006 – 7
2003
Transition ?
2003-04
Transition
17

Executive S
E ti Summary -- O
Overall
ll
Print this report
Adobe Acrobat
Document
18

Measurement Trends by Y
M t T d b Year
+
+
+ Improved since 2002
• Estimates – > +50%
• Reqts Volatility – >+70% +
• Production Defects - +65% x
• Cost
C t per hour – +20%
h
x Decline since 2002
• Satisfaction – -16%
+
19

Process Impacts – Pre- vs Post-MAP
+
+
+ MAP improvements
• Estimates – >+35%
• Reqts – +55%
•Production Defects - +30%
• Cost per hour – +20% x
x Declined with MAP
• Satisfaction – -14% +
+
20

Offshore Impacts
x
+ Improved Offshore: =
• Cost per Hour -- -28%
• System Delivery (1) -- +70% +
x Declined Offshore (2)
• Productivity -- -30%
= Same
• Unit Cost
+
Notes:
(1) Lower ABN Delivery due to ABN Resource Limit working on multiple projects in a given calendar month
(2) Lower Offshore productivity due Harvest staff turnover with each release
21

Project E ti ti Validation
P j t Estimation V lid ti
Requirements and Design Documents were used to determine
number of Function Points
3 Vendors and The Organizationcounted FPs from Software
Requirements Spec
Results:
# FPs Bid $ / FP
Vendor 1 1200 $ 685K $ 571
Vendor 2 * 520 $ 392K $ 754 x
Vendor 3 540 $ 259K * $ 480 *
ABN AMRO 585
22

Global Productivity Measures
Gl b l P d ti it M
Productivity Global Project
Only 2 BUs do Function Point (FP) counting –
NL and NA
BUNA on steering committee setting up FP
counting practices worldwide
Key performance metric in the Harvest Vendor
contracts
Selected Global Vendor for FP Counting
Reduce cost by eliminating in-house FP
y g
consultant
Piloting new vendor now
Expect vendor to be ramped up by end of
November
No embe
23

Plan to Embed Measures into Clarity
Automate Data Collection in 2007
Add fields to Clarity
Connection to requirements, testing, and
deployment tools
Automatic calculation of measures
NTO Executive Management Reporting
and Visibility
Individual project measures
Organizational rollups of measures
Executive Reporting
24

Appendix
A di
October 30, 2006 25

Oct
O t 2006 S
Spend vs M
d Measured
d
26

Executive Summary
With Min/Max -- Appendix
Adobe Acrobat
Document
27