Project Metrics & Measures

Project Metrics & Measures Roadmap to gain agreement on key metrics vital to measure project success Anand Subramaniam

“If Columbus had turned back, no one would have blamed him. Of course, no one would have remembered him either.”

- Source Unknown

Highlights

Project Metric & Measures - Overview

Project Metric & Measures - Roadmap

Project Process Metric - Model & Guidance

Project Metrics

Project Measures

Monitor & Control

Earned Value

Project Metric & Measures - Overview

Definition Metrics Provides measuring units to depict values, thresholds, constraints, scope, duration, maximums and minimums, averages, etc. Measures Represent information used to establish a common understanding of status, condition, and position of something

Definition Metrics A precise description of what is measured including: numerator, denominator, inclusions and exclusions Targets The agreed upon metric value the organisation desires to achieve

Metric - Challenges Metric Challenges No defined approach to measure return on investment No data validation for accuracy Lack of resources to perform function No formal approach to establish measures and metrics (program & projects) Lack data for reviewing metrics pre & post-live reporting No top management support or participation for metric definition & management No timeline for refining and reporting metrics in line with implementation activities Lack of organisation buy-in towards a single methodology and standardisation of data elements Lack continuous reinforcement, correlation of metrics to project outcomes Lack training / buy in

Metrics – Key Success Factors Key Success Factors Clear charter, scope, roles & responsibilities and direction vital to measure project success Consistent reinforcement of the terminology and metric definition / cal Gain agreement on key metrics for projects & program Governance process defined for metric management Metrics & measures clear and concise Support and ownership from business Dashboard” and summary reports developed pre & post implementation Gain agreement on reporting and ongoing monitoring Develop metric program early in the project Defining metrics at the beginning of the project

Project Metric & Measures - Roadmap

Metrics & Measures - Roadmap #1 Scope & Governance #2 Metric Identification #3 Baseline Measurement #4 Industry Benchmarks #5 Targets #6 Establish Reporting Process #7 Implement #8 (Post) Implement Review / Audit

#1 Governance & Scope

Process

Define scope – In / Out

Define governance process

Identify key individuals

Define authority and accountability levels

Define participants and reporting structure

Risks

Lack organisational understanding of scope

Lack participation

Lack commitment from steering committee team

Lack clear definition of authority (who can make decisions)

Metrics – Scope Validation Metrics – Identification & Validation

Involve stakeholders

Correlate to business goals and objectives

Develop a manageable list of metrics

Draft calculations and definitions

Perform industry research and recommend improvement

Set appropriate targets (lag & lead)

Create plans to achieve goals

Monitor and communicate results

Periodic review of improvement ranges

Recommend metric owners

Collect baseline data

Develop tools & techniques

Prepare comparative reporting tool

Generate & distribute report / changes

Operating Processes Management Process

#2 Metric Identification

Process

Define terminology

Basic definitions

Identify guidelines

Define what to consider

Ensure guidelines support project’s objectives

Define approach, metrics and calculations

Define metric criteria

Identify anticipated project impacts

Risks

Not gaining agreement on terminology and definitions

Not knowing what to measure

Lack standardisation on the approach

Lack well defined guidelines

Metric Identification - Criteria

Effort and cost required to obtain the metric target

Financial quantification associated with the metric

Availability of industry information to benchmark the metric

Whether or not the metric would have a negative impact on employees and culture

Reliability (believability) of the baseline data supporting the metric

#3 Baseline Measurements

Process

Measurement Criteria

Identify required sample size

Measurement period

Audit review process

Statistically validity

Identify how to obtain

Ensure consistency pre- and post-system implementation

Risks

No resources or man hours to collect baselines

No standardisation of data definition, source and / or the collection process

Measurement period not defined

No audit review process defined

#4 & 5 Industry Benchmarks & Targets

Process

Review terminology

Perform industry research

Obtain benchmark data

Define improvement ranges and tolerances

Establish targets

Identify Metric Owners

Gain approval

Risks

No time commitment for research

Improvement ranges lack clear definition

Tolerances not defined

No target defined and / or lack understanding

Targets not approved

#6 Establish Metric Reporting Process

Process

Define reporting approach

Define responsibilities as to who will collect, compile and validate data

Define executive reporting needs, training, frequency and distribution

Define data repository required

Define what will be done with the results

Risks

Lack ownership nor time commitment

Lack agreement on data sources

Executive reporting not defined and / or does not make sense

No training or education given on the reports

No data repository in place

#7 & 8 (Post)Implementation Review

Process

Implement metrics and compare target to actual metric reports

Review for consistency

Review for executive leader ownership and action plans

Activate internal audit validation

Risks

Lack agreement on source of variance and action plans

Lack resources for monitoring and reporting

Not a metric driven implementation project

Project Process Metric - Model & Guidance

Project Process Metrics Model Output Input Process Project Metrics Productivity Rate Test Defect Cost / Schedule Variance Change request

Attributes

People

Environment

Tools

Techniques

Time

Cost

Size

Defects

Resources & People Deliverables Time - Schedule Risk Risk Risk Risk Initiate Plan Design Build Test Control Close Operate

Guidance on Metrics

Set clearly defined goals

Train and allocate adequate resources

Focus on project issues (risk management)

Provide timely feedback on metrics analysis

Begin by only collecting a few key measurements

Obtain top management commitment and “buy-in”

Automate the collection process, as much as possible

Understand that metrics are not an end to themselves

Metrics cannot identify, explain or predict everything

Most issues / problems require more than one measurement to characterise and understand

Metrics should augment, not replace, good management judgment

Guidance on Measurement Program

Encourage feedback from participants

Must be institutionalised and repeatable

Develop a “Project / Program Measurement Plan”

Reward participation in the measurement program

Measurements - not analysed & acted upon is useless

Should be part of an overall process improvement effort

Plan for changes on the measurements types collected

Measurement results should remain anonymous

Do not measure individuals

Focus on the process and the product

Project Dashboard 20/07/2009 27/07/2009

Project Metrics

Metrics – Process Steps 1. Identify Metrics Customers 3. Ask Questions 4. Select Metrics 5. Standar-dise Definitions 6. Choose a Model 2. Target Goals 7. Establish Counting Criteria 9. Define Reporting Mechanisms 10. Determine Additional Qualifiers 11. Collect Data 12. Consider Human Factors 8. Decide On Decision Criteria

Focus - Project Metrics & Measures Schedule Quality Risk Cost / Budget Contracts Resources Focus - Metrics & Measure Status Stakeholder Involvement Scope Change Manage-ment

Example – Project Metrics

Cost

EV, AC, PV,

CPI, SPI, EAC, VAC

Schedule

Activities

Milestones

Reviews

Commitments

Completion dates

Resources

Response times

Capacity / load

Forecast / Actual Usage

Risk

Schedule, Cost, Resources, Deliverables

Quality

Issue

Defects

Re-work

Example – Project Metrics Description Metric # of training hours per project team member. Training Hours # of project team members who quit or terminated. Turnover # of resources assigned to more than one task. Over allocated Resources Budgeted Cost of Work Performed (BCWP) Earned Value # of tasks that have cost more to complete than expected Over budgeted tasks # of task that should have started but have been delayed Tasks # of tasks that were started but not finished on time Overdue tasks # of scope changes that were approved Scope Change Approvals # of scope changes requested by the client or sponsor Scope Change Requests

Metrics – Do & Don’t

Do

Obtain “buy-in”

Provide feedback

Select metrics based on goals

Focus on processes, products & services

Don’t

Ignore the data

Use only one metric

Measure individuals

Use metrics as a “carrot” & “stick”

Project Measures

Measure - Purpose Processes, Products & Services Understand Predict Control Evaluate

Measurement Process Tailor Measures Apply Measures Implement Process Stakeholder Needs Issues, Process Characteristics Measurement Plan External Constraints Measurement Needs Data Actions Project Boundary

Monitor & Control

Monitor & Control - Purpose

To track progress

To take corrective action

To detect variance from plan

Monitor & Control – Tracking

Periodic project status meetings & progress reports

Formal project milestones review

Compare actual vs. planned start date for each project task

Earned Value Management

Pitfalls to Avoid

Not assessing significant variances or unfavorable trends

Not monitoring status frequently and / or all aspects of the project

Not monitoring the effects of corrective actions applied

Not addressing the cause of a variance, but addressing the effects

Not enough interim milestones or measurement points to objectively monitor status and when to apply backup plans

Not considering potential impacts of corrective actions – for example, actions taken to address schedule variances will generally affect cost

Earned Value

Definition

Planned Value (PV) or Budgeted Cost of Work Scheduled (BCWS)

Planned expenditure cash flows based on the completion of tasks in accordance with the project’s budget and schedule

Actual Cost (AC) or Actual Cost of Work Performed (ACWP)

Actual Project Expense based on completed tasks

Earned Value (EV) or Budgeted Cost of Work Performed (BCWP)

The amount of the budget that we should have spent for a given amount of work completed

Example - Budget, Schedule, Tasks

Project Budget = $40,000

Schedule = 4 months

Tasks = 20 Tasks (evenly divided over 4 months)

$2,000 per task

5 tasks per month

Planned Value or BCWS = $10,000 / month

Example - Budget, Schedule, Tasks

Invoice – 1 st Month Payment = $8,000 Actual Cost or ACWP

Work Completed – 1 st Month, 3 tasks costing= $8,000 as against $6,000 ($2,000 x 3)

Earned Value or BCWP = $6,000

Comparison – PV, AC & EV

Cost Performance Indicators 1 month $1000 Scheduled / Budgeted to do $10,000 work over 5 tasks in a month window BCWS = $10,000 PV / BCWS 1 month $6000 $4000 EV / BCWP Schedule slippage permits only 3 tasks/$6,000 work to be performed BCWP = $6,000 Schedule variance = $4,000 1 month $8000 $2000 AC / ACWP Actual cost of work performed = $8,000 ACWP = $8,000 Actual cost variance = $2,000

Cost Metrics