Driving Innovative IT Metrics (Project Management Institute Presentation)

Driving Innovative IT Metrics
Focusing on the CONDITIONS for Success
© 2013 Joe Hessmiller
1

2
Schedule Check
You Are
Here

Joe Hessmiller
• Over 30 years in IT; analyst, programmer, technical
training manager, project manager, user interface
designer, project management consultant, consulting
sales and marketing manager.
• Worked with dozens of IT managers from Fortune 50
corporations and large government agencies.
• Current focus on software project risk management,
specifically, identifying, monitoring and use of project
risk “early warning signs”.
3

1. To establish the need for monitoring the
metrics that really matter.
2. To identify the types of metrics that really
matter.
3. Show how a familiar framework can be
adapted for metrics identification (and
communication).
4. Give you enough to use back at your office to
improve your metrics program.
4
Objectives of Session

Agenda
1. The Innovative Metrics Opportunity
2. Why Do These Opportunities Still Exist?
3. What Metrics Should We Monitor?
4. Working With Conditions Data
5. Developing Innovative Metrics for Your
Organization
5

Part One
The Innovative Metrics
Opportunity
(in Project Management)
6

7
• Despite Everything
We’ve Tried, Project
Success Rates Little
Changed in 30 Years
– McKinsey (17% threaten)
– IBM (40% met 10X range)
– KPMG (70% orgs)
– Standish CHAOS Report
Source: http://calleam.com/WTPF/?page_id=1445
Opportunity

8
• There Is Plenty of Opportunity for
Improving Project Success Rates
Source:
http://versionone.com/assets/img/files/ChaosMa
nifesto2013.pdf
Source:
http://versionone.com/assets/img/files/ChaosMa
nifesto2013.pdf
Opportunity

9
• Strategic Opportunity to Increase
Market Competitiveness
Organizations with a mature PMO outperform those
with an immature PMO by:
28% for on-time project delivery;
24% for on-budget delivery; and
20% for meeting original goals and business
intent of projects. Source: www.metier.com,
According to PMI
Opportunity

10
Source: Paul D. Nielsen, “About Us: From Director and CEO Paul D.
Nielsen,” Carnegie Mellon Software Engineering Institute,
http://www.sei.cmu.edu/about/message/
• Operational Opportunity to Address
Causes of Largest Project Cost
Component
According to the Carnegie Mellon Software Engineering
Institute, “Data indicate that 60-80% of the cost of software
development is in rework.”
Opportunity

Part Two
Why Do These “Opportunities”
Still Exist?
11

12
• The “99% Complete”
Syndrome
– Everything is Fine…Everything is
Fine…
– Oh, Oh, SURPRISE! We’re in Trouble!
• The real challenge to
successful software project
management is:
– having the information
– that is very DIFFICULT to collect,
analyze and act on
– in time to make a difference.
What
You
Know
What
You
Don’t
Know…
but
Need
To
Know
Why Do These Opportunities
Still Exist?

13
• It’s DIFFICULT to Have Project Managers
Who Can Predict the Future
Source:
http://versionone.com/assets
/img/files/ChaosManifesto20
13.pdf
Still Exist?

14
• It’s DIFFICULT to find project managers
who can keep everyone on the same
accurate, relevant and timely page.
Source:
13.pdf
Still Exist?

15
• It’s DIFFICULT to Trust the Service
Providers You Rely On When You Can’t
Verify Their Behavior/Performance
Source:
13.pdf
Still Exist?

SO…What Does an Innovative
Project Manager Need to Do?
16
• Address these DIFFICULT Project
Management Challenges
• By Having the Data That’s Really
Needed to Be Successful

Part Three
What Metrics Should We
Monitor?
17

Monitor?
• Tracking Progress
–Backward Looking
• Managing Risk
–Forward Looking
18

Tracking Progress
Looking Backward
• Volume
• Quality
• Cost
19
What did we do?

• Alignment of IT Investments to Business Strategy
• Cumulative Business Value of IT Investment
• IT Spend Ratio – New Versus Maintenance
• Critical Business Services
– Customer Satisfaction
– Service Level Performance
• Operational Health
– Outages
– Security Incidents
– Project Success Rate
– Average Defect Rate
Source: Craig Symons, Forrester Research, 4-4-08
20
Tracking Progress
Looking Backward – Enterprise Level

21
http://www.slideshare.net/anandsubramaniam/project-metrics-measures
Tracking Progress
Looking Backward – Project Level

22
Managing Risk
Looking Forward – Experts Agree on EWS
What should we do?

Managing Risk
Looking Forward – Kappelman Research
• Kappelman
Research
– Derived List of
• Six People Factors
• Six Process Factors
• For In-process
Audits
23

24
Managing Risk
Looking Forward – Dominant Dozen

• Project
Management
Institute
– 10 Knowledge Areas
– Things You Should
Know
– Things You Should
Do
• For Gate Audits
25
Managing Risk
Looking Forward – Practitioners Too

• In Addition to Traditional
–Key Performance Status
• What To Collect
–Key Performance Conditions
• Intra-process Conditions
• Inter-process Conditions
–Key Process/Practice Compliance
26
Summary: What Metrics Should
We Monitor?

• How Collect
–Intuition
–MBWA
–Survey Software
–Purpose Designed
Software
27
Monitor?

• Pattern Recognition
– Correlations Between Outcomes and
Precedent Conditions
• Feedback Loops
– Correlations Drive KPI Algorithms
Need to Collect the Data for the Insights
28
The Future: PMBD
Project Management Big Data

Part Four
Working with Conditions Data
(The Four Missing Metrics)
29

Three Important CONDITIONS to Monitor
• Expectations Management
• Sponsor Involvement
• Process Compliance
To Minimize Project Risk Factor
• Rework Probability
At 40% of total project costs, Rework is the leading cause of
projects running over budget and beyond schedule.
30
Managing Risk
ESPR “The Missing Metrics”

The Four Missing Metrics
• SMART – Are expectations clear?
• SMPL – Is sponsor engaged?
• PAL - Are processes being
followed?
• PRPL – Are causes of
Rework being avoided?
31

SMART
Level
Tracks the clarity of
assignments. The higher
the SMART Level, the
higher the level of
understanding of what is
expected. Therefore,
less Rework and less
management intervention
required.
The SMART Level
0.0
20.0
40.0
60.0
80.0
100.0
120.0
SMART Index
Upper Control Limit
Lower Control Limit
The SMART Level
32

SMART
Level
Metric derived from
participant feedback on the
perceived Specificity,
Measurability, Achievability,
Relevancy and Timeframes
for their
assignments/deliverables.
SMART Index
Week
SMART
Index
Upper
Control
Limit
Lower
Control
Limit
Specific Measurable Achievable Relevant
Time-
Based
8/10/2010 76.3 100 75 75 50 80 100 95
8/17/2010 76.3 100 75 75 50 80 100 90
8/24/2010 75.0 100 75 75 50 80 95 95
8/31/2010 82.5 100 75 80 70 85 95 95
9/7/2010 83.8 100 75 80 70 85 100 95
9/14/2010 82.5 100 75 90 70 85 85 90
9/21/2010 87.5 100 75 90 90 80 90 90
9/28/2010 92.5 100 75 100 90 80 100 90
10/5/2010 88.8 100 75 100 90 75 90 95
10/12/2010 86.3 100 75 90 90 75 90 95
10/19/2010 87.5 100 75 90 90 75 95 85
10/26/2010 88.8 100 75 90 90 80 95 85
11/2/2010 92.5 100 75 95 95 80 100 90
11/9/2010 92.5 100 75 95 90 85 100 95
11/16/2010 93.8 100 75 100 90 85 100 95
11/23/2010 93.8 100 75 100 100 85 90 100
11/30/2010 93.8 100 75 100 100 80 95 95
12/7/2010 88.8 100 75 90 90 85 90 90
12/14/2010 92.5 100 75 90 90 90 100 95
12/21/2010 95.0 100 75 95 95 95 95 95
12/28/2010 92.5 100 75 90 80 100 100 100
Components of the SMART
Level
The SMART Level
33

SMART
Level
Looks like this project was
back under control once the
Relevance of the
assignments were made
clear to the staff.
40.0
50.0
60.0
70.0
80.0
90.0
100.0
110.0
SMART Index
Upper Control Limit
Lower Control Limit
Specific
Measurable
Achievable
Relevant
Time-Based
The SMART Level
SMART Level Influencers
34

SMPL
Line
Tracks the
participation level of
the senior
management and/or
sponsor.
The SMPL Line
Senior Management Participation Level
Attention
Needed
0.0
20.0
40.0
60.0
80.0
100.0
120.0
SMPL
Upper Control Limit
Lower Control Limit
35

SMPL
Line
In this example, the
level of participation in
meetings, surveys,
and other activities is
compared to the
planned participation
by senior
management
Components of the SMPL Line
36

SMPL
Line
In this example, the
components of the
SMPL are shown
separately to identify
individual areas
needing improvement.
0.0
20.0
40.0
60.0
80.0
100.0
120.0
SMPL
Upper Control Limit
Lower Control Limit
Status Meeting
Attendenace Rate
APO Response Rate
Decisions
Made/Delegated Rate
SMPL Line Influences
37

PAL
Measures likely level of
process adherence
based on conditions that
would tend to lead to
‘short cuts’ on process..
Process Adherence Likelihood
38

PAL
This table shows how
problems with EVA
lead to likely
abandonment of
process until project
gets back on track.
May be OK, must be
identified to be
managed.
Process Adherence Likelihood
Components of PAL Line
39

PAL
This illustrates the various
components of PAL.
PAL Line Influences
40

Project Rework Probability Level
PRPL
Line
Tracks the ‘probability’
of Rework based on
changes in the
conditions that are
known to cause
Rework.
0.0
5.0
10.0
15.0
20.0
25.0
PRPL
Upper Control Limit
Lower Control Limit
Attention
Needed
The PRPL Line
41

PRPL
Line
SME Involvement,
Team Confidence
Level, Technical
Capability and
Requirements
Stability are
components of PRPL
Line.
Project Rework Probability Level (PRPL)
Week PRPL
Upper
Control
Limit
Lower
Control
Limit
SME
Involvement
Technical
Capability
Requirements
Stability
Confidence
Level
8/10/2010 5.0 20 0 100 100 80 100
8/17/2010 10.0 20 0 80 100 80 100
8/24/2010 12.5 20 0 80 100 80 90
8/31/2010 22.5 20 0 70 90 70 80
9/7/2010 17.5 20 0 80 90 70 90
9/14/2010 16.3 20 0 80 90 75 90
9/21/2010 11.3 20 0 80 90 85 100
9/28/2010 23.8 20 0 80 60 85 80
10/5/2010 23.8 20 0 80 60 85 80
10/12/2010 23.8 20 0 80 60 85 80
10/19/2010 18.8 20 0 65 90 85 85
10/26/2010 21.3 20 0 70 90 70 85
11/2/2010 16.3 20 0 80 90 70 95
11/9/2010 16.3 20 0 80 90 70 95
11/16/2010 16.3 20 0 80 80 80 95
11/23/2010 16.3 20 0 80 80 80 95
11/30/2010 17.5 20 0 80 80 80 90
12/7/2010 12.5 20 0 80 90 80 100
12/14/2010 8.8 20 0 85 90 90 100
12/21/2010 5.0 20 0 90 100 90 100
12/28/2010 0.0 20 0 100 100 100 100
Components of PRPL Line
42

PRPL
Line
SME Involvement,
Team Confidence
Level, Technical
Capability and
Requirements
Stability influence
rework probability.
0.0
20.0
40.0
60.0
80.0
100.0
120.0
PRPL
Upper Control Limit
Lower Control Limit
SME Involvement
Confidence Level
Technical Capability
Requirements Stability
PRPL Line Influences
43

Part Five
Developing Innovative Metrics
for Your Organization
44

Five “C”s of Effective Metrics
• Connected to important goals (G-Q-M)
• Complete so that all the factors important to
achieving the goal are included
• Current so that they reflect current goals
• Communicated so that they drive behavior
• Calibrated so that values are comparable
across organizations and over time.
Source: Bob Lewis, InfoWorld, 12-28-11
45

A Gauge for Every Condition
Automotive Engineers Long Ago Defined
the Critical Measures for Safe, Effective
Engine Operation.
46

The Basic Automobile
Dashboard
Automotive Gauge
Odometer
Clock
Fuel Level
Speedometer
Tachometer
Oil Pressure
Oil Temperature
Water Pressure
Water Temperature
Voltmeter
47

What Questions Are We
Asking?
Automotive Gauge Asks the Question
Odometer
How far have we gone?
Clock How much time has it taken?
Fuel Level How far can we go?
Speedometer How fast are we going now?
Tachometer How intensely is engine working?
Oil Pressure Do we have enough ‘lubrication’ to keep parts
working well together?
Oil Temperature How smooth are interactions?
Water Pressure Do we have enough coolant to keep the engine
producing?
Water Temperature How effective is the coolant in keeping the
engine cool?
Voltmeter Is enough energy being applied to the other
important systems?
48

The Basic Measures
Automotive Gauge Asks the Question To Measure
Odometer How far? Deliverables Delivered
Clock How long? Duration
Fuel Level How much further? Input Units Available
Speedometer How fast? Deliverables per Unit of Time
Tachometer How intensely? Effort Intensity
Oil Pressure Do we have enough lubrication to
smooth interactions?
Supply of Lubricant to Smooth Interaction
Between Components
Oil Temperature How smooth are interactions? Ability of Lubricant to smooth Interaction
Between Components
Water Pressure Do we have enough coolant to keep the
engine producing?
Supply of Coolant to dissipate excess
engine heat
Water Temperature How effective is the coolant in keeping
the engine cool?
Ability of Coolant to dissipate engine heat
Voltmeter Is enough energy being applied to the
other important systems?
Ability to Support other Control and
Comfort Systems
49

Comparative Metrics
To Measure
Automotive
Metric IT Metric
Deliverables Delivered Miles Service Level Achieved, Function Point Delivered
Duration Hour Hour
Input Units Available Gallons Resource Hour
Deliverables per Unit of Time Miles Per Hour Earned Value Per Clock Hour
Effort Intensity RPM Hours Worked Per Week/Available Hours
Supply of Lubricant to Smooth
Interaction Between Components
PSI Stakeholder Interaction Satisfaction
Ability of Lubricant to Smooth Interaction
Between Components
Degrees Number of Open Issues from Stakeholder Interactions
Supply of Coolant to dissipate excess
engine heat
PSI Duration to Close Issues/Number of Issues
Ability of Coolant to dissipate engine
heat
Degrees Number of Escalated Issues
Ability to Support other Control and
Comfort Systems
Volts On Time Process Deliverables (Status, Reporting, Training)
50

The More Complex the Project
Environment…
51

• To establish the need for monitoring the metrics
that really matter.
• To identify the types of metrics that really
matter.
• Show how familiar framework can be adapted
for metrics identification (and communication).
• Give you enough to use back at your office to
improve your metrics program.
52
Objective of Session

Driving Innovative IT Metrics (Project Management Institute Presentation)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Driving Innovative IT Metrics (Project Management Institute Presentation)

Similar to Driving Innovative IT Metrics (Project Management Institute Presentation) (20)

More from Computer Aid, Inc

More from Computer Aid, Inc (20)

Recently uploaded

Recently uploaded (20)

Driving Innovative IT Metrics (Project Management Institute Presentation)

Editor's Notes