Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key

Copyright Projecting Success 2018
Martin Paver
CEO / Founder
www.projectingsuccess.co.uk
Lessons Learned Don’t Work:
Why a Data Centric Approach May Hold the Key
APM South Wales and West of England
Branch Event
22 Nov 2018

Why ‘Lessons Learned’?
Project Lead for $Bn
Nuclear New Build
Programme Manager
for £400m portfolio of
60 projects
PMO roles in multi
$Bn portfolios
I’ve always struggled to leverage the broad
experience that I know exists.
"Those that fail
to learn from
history are
doomed to
repeat it."

Leverage Experience – A Wicked Problem?
Small
Margins
Source:APM
Project Success Factors
https://commons.wikimedia.org/wiki/File:Tour_de_France_2016,_team_sky_(28595456475).jpghttps://upload.wikimedia.org/wikipedia/commons/7/76/Mclaren_racing_usgp_2004.jpg
Avoiding the
avoidable
Certainty in delivery
Reduced outliers
Strategic focus
Reputation
Reduce incidents
Increased profit
Cost avoidance
Sector 1
International A
Sector 2
ScheduleRiskLessons Cost
Project Delivery Experience
Company A

The failure of the FiReControl project
The Stark Reality of Not Learning Lessons
Common Agricultural Policy Futures
programme
We do not expect the programme to deliver value for money. The cost of
£178 million is 74 per cent more than the original business case, and the
programme will not fully deliver the anticipated benefits. Farmers reported
they have been adversely affected by the programme delays, highlighting cash
flow problems that have knock-on effects for the wider rural economy.
http://www.audit-scotland.gov.uk/uploads/docs/report/2016/nr_160519_cap_futures.pdf
‘911 Upgrade’
(Emergency Communications
Transformation Program)
The project started in 2004 with an estimated cost of $1.3 billion and a three
year time frame.
According to an audit from City Comptroller, released 21 March 2012, the
project is now $1 billion over budget and seven years behind schedule.
The Comptroller blamed the cost overruns on inadequate project
management within the city.
http://sa.acs.org.au/gallery/leaders_lunch/20120516_Common_causes_of_failure_in_major_ICT-enabled_projects.pdf
NHS 24
The total cost of the Future Programme had risen by 55 per cent to £117.4
million, compared to a business case cost of £75.8 million covering the ten-
year contract period. The increase was due to changes in the contract
specification and costs associated with the delay in implementing the IT
system. [2016]
http://www.audit-scotland.gov.uk/uploads/docs/report/2016/s22_161006_nhs_24.pdf
Disclosure Scotland
The whole programme was estimated to cost £31 million and included a
£29 million project to design and build a new ICT system to support the new
arrangements and replace the existing system, along with other information
management projects, training and guidance.
The programme experienced a number of significant problems when it went
live in February 2011. Disclosure Scotland and the supplier continue to work
on these problems and core functionality is now expected to be established by
autumn 2012.
Crown Office and Procurator Fiscal
Service (COPFS)
The Phoenix programme was initiated in early 2009, at an estimated cost
of £10 million, because a number of software elements of the existing case
management system used to help prepare criminal cases for prosecution were
outdated and lacked readily available technical support.
Management of the programme exhibited a number of weaknesses but it was
cancelled in November 2010 before construction started due to increased
costs and a significantly reduced capital budget.
National Polar-orbiting Operational
Environmental Satellite System
Department of Veterans Affairs’ Scheduling Replacement Project was
terminated in September 2009 after spending an estimated $127 million over
9 years.
http://www.gao.gov/assets/690/681420.pdf
Secure Border Initiative Network program
The Department of Homeland Security’s Secure Border Initiative Network program
was ended in January 2011, after the department obligated more than $1 billion to
the program, because it did not meet cost-effectiveness and viability standards.
Financial and Logistics Integrated Technology
Enterprise program
The Department of Veterans Affairs’ Financial and Logistics Integrated Technology Enterprise program
was intended to be delivered by 2014 at a total estimated cost of $609 million, but was terminated in
October 2011 due to challenges in managing the program
eCare
eCare programme was supposed to solve all our data communication
problems in the health and social care service, but it was abandoned
completely after seven or eight years at—we are told—a cost of £56 million,
although I suspect that the cost was substantially more.
http://www.parliament.scot/S4_PublicAuditCommittee/20_January_2016.pdf
Registers of Scotland (RoS)
Responsible for compiling and maintaining registers relating to
property and other legal documents. Its annual operating costs were
£78 million in 2010/11. While both RoS and BT consider the ongoing
ICT service provision has largely been successful, RoS now considers
that the partnership outsourcing of all ICT was inappropriate and it
has terminated the SPA.
Two individual projects within the programme now cancelled, with
costs of £6.7 million written off in March 2011.
Expeditionary Combat Support System
The Department of Defense’s Expeditionary Combat Support System was
canceled in December 2012 after spending more than a billion dollars and
failing to deploy within 5 years of initially obligating funds.
Retirement Systems Modernization program
The Office of Personnel Management’s Retirement Systems Modernization program was canceled in
February 2011, after spending approximately $231 million on the agency’s third attempt to automate
the processing of federal employee retirement claims.
>£20,000,000,000

Stagnating Productivity

But does it really matter?
Within the next 10 years….
How much could be saved?
The analysis of National Infrastructure and
Construction Pipeline contains projections of £600
billion of public and private investment in
infrastructure over the next 10 years.
McKinsey analysis estimates that global
capital spending will total $77 trillion between
2018 and 2023, which places the annual value
of that spending at more than $10 trillion.
…. >$100 trillion over 10 years

Pursuing the Case for Change: PMI

Programme Manager for
£400m portfolio of 60
projects
Sept 2018 Blog
Pursuing the Case for Change: APM

Papers and ArticlesPapers and Articles

The Journey
Lessons learned
systems do not work Instinctively, we
should be able to
leverage experience Why don’t the
methods work?
What can we do to fix
it?
What have others
done?
Is it fixable?
• NAO Reports
• GAO Reports
• Public hearings
• Public Accounts Committees
• Lessons learned data (Aus, Sco, NI)
• Audit Reports (Can, Aus, NZ, Holland)
• FOI requests
• Commercially sensitive data

Capacity to Learn Lessons
Home Office Government Major Projects Portfolio 2016
Ministry of Justice Government Major Projects Portfolio 2016
With so many projects not delivering to plan there should be a rich opportunity for
leveraging experience for the collective benefit of UK PLC….
…..but there is limited evidence of this happening.
Dept of Health Government Major Projects Portfolio 2016
Ministry of Defence Government Major Projects Portfolio 2016

Digging a Little Deeper
Year RAG Forecast End Forecast Whole Life
Costs £m
2012 31/12/2014 138
2013 01/04/2016 1375
2014 01/09/2016 1205
2015 30/11/2017 1706
2016 30/11/2017 2879
Source: GMPP Reports

projectdataanalytics.uk

Reasons for Not Recording Lessons?
Capitalizing from past projects the value of lessons learned. PMI CONFERENCE
PAPER 2008
By Trevino, Stephanie A. | Anantatmula, Vittal S.
Its a matter of priorities for:
• Senior Management
• Project Leadership
Priority not given because:
• Benefits are not measurable
• Lessons recorded aren’t used
• Process driven approach
Therefore, effort is prioritised on
delivery.
With exceptions for:
• Projects in heavily regulated
industries
• Public sector projects with learning
legacy as a key factor
• Projects in consultancies where
learning provides clear competitive
advantage.

Lessons Learned Processes
DH does not have any policies or processes on lessons learned.
Lesson learned methodology is standard Programme and Project
Management practice which programmes and projects deploy as
they see fit.

But aren’t the NAO fixing the problem?
https://www.nao.org.uk/naoblog/a-systematic-look-at-major-programmes/
It is the role of Departments to leverage the experience, not the NAO

Nearly 20,000 Lessons….. Overview
>50 Organisations
>1,000 Projects
>2 Years of effort to collate
Covering 3Continents
Estimated Portfolio ~$30 bn
From ….
”Benefits clearer”
to …”There was insufficient clarity about
accountability for each budget line, reflecting
a centralised approach to budget
management. Poor links between the Finance
Team and key programmes compounded this
lack of accountability, which meant that
forecasting, monitoring and budget
discussions, as well as early planning of
procurement strategies, did not take place in a
sufficiently timely manner to identify and
address the key risks”.

Existing Lessons Learned Analysis
http://www.treasury.govt.nz

Is This A Knowledge Management Problem?
Project, Programme, Portfolio Management
Technical Delivery
I don’t
know
I do
know
Knowable
Unknowable

Every Lesson has An Action… Or does it?
Lessons learned…..
You experienced XYZ and if you do ABC in the future then the lesson
won’t be repeated.
Observation 1: We should be getting better at this… but the list of
lessons has been similar for the last 30 years.
Observation 2: An effective project manager needs to know every
lesson, in every context about every sort of project.
Observation 3: There are organisational factors that inhibit learning

Findings From the Lessons
• Lack of forensic insight
• ‘The number of technical reviews increased during this time
and were discussed at the project board but there is limited
evidence that the risks highlighted in the reviews were
acted upon or resolved’.
• ‘Lack of robust business requirements at the outset, leading
to unrealistic IT project budgets and timescales’.
• Lots of lessons on reporting, but few define the problem or what to do about it.
• Very variable.
• Process rather than outcome focus.
• Repetition. Some lessons repeated up to x10.
• Often lack context.
• Lack consequence. Difficult to understand what was a big problem.
• Lacks root cause analysis.
• Difficult to identify correlations between lessons.
The best lessons address the above.

Nature & Quality of the Lessons
The SRO and Project Team should develop a
robust and comprehensive Stakeholder
Management and Communication Plan.
The project should consider a workshop to
capture all Lessons Learned to date and
ensure that the learning is embedded within
all future development
Review and implement project
governance arrangements.
Project reporting format should be
reviewed.
Observation?
Something to consider in the next phase?
Missing the basics and don’t know better
Should know better
Fundamental shortfall with minor impact
Fundamental shortfall with major impact

‘Avoidability’ of lessons
•Act of God, environmental factors
•Unknown/unknowable external event
•Unknown/knowable external event
•Known external event
•Internal decision or organisational constraint
•Lesson identified or known but deemed irrelevant
•Lesson identified or known but ignored due to other priorities/resourcing
Unavoidable lesson
But may be aware of
emerging environmental
conditions
Avoidable lesson
Priority, resourcing, SQEP or
judgment call
Strategic lesson
Organisation proceeded
knowing that strategic
decisions may/will likely
impact the project. Lesson
learned was foreseeable
and wasn’t mitigated.

The Link Between Risk and Lessons
Risk
IssueLesson
If the lesson is not acted
upon it becomes a risk
for the next project
How many organisations link lessons learned and risk management?
What is the correlation between
risk drawdown and lessons
learned?
Tolerate
Treat
Transfer
Terminate

Projects: An Enabler or Primary Purpose?
Project Duration
Similarity
Short Long
SimilarDissimilar
Major IT upgrades
Business
transformation
Enabling Activity
Office Move
New office
Call Centre
Project Duration
Similarity
Short Long
SimilarDissimilar
Event management
Major IT upgrades
Business
transformation
Office Move
Crossrail
New office
Primary Activity
HS2
Call Centre
Station refit
Station build 1, 2, 3

Leveraging Experience: Events Scotland
Security
Transport
Ticketing
Funding
Marketing
Infrastructure
…….
Opportunity to recycle experience:
• Capturing schedules, assumptions, risks and budgets.
• Identifying areas of stability and uncertainty and which
are likely to change. Management effort is then focused
on areas of instability.
• Understanding the effectiveness of management action
associated with risks, assumptions and delivery.
• Embedding members of a team into an existing project so
that the learning can be recycled into the next project.
“A 6-7year journey”
Immovable deadlines
• Context is critical in understanding application of experience
• Cost over-runs tend to be associated with infrastructure
• Organisational Capacity to leverage experience
• International sharing of experience

One Dataset. Different Perspectives
Assurance
reviewer
Governance
Supply chain
Project
team
Next project
Other organisations
Process of personal
promotion, personal
reflection & learning
for the next phase
Holding to
account
Preparation for key
gateway reviews
Enabling the supply
chain and improving
efficiency
Leveraging experience
for aligned projects
Leveraging experience
for other
organisations or other
parts of ‘the group’

A Tailored Approach
Any approach to improving project delivery productivity must be tailored to the organisation:
Stable workforce & suppliers
Experience is ‘recycled’
Transient workforce. Churn in supply base
Experience is fragmented
Multiple and parallel implementations of
similar projects
Bespoke, unique projects delivered
infrequently
Experience is primary guide Process is primary guide
Highly competitive where productivity is a
major differentiator
Public sector or monopoly, incremental
improvement
Cross fertilisation between departments Experience is typically siloed within the
organisation
Internal governance External regulation/governance
Workflows, Wikis, Processes, Assurance
Key considerations
Knowledge sharing, project controls, lean
Spectrum of approaches

An Example: Crossrail

Exhaust plume
from project
delivery
An Example: Crossrail

What Happens to the Data?

An Realisable Vision

Instinctive Behaviours
System 1
System 1 is an automatic, fast and often
unconscious way of thinking.
•It is autonomous and efficient, requiring little
energy or attention,
•Is susceptical to biases and systematic errors.
System 2 is an effortful, slow and
controlled way of thinking. It requires
energy and can’t work without
attention but, once engaged, has the
ability to filter the instincts of System 1
System 2
Nudge Theory
We adjust
our driving
to the
conditions
of the road
We adjust our
speed if we
know that
others have
experienced
costly
accidents
VS
System 1 and 2 thinking

Leveraging the Connected Data
Projects LessonsRisks$
Graph
Data Stored in Silos
Lesson X
Draw
down
Cost
impact
Time
impact
Mitigate
Cost
Mitigate
effective
-ness
Project 1
Project 2
Taxon-
omy
TechnicalSafetySecurity
Technical
issue
Security
Issue
Safety
Issue

Exploitation of Data
Actors
Views
User Stories Degree of fit
‘Perfect Data’ ‘Typical Data’
For most organisations, the ‘lessons learned’ process is only solving 25% of the requirement, at best.
“As a Project Manager I want to be able to drill
into lessons that I am interested in so that I can
gain a better understanding of context, relevance
and how the lesson may impact my own project”

EVM data
Resourcing
Weather
Supplier performance
Dependencies
Risks etc
Real time
update of
assigned tasks
WBS Elements
Scheduling Corpus and
Context Extract Triples
Benchmarking
Adaptive SchedulingRecommendations
Scheduling

A once through process
Risk lifecycle
Leveraging Risk Experience
Connected risks Risks-Issues-Lessons
Informed risk
registers
Risk
trends
Risk mitigations
Risk
budget
Systemic Risk
Risks

Benefits

So... What is Advanced Data Analytics

Icons made by Freepik from www.flaticon.com
Machine Learning: Bid Data – a worked example

Bent Flyvbjerg’s work on Major Projects

Counteracting Bias
• Consistency/desirability bias
• Recency effects
• Human memory bias
• Optimism bias
• ‘Strategic Deception’
Data

Fundamentally:
• What is the predisposition of the work to variance?
• Can we predict it?
• How do we test for it?
• How do we treat it and change the future?
Evidence based, tempering against bias.
Project DNA

Barriers to Adoption
Its not on the corporate ‘to do’ list
• Lack of a shared vision
• Lack of evidence to support the vision
• Lack of skilled horsepower
• Lack of data
• Siloed
• Poor quality
• Understanding the investment case

Summary
• The majority of organisations in the sample are not leveraging experience
• The opportunity to leverage experience is significant
• Potential impact on project delivery productivity is huge (up to 30%)
• The approaches do not differentiate between P3M and technical lessons. They are different!
• Does your organisation believe there is value in leveraging past investments?
• STOP box ticking processes and START to link value to your particular challenge
• Pivot towards a data driven insights approach where evidence shapes future decisions
• Deliver information and insights when required, in the right form, which are consumable
• Supported by wikis, peer assists, communities of practice etc.
• Machine learning is marching towards us which will be transformational.
• We need to develop an approach to support it.
"Those that
fail to learn
from history
are doomed
to repeat it."

• Free to join
• Free to attend
• Crosses professional boundaries
• Developing a community
• Developing into a force for good
Developing a Community

Any questions, comments….
…. Or could you share your own experience?

Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key

Recommended

Recommended

More Related Content

Similar to Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key

Similar to Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key (20)

Recently uploaded

Recently uploaded (20)

Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key