Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key

Copyright ProjectingSuccess2018
Martin Paver
CEO / Founder
www.projectingsuccess.co.uk
Lessons Learned Don’t Work:
Why a Data Centric Approach May Hold the Key
APM South Wales and West of England
Branch Event
22 Nov 2018

Why ‘Lessons Learned’?
Project Lead for $Bn
Nuclear New Build
Programme Manager
for £400m portfolio of
60 projects
PMO roles in multi
$Bn portfolios
I’ve alwaysstruggled to leverage the broad
experience that I know exists.
"Those that fail
to learn from
history are
doomed to
repeat it."

Leverage Experience – A Wicked Problem?
Small
Margins
Source:APM
Project Success Factors
https://commons.wikimedia.org/wiki/File:Tour_de_France_2016,_team_sky_(28595456475).jpghttps://upload.wikimedia.org/wikipedia/commons/7/76/Mclaren_racing_usgp_2004.jpg
Avoiding the
avoidable
Certainty in delivery
Reduced outliers
Strategic focus
Reputation
Reduce incidents
Increased profit
Cost avoidance
Sector 1
InternationalA
Sector 2
ScheduleRiskLessons Cost
Project Delivery Experience
Company A

The failure of the FiReControl project
The Stark Reality of Not Learning Lessons
Common Agricultural Policy Futures
programme
We do not expectthe programme to delivervaluefor money. The cost of
£178 millionis 74percent morethan theoriginalbusiness case,and the
programmewill not fully deliver theanticipatedbenefits. Farmers reported
they have been adversely affectedby the programmedelays, highlighting cash
flow problems that haveknock-on effects for thewiderruraleconomy.
http://www.audit-scotland.gov.uk/uploads/docs/report/2016/nr_160519_cap_futures.pdf
‘911 Upgrade’
(Emergency Communications
Transformation Program)
The project started in2004with an estimated costof$1.3billion and a three
year timeframe.
According to an audit fromCity Comptroller,released 21March2012, the
project is now $1billion over budget andsevenyears behind schedule.
The Comptroller blamed the cost overruns on inadequate project
management within thecity.
http://sa.acs.org.au/gallery/l ead ers _lunch/20120516_Common_causes _of_failur e_in _m ajor_ICT- enabl ed_proj ects.pdf
NHS 24
The total cost oftheFuture Programmehadrisen by 55 per centto £117.4
million,compared toa business casecostof£75.8million covering theten-
year contractperiod. Theincreasewas dueto changes inthecontract
specification and costs associatedwith the delay in implementing theIT
system. [2016]
http://www.audit-scotland.gov.uk/uploads/docs/report/2016/s22_161006_nhs_24.pdf
Disclosure Scotland
The whole programme was estimated to cost £31millionand included a
£29 million projectto designandbuild a new ICTsystem tosupport thenew
arrangements and replacetheexisting system, along with otherinformation
management projects, training and guidance.
The programme experienced a number ofsignificantproblems whenit went
live in February 2011. DisclosureScotland and the supplier continuetowork
on these problems and corefunctionality is now expectedto beestablishedby
autumn 2012.
Crown Office and Procurator Fiscal
Service (COPFS)
The Phoenixprogrammewas initiated inearly 2009, at an estimated cost
of£10 million, becausea number ofsoftware elements ofthe existing case
management system used tohelp prepare criminal cases for prosecution were
outdated and lackedreadily availabletechnicalsupport.
Managementofthe programmeexhibiteda number ofweaknesses butit was
cancelled in November 2010before construction started dueto increased
costs and a significantlyreduced capital budget.
National Polar-orbiting Operational
Environmental Satellite System
Department ofVeterans Affairs’ Scheduling Replacement Project was
terminated inSeptember2009afterspending an estimated $127millionover
9 years.
http://www.gao.gov/assets/690/681420.pdf
Secure Border Initiative Network program
The Departmentof Homeland Security’s Secure Border InitiativeNetwork program
was ended inJanuary 2011, after the departmentobligatedmorethan$1billion to
the program, becauseitdidnot meet cost-effectiveness andviability standards.
Financial and Logistics Integrated Technology
Enterprise program
The DepartmentofVeterans Affairs’ Financial and Logistics IntegratedTechnology Enterprise program
was intended to bedelivered by2014ata totalestimatedcostof$609million,butwas terminated in
October 2011duetochallenges in managing theprogram
eCare
eCare programme was supposedto solve allourdata communication
problems in thehealth andsocialcareservice, but itwas abandoned
completely after sevenor eightyears at—wearetold—a cost of£56million,
although I suspectthatthecostwas substantially more.
http://www.parliament.scot/S4_PublicAuditCommittee/20_January_2016.pdf
Registers of Scotland (RoS)
Responsiblefor compiling and maintaining registers relating to
property and other legal documents.Its annualoperating costs were
£78 million in 2010/11. While bothRoS and BT consider the ongoing
ICT serviceprovision has largelybeen successful,RoSnow considers
that the partnership outsourcing ofallICT was inappropriate and it
has terminated theSPA.
Two individualprojects within theprogrammenowcancelled,with
costs of£6.7millionwritten offin March2011.
Expeditionary Combat Support System
The DepartmentofDefense’s Expeditionary CombatSupportSystem was
canceled in December2012afterspending more than a billion dollars and
failing to deploy within 5 years ofinitiallyobligating funds.
Retirement Systems Modernization program
The Office ofPersonnel Management’s Retirement Systems Modernization program was canceled in
February 2011, afterspending approximately $231millionon theagency’s third attemptto automate
the processing of federal employeeretirement claims.
>£20,000,000,000

Stagnating Productivity

But does it really matter?
Within the next 10 years….
How much could be saved?
The analysis of National Infrastructure and
Construction Pipeline contains projections of £600
billion of public and private investment in
infrastructure over the next 10 years.
McKinsey analysis estimates that global
capital spending will total $77 trillion between
2018 and 2023, which places the annual value
of that spending at more than $10 trillion.
…. >$100 trillion over 10 years

Pursuing the Case for Change: PMI

Programme Manager for
£400m portfolio of 60
projects
Sept 2018 Blog
Pursuing the Case for Change: APM

Papers and ArticlesPapers and Articles

The Journey
Lessons learned
systems do not work Instinctively,we
should be able to
leverage experience Why don’t the
methods work?
What can we do to fix
it?
What have others
done?
Is it fixable?
• NAO Reports
• GAO Reports
• Public hearings
• Public Accounts Committees
• Lessons learned data (Aus, Sco, NI)
• Audit Reports (Can, Aus, NZ, Holland)
• FOI requests
• Commercially sensitive data

Capacity to Learn Lessons
Home Office Government Major Projects Portfolio 2016
Ministry of Justice Government MajorProjects Portfolio 2016
With so many projects not delivering to plan there should be a rich opportunity for
leveraging experience for the collectivebenefit of UK PLC….
…..but there is limitedevidence of this happening.
Dept of Health Government Major Projects Portfolio 2016
Ministry of Defence Government MajorProjects Portfolio 2016

Digging a Little Deeper
Year RAG Forecast End Forecast Whole Life
Costs £m
2012 31/12/2014 138
2013 01/04/2016 1375
2014 01/09/2016 1205
2015 30/11/2017 1706
2016 30/11/2017 2879
Source: GMPP Reports

projectdataanalytics.uk

Reasons for Not Recording Lessons?
Capitalizing from past projects the value of lessons learned. PMI CONFERENCE PAPER 2008
By Trevino, Stephanie A. | Anantatmula, Vittal S.
Its a matter of priorities for:
• Senior Management
• Project Leadership
Priority not given because:
• Benefits are not measurable
• Lessons recorded aren’t used
• Process driven approach
Therefore, effort is prioritised on
delivery.
With exceptions for:
• Projects in heavilyregulated
industries
• Public sector projects with learning
legacy as a key factor
• Projects in consultancieswhere
learning provides clear competitive
advantage.

Lessons Learned Processes
DH does not have any policies or processes on lessons learned.
Lesson learned methodologyis standardProgramme and Project
Managementpractice which programmes and projects deploy as
they see fit.

But aren’t the NAO fixing the problem?
https://www.nao.org.uk/naoblog/a-systematic-look-at-major-programmes/
It is the role of Departments to leverage the experience, not the NAO

Nearly 20,000 Lessons….. Overview
>50 Organisations
>1,000 Projects
>2 Years of effort to collate
Covering 3Continents
Estimated Portfolio ~$30 bn
From ….
”Benefits clearer”
to …”There was insufficient clarity about
accountability for each budget line, reflecting
a centralised approach to budget
management. Poor links between the Finance
Team and key programmes compounded this
lack of accountability, which meant that
forecasting, monitoring and budget
discussions, as well as early planning of
procurement strategies, did not take place in a
sufficiently timely manner to identify and
address the key risks”.

Existing Lessons Learned Analysis
http://www.treasury.govt.nz

Is This A Knowledge Management Problem?
Project, Programme, Portfolio Management
TechnicalDelivery
I don’t
know
I do
know
Knowable
Unknowable

Every Lesson has An Action… Or does it?
Lessons learned…..
You experienced XYZ and if you do ABC in the future then the lesson
won’t be repeated.
Observation 1: We should be getting better at this… but the list of
lessons has been similarfor the last 30 years.
Observation 2: An effective project manager needs to know every
lesson, in every context about every sort of project.
Observation 3: There are organisational factorsthat inhibitlearning

Findings From the Lessons
• Lack of forensicinsight
• ‘The number of technical reviews increased during this time
and were discussed at the project board but there is limited
evidence that the risks highlighted in the reviews were
acted upon or resolved’.
• ‘Lack of robust business requirements at the outset, leading
to unrealistic IT project budgets and timescales’.
• Lots of lessons on reporting, but few define the problem or what to do about it.
• Very variable.
• Processrather than outcome focus.
• Repetition. Some lessons repeated up to x10.
• Often lack context.
• Lack consequence. Difficult to understandwhat was a big problem.
• Lacks root cause analysis.
• Difficult to identify correlationsbetween lessons.
The best lessons address the above.

Nature & Quality of the Lessons
The SRO and ProjectTeam should develop a
robust and comprehensive Stakeholder
Management and CommunicationPlan.
The projectshould considera workshop to
capture all LessonsLearned to date and
ensure that the learning is embeddedwithin
all future development
Review and implementproject
governance arrangements.
Projectreporting formatshould be
reviewed.
Observation?
Something to consider in the next phase?
Missing the basics and don’t know better
Should know better
Fundamentalshortfallwith minor impact
Fundamentalshortfallwith major impact

‘Avoidability’ of lessons
•Act of God, environmental factors
•Unknown/unknowableexternal event
•Unknown/knowableexternal event
•Known external event
•Internal decision or organisationalconstraint
•Lesson identifiedor known but deemed irrelevant
•Lesson identifiedor known but ignored due to other priorities/resourcing
Unavoidable lesson
But may be aware of
emerging environmental
conditions
Avoidable lesson
Priority, resourcing, SQEP or
judgmentcall
Strategic lesson
Organisation proceeded
knowing thatstrategic
decisions may/willlikely
impact the project. Lesson
learned wasforeseeable
and wasn’tmitigated.

The Link Between Risk and Lessons
Risk
IssueLesson
If the lesson is not acted
upon it becomes a risk
for the next project
How many organisations link lessons learned and risk management?
What is the correlationbetween
risk drawdown and lessons
learned?
Tolerate
Treat
Transfer
Terminate

Projects:An Enabler or Primary Purpose?
ProjectDuration
Similarity
Short Long
SimilarDissimilar
Major IT upgrades
Business
transformation
Enabling Activity
Office Move
New office
Call Centre
ProjectDuration
Similarity
Short Long
SimilarDissimilar
Event management
Major IT upgrades
Business
transformation
Office Move
Crossrail
New office
Primary Activity
HS2
Call Centre
Station refit
Station build 1, 2, 3

Leveraging Experience: Events Scotland
Security
Transport
Ticketing
Funding
Marketing
Infrastructure
…….
Opportunityto recycle experience:
• Capturing schedules, assumptions, risks and budgets.
• Identifying areas of stability and uncertainty and which
are likely to change. Management effortis then focused
on areas of instability.
• Understanding the effectiveness of management action
associated with risks, assumptions and delivery.
• Embedding members of a team into an existing project so
that the learning can be recycled into the next project.
“A 6-7year journey”
Immovable deadlines
• Context is critical in understanding application of experience
• Cost over-runs tend to be associated with infrastructure
• Organisational Capacityto leverage experience
• International sharingof experience

One Dataset. Different Perspectives
Assurance
reviewer
Governance
Supply chain
Project
team
Next project
Otherorganisations
Process of personal
promotion,personal
reflection & learning
for the next phase
Holding to
account
Preparationfor key
gatewayreviews
Enablingthe supply
chain and improving
efficiency
Leveraging experience
for aligned projects
Leveraging experience
for other
organisationsor other
parts of ‘the group’

A Tailored Approach
Any approach to improvingproject delivery productivitymust be tailored to the organisation:
Stable workforce & suppliers
Experience is ‘recycled’
Transient workforce. Churn in supply base
Experience is fragmented
Multiple and parallel implementations of
similar projects
Bespoke, unique projects delivered
infrequently
Experience is primary guide Process is primary guide
Highly competitive where productivity is a
major differentiator
Public sector or monopoly, incremental
improvement
Cross fertilisation between departments Experience is typically siloed within the
organisation
Internal governance External regulation/governance
Workflows, Wikis, Processes, Assurance
Key considerations
Knowledge sharing, project controls, lean
Spectrum of approaches

An Example: Crossrail

Exhaust plume
fromproject
delivery
An Example: Crossrail

What Happens to the Data?

An Realisable Vision

Instinctive Behaviours
System 1
System 1 is an automatic, fast and often
unconscious way of thinking.
•It is autonomous and efficient, requiring little
energy or attention,
•Is susceptical to biases and systematic errors.
System 2 is an effortful, slow and
controlled way of thinking. It requires
energy and can’t work without
attention but, once engaged, has the
ability to filter the instincts of System 1
System 2
Nudge Theory
We adjust
our driving
to the
conditions
of the road
We adjust our
speed if we
know that
others have
experienced
costly
accidents
VS
System 1 and 2 thinking

Leveraging the Connected Data
Projects LessonsRisks$
Graph
Data Stored in Silos
LessonX
Draw
down
Cost
impact
Time
impact
Mitigate
Cost
Mitigate
effective
-ness
Project 1
Project 2
Taxon-
omy
TechnicalSafetySecurity
Technical
issue
Security
Issue
Safety
Issue

Exploitation of Data
Actors
Views
User Stories Degree of fit
‘Perfect Data’ ‘Typical Data’
For most organisations,the ‘lessons learned’ process is only solving 25% of the requirement, at best.
“As a Project Manager I want to be able to drill
into lessons that I am interested in so that I can
gain a better understanding of context, relevance
and how the lesson may impact my own project”

EVM data
Resourcing
Weather
Supplier performance
Dependencies
Risks etc
Real time
update of
assigned tasks
WBS Elements
Scheduling Corpus and
Context
Extract Triples
Benchmarking
Adaptive SchedulingRecommendations
Scheduling

A once through process
Risk lifecycle
Leveraging Risk Experience
Connected risks Risks-Issues-Lessons
Informed risk
registers
Risk
trends
Risk mitigations
Risk
budget
Systemic Risk
Risks

Benefits

So... What is Advanced Data Analytics

Icons made byFreepik fromwww.flaticon.com
Machine Learning: Bid Data – a worked example

Bent Flyvbjerg’s work on Major Projects

Counteracting Bias
• Consistency/desirability bias
• Recency effects
• Human memory bias
• Optimism bias
• ‘Strategic Deception’
Data

Fundamentally:
• What is the predisposition of the work to variance?
• Can we predict it?
• How do we test for it?
• How do we treat it and change the future?
Evidence based, tempering against bias.
Project DNA

Barriers to Adoption
Its not on the corporate ‘to do’ list
• Lack of a shared vision
• Lack of evidence to support the vision
• Lack of skilled horsepower
• Lack of data
• Siloed
• Poor quality
• Understanding the investment case

Summary
• The majority of organisationsin the sample are not leveraging experience
• The opportunityto leverage experience is significant
• Potentialimpact on project delivery productivityis huge (up to 30%)
• The approachesdo not differentiate between P3M and technicallessons. They are different!
• Does your organisation believethere is value in leveragingpast investments?
• STOP boxticking processes and START to link valueto your particularchallenge
• Pivot towardsa data driven insights approach where evidence shapes future decisions
• Deliver information and insights when required, in the right form, which are consumable
• Supported by wikis, peer assists, communities of practice etc.
• Machine learning is marching towardsus which will be transformational.
• We need to developan approachto support it.
"Those that
fail to learn
from history
are doomed
to repeat it."

• Free to join
• Free to attend
• Crosses professional boundaries
• Developing a community
• Developing into a force for good
Developing a Community

Any questions, comments….
…. Or could you share your own experience?

Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key

Recommended

Recommended

More Related Content

Similar to Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key

Similar to Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key (20)

More from Association for Project Management

More from Association for Project Management (20)

Recently uploaded

Recently uploaded (20)

Lessons Learned Don’t Work: Why a Data Centric Approach May Hold the Key