Nice Model, But What Is The Evidence?

Nice model,
but what is the evidence?
EHMA Annual Meeting, Porto, June 15th, 2016
Evidence Based Practice,
basic principles and practical examples

CEBMa: what we do
 Promote (seminars, papers, blogs, tweets)
 Educate (universities & business schools)
 Train & coach (companies > projects)
 Support / REAs (companies)
 Support / 2nd opinion (BS detector)

15 min Management models
25 min EBP: What is it and why do you need it?
20 min A practical example
… min Discussion
Our mission for today
To disturb
To reassure

Dangers of believing things that aren’t true
 It ain't what you don't know that gets
you into trouble. It's what you know for
sure that just ain't so. (Mark Twain)
 The greatest enemy of knowledge is
not ignorance, it is the illusion of
knowledge. (Stephen Hawking)

Some examples
What’s happened to average job tenure in
the UK and US over the past 10 years?
 Up?
 Down?
 Stayed the same?

Some examples: Job satisfaction
What’s happened to job satisfaction in the
US over the past 10 years?
 Up?
 Down?
 Stayed same?

Some examples: VUCA
Is the world moving ever faster – more
VUCA (volatility, uncertainty, complexity and ambiguity)?
 Faster?
 Slower?
 Same?

 The idea that time is speeding up is clearly popular. It is
also plausible. There is just one problem. It is very hard to
prove that it is actually happening.
 Hard evidence of a great acceleration is hard to come
by. The Economist has considered a variety of measures
by which the speed of business in America can be
quantified. A few do show some acceleration. But a lot do
not.

Some examples: Generational differences
Are there generational differences in work
attitudes?
 Big differences?
 Small differences?
 No differences?

So what?
 There are many taken-for-granted assumptions
and ‘truths’ in management that turn out to be
wrong or at least overstated
 It’s important to examine these assumptions
because if they are wrong will lead to poor
decisions
 Possibly also in the case of Management Models

Model
Taken-for-granted assumptions?
Fad? Evidence?
The question is …

Model
Most of these models are developed
within the domain of management

The problem with managers
They love fads!

38
 Scientific Management/Taylorism
 Business Process Reengineering
 Total Quality Management
 Learning Organizations
 Knowledge Management
The nearly forgotten fads

40
 Talent management
 Emotional intelligence
 Employee engagement
 Lean / six sigma / theory of constraints
 Autonomous teams (it’s back!)
 Agile
The fads that haven’t been forgotten (yet)

41
Fads seem to be attractive, compelling
and irresistible
 Promise to deliver a lot and fast
 Appear simple
 Will make everything alright and help contain
anxieties around intractable problems
 Help user feel effective and cutting edge
 Bits of some fads may work in some context

1. What tells you a model is a fad?
2. Can you give an example in
healthcare management?
Discuss with your neighbour
(2 min)
Discussion
60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210

1. Simple, straightforward: Easy to communicate,
comprehend, and reduce to a small number of
factors or dimensions.
2. Promising results: Fad auteurs are confidently
didactic. There is no false humility or hedging.
3. Universal: Solutions for everyone everywhere.
4. Step-down capability: Can be implemented in
ritualistic and superficial ways.
How to spot a management fad

5. In tune with zeitgeist: Resonate with trends or
perceived business problems of the day.
6. Novel, not radical: Novelty is not so much new
discovery as a rediscovery and repackaging of older
ideas.
7. Lively, entertaining: Articulate, bold, memorable and
upbeat. Filled with labels, buzzwords, lists, acronyms
and fun anecdotes > your system 1 loves it!
8. Legitimacy via gurus and star examples: Supported by
stories of excellent companies and the status of gurus,
not by solid evidence.
How to spot a management fad

Model
So, we need a meta-model to check
whether the model is evidence based

Evidence based practice:
What is it?

Evidence based medicine?
 Never heard off
 Heard off
 Heard off and can explain

Evidence based management?
 Never heard off
 Heard off
 Heard off and can explain

Evidence-based practice
Central Premise:
Decisions should be based on
the ‘best available evidence‘.

Evidence?
information, facts or data
supporting (or contradicting)
a claim, assumption or hypothesis

Evidence?
outcome of scientific research,
organizational facts & figures,
benchmarking, best practices,
personal experience

All managers and leaders base
their decisions on ‘evidence’

But…many managers pay
little or no attention to
the quality of the evidence
they base their decisions on
and use
too few sources of evidence

Trust me, 20 years of
management experience

Sources of evidence
problem solution
Practitioners
professional expertise
Organization
internal data
Stakeholders
values and concerns
Scientific literature
empirical studies
Ask
Acquire
Appraise
Aggregate
Apply
Assess

Professional
experience and
judgment
Multiple sources?

Multiple sources?
Organizational data,
facts and figures

Multiple sources?
Scientific
research
outcomes

Evidence based practice:
Where does it come from?

McMaster University Medical School, Canada
Medicine: Founding fathers
David Sackett Gordon Guyatt

1. Ask: translate a practical issue into an answerable question
2. Acquire: systematically search for and retrieve the evidence
3. Appraise: critically judge the trustworthiness of the evidence
4. Apply: incorporate the evidence into the decision-making process
5. Assess: evaluate the outcome of the decision taken
5 steps of EBmed

Evidence-Based Practice
1991 Medicine
1998 Education
2000 Social care, public policy
Nursing, Criminal justice,
Policing, Architecture, Conservation
2010 Management

Evidence-based … whatever
=
the use of evidence from multiple
sources to increase the likelihood of a
favourable outcome
Focus on the decision making process
Think in terms of probability

Evidence-Based Decision-Making
Why do we need it?

 It ain't what you don't know that gets
you into trouble. It's what you know for
sure that just ain't so. (Mark Twain)
 The greatest enemy of knowledge is
not ignorance, it is the illusion of
knowledge. (Stephen Hawking)

Advice: lie babies down to sleep on their belly
(unanimous support through to the 1990s)
Example: medicine

Example: economy / finance
Nov 15, 2005
"With respect to their safety, derivatives, for the most part, are traded
among very sophisticated financial institutions and individuals who
have considerable incentive to understand them and to use them
properly.”
Jan 10, 2008
”The Federal Reserve is not currently forecasting a recession”

Collateralized Debt Obligations > AAA
0.12% (about 1 chance in 850) default in 5 years

Forecasted Actual
Forecasted and actual 5-year default rates for
AAA-rated CDO tranches
0.12%
28%

 5 Trillion dollars in pension money, real estate
value, savings and bonds disappeared
 8 million people lost their job
 6 million people lost their houses
(and that was only in the USA)

Scared straight
Example: policy / prevention

1. Incompetent people benefit more from feedback than
highly competent people.
2. Task conflict improves work group performance while
relational conflict harms it.
3. Encouraging employees to participate in decision
making is more effective for improving organizational
performance than setting performance goals.
Likely or unlikely?

How evidence-based are HR managers?
 959 (US) + 626 (Dutch) HR professionals
 35 statements, based on an extensive body of evidence
 true / false / uncertain
HR Professionals' beliefs about effective human resource practices: correspondence
between research and practice, (Rynes et al, 2002, Sanders et al 2008)

Outcome: not better than random chance

So where does the evidence-based
stuff come in?

Relying on only 1 source: bad idea!
problem solution
Practitioners
Organization
internal data
Stakeholders
values and concerns
empirical studies

Discuss with your neighbor (1 min)
Over a 5 year period,
why is an orthopedic surgeon's
experience, as a rule, more trustworthy
than a change manager’s experience?
60595857565453525150494847464544434241403938373635343332313029282726252423222120191817161514131211109876543210

Developing expertise
1. A sufficiently regular, predictable environment
2. Numerous opportunities to practice
3. Receive accurate (objective) feedback
The management domain is not highly
favorable to expertise!

How your brain works
System 1
 Fast
 Intuitive, associative
 heuristics & biases
 emotional
System 2
 Lazy
 Slow
 Deliberate
 Rational

System 1 or system 2?
10 seconds

System 1 or system 2?
 A bat and a ball cost $1.10 in total.
 The bat costs $1 more than the ball
 How much does the ball cost?
109876543210

System 1: necessary to survive
95%

 Pattern recognition
 Overconfidence bias
 Halo effect
 False consensus effect
 Group think
 Self serving attribution bias
 Sunk cost fallacy
 Cognitive dissonance reduction
System 1: prone to cognitive errors
 Confirmation bias
 Authority bias
 Small numbers fallacy
 In-group bias
 Recall bias
 Anchoring bias
 Availability bias

We are predisposed to see order, pattern and causal
relations in the world.
Patternicity: The tendency to find meaningful patterns in
both meaningful and meaningless noise.
pattern recognition

We are pattern seeking primates: association learning
pattern recognition

Points of impact of V-1 bombs in London

 A Type I error or a false positive, is
believing a pattern is real when it is not
(finding a non existent pattern)
 A Type II error or a false negative, is
not believing a pattern is real when it is
(not recognizing a real pattern)
Dr. Michael Shermer
(Director of the Skeptics Society)
Pattern recognition

 A Type I error or a false positive: believe that the
rustle in the grass is a dangerous predator when it is
just the wind (low cost)
Pattern recognition

 A Type II error or a false negative: believe that the
rustle in the grass is just the wind when it is a
dangerous predator (high cost)
Pattern recognition

 A Type I error or a false positive: believe that the
rustle in the grass is a dangerous predator when it is
just the wind (low cost)
 A Type II error or a false negative: believe that the
rustle in the grass is just the wind when it is a
dangerous predator (high cost)
Pattern recognition

Doctors and managers hold many erroneous beliefs,
not because they are ignorant or stupid, but because
they seem to be the most sensible conclusion
consistent with their own professional experience!

“I’ve been studying judgment for 45 years, and I’m no better
than when I started. I make extreme predictions. I’m over-
confident. I fall for every one of the biases.”

Practitioners
Organization
internal data
Stakeholders
values and concerns
empirical studies
Four sources of evidence (not only 1)

The performance of
knowledge workers
A Practical Example

 550 beds
 3300 employees
 210 medical specialists
 225,000 admissions
 Top Clinical & Teaching hospital
Organization

A happy & engaged employee
is a productive employee
Fundamental assumption

Let’s have a look at the evidence
Professional
experience and
judgment
Organizational data,
facts and figures
Stakeholders’ values
and concerns
Scientific
research
outcomes
Ask
Acquire
Appraise
Apply
Assess
problem solution

Evidence-based approach, step 1: ASK
problem solution
Practitioners
Organization
internal data
Stakeholders
values and concerns
empirical studies
Ask
Acquire
Appraise
Aggregate
Apply
Assess

Population? Knowledge workers
Whether nurses, lawyers, engineers, managers, or staff members,
nowadays most workers in organizations are highly dependent on
information and communication technology and are involved in
work that involves a high level of cognitive activity.

Question
“Which of the factors that are related to
the performance of knowledge workers
are most widely studied and what is
known of their effect?”

I Don’t Know
(but I know how to find out)
The 3 hardest words in management

Step 2: ACQUIRE
Search for the best available scientific evidence

 ABI, BSP, PsycINFO
 Scholarly journals, peer reviewed
 1980 – 2013
 English
 performance, productivity, knowledge work*
ACQUIRE

Most studied & largest effect size
1. Social cohesion .5 / .7
2. Perceived supervisory support .5
3. Information sharing / TM .5
4. Vision / goal clarity .5
5. Trust .3 / .6

step 3b: CROSS VALIDATE – multiple sources

Three examples
social cohesion supervisory
support
information
sharing

Social cohesion
… a shared liking or team attraction
that includes bonds of friendship, caring,
closeness, and enjoyment of each
other’s company.

…how employees feel the supervisor
helps them in times of need, praises
them for a job well done or recognizes
them for extra effort.
Perceived supervisory support

Information sharing?
…refers to how teams pool and access their
knowledge and expertise – which positively
affects decision making and team processes.
This has led to the idea of a team ‘Transactive
Memory System’ (TMS), which can be thought of
as a collective memory in a collective mind -
enabling a team to think and act together

Step 5: ASSESS the outcome
The departments with the lowest performance
scored under average on most factors

Nice Model, But What Is The Evidence?

More Related Content

What's hot

Viewers also liked

Similar to Nice Model, But What Is The Evidence?

More from Center for Evidence-Based Management

Recently uploaded

Nice Model, But What Is The Evidence?

Editor's Notes