1. Evidence-Based Management:
Valid and Reliable Organizational Data
Denise M. Rousseau
H.J. Heinz II University Professor of Organizational Behavior
Carnegie Mellon University
denise@cmu.edu

2. EBMgt Overcomes Limits of Unaided Decisions
 Bounded Rationality
 The Small Numbers
Problem of Individual
Experience
 Prone to See Patterns
Even in Random Data
 Critical Thinking &
Ethics
 Research
• Large Ns > individual
experience
• Controls reduce bias
 Relevant Org’l Facts
• Reliable and valid
• Interpreted using
appropriate logics
 Decision Supports
The “Human” Problem Evidence-Based Practice

3. 1. Get Evidence into the Conversation
2. Use Relevant Scientific Evidence
3. Use Reliable and Valid Business Facts
4. Become “Decision Aware” and Use
Appropriate Processes
5. Reflect on Decision’s Ethical and Stakeholder
Implications
Five Good EBMgt Habits

4. Different Ways of Thinking about Problems
4
Problem Prototype
Process of Thought
Solution
Rearrangement of
Elements
Local Optimization
Mechanical Thinking Intuitive Thinking Strategic Thinking
Transformation or
changed configuration
Analysis
of
Essence

5. 5
Example (1 of 4): the number of operations centres
at a major bank were reduced by 82%
Consolidation in Operating Units over 2 Years
Call Centers
Credit Granting
Credit Management and
Recoveries
Business Service Centers
Personal Service Centers
Operations Service Centers
35
15
20
50
15
7
142
4
4
4
5
5
3
2582% reduction
Centres with over 200 FTEs
Centres with under 200 FTEs
65%
35%
1996 1998-9
Example of Consolidation

6. 6
Example (2 of 4): intuition suggests that the smaller
centres should be less efficient than the large centres
due to scale economie
Scale Curve for Traditional Business
R2
= 59%
0%
10%
20%
30%
40%
50%
60%
- 200 400 600 800
FTEs in Center
Expected
scale curve
Indicator of
efficiency
Indicator of
size of centre
Expected Result
Non-staff cost/All cost

7. 7
Example (3 of 4): however, the reverse was found to be true
(1) total cost/FTE vs. FTEs shows same relationship, but with r2 of 44%
Source: interviews, financial Q1 1999 operating results; call center economics excludes some
centers for which data was not available
Contrary to Expectations, Units Don’t Show Expected
Scale Benefits (1)…
R2
= 59%
0%
10%
20%
30%
40%
50%
60%
- 200 400 600 800
FTEs in Center
Actual
Actual Result
Expected
scale curve
Non-staff cost/All cost

8. 8
Example (4 of 4): this radically altered the course of the project
Assuming that Scale
Economies Existed
Consolidate further - further
worsening the situation
Aim for generalized headcount
cuts to improve financials -
further stretching staff
Knowing that Reverse Scale
Economies Existed
Understand why reverse scale
economies exist (reason:
inconsistent policies, practices
and processes layered on top of
each other from consolidated
centres)
Set in place programs to fix the
cause of the problem (solution:
standardize policies, practices
and processes and reduce
headcount and cost accordingly)
Successful Engagement Possible Disaster
Impacts of the Analysis

9. 9
85
An interesting example of the need for facts, correct analysis and
presentation
Dots indicate O-ring damage for 24 successful space shuttle
launches prior to the Challenger failure.
Challenger was launched on January 28, 1986 with 31 degree
forecast temperature.
NumberofdamagedO-rings
perlaunch
1
2
3
0
Temperature - Degrees Fahrenheit
30 35 40 45 50 55 60 65 70 75 80
Sources: 1) From Edward Tufte, Visual Explanations, MS 1991.
2) Report of the Presedential Commission on the Space Shuttle Challenger Challenger
Accident (Washington DC, 1986) volume V, pages 895-896, ref. 2/26 1 of 3 and ref. 2/26 2 of 3.
3) Siddhartha R. Dalal, Edward B. Fowlkes and Bruce Hoadley, “ Risk Analysis of the Space
Shuttle: Pre-Challenger Prediction of Failure.” Journal of the American Statistical
Association, December 1989, page 946.

10. Best “Available” Business Facts Should
 Present Representative Numbers Sampled From
Entire Population
 Be Interpretable In Context of Use
 Provide Key Indicators for Business Decisions
 Inform Well-thought Out Logics
 Make Sense to Relevant Decision Makers
Use Reliable and Valid Business Facts

11. Best “Available” Business Facts
 Present Representative Numbers Sampled From
Entire Population (No Cherry Picking!)
 NOT biased single or isolated cases (Look at
TOTAL successes & failures not just one or the
other)
 NOT based on small sample sizes (Aggregate or
combine small units to increase reliability and
reduce random variation)
Use Reliable and Valid Business Facts

12.  # Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patient
safety worse in Unit 1?
 Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10%
turnover. Is retention better in one?
 McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem?
How to Interpret Business Facts

13. Organizational Facts
Take Many Forms
Data Information Knowledge
Problem
Solutions

14. DATA
Raw Observations
“Your score is six”
“Eight medication errors occurred”
Ignores total possible
#items on test
Neglects Base Rate
How many total administrations of medication?
#errors plus #non-errors
Organizational Context
How many in same department? By Same Person?

15. DATA
Challenges: Are data reliable?
Complete?
Unbiased?

16. INFORMATION
Making Data Interpretable
Possible Totals or Percentages
“6 out of 10” “60% correct”
Base Rate Represented
“8 errors out of 450 administrations”
Organizational Context Connected
When? Where? By Whom?
4 errors made by same person are likely due to different
factors than are 4 errors made by different individuals in
different departments

17. Using Information Effectively
Informs Well-thought Out Logics that Decision Makers
are Skilled in Using
Based on critical thinking, consideration of
alternatives, and systematic evidence regarding
appropriateness
InputsProcessesLead OutcomesLagged Results
KNOWLEDGE

18. KNOWLEDGE
KNOWLEDGE PROVIDES KEY INDICATORS
FOR ORGANIZATIONAL DECISIONS
DIAGNOSIS
Is 18% turnover a problem? A good thing?
KNOWN IMPACT ON KEY OUTCOMES
What’s the success rate of applicants scoring at or
above 60% on a test in the first year on the job?

19. Best “Available” Business Facts Should
 INFORM ON THE WELL-BEING, HEALTH AND
PERFORMANCE OF THE ORGANIZATION
 Lead and lagged indicators reflecting
organization’s performance pathways
– Lead indicators: critical conditions to be managed in order to
achieve important (lagged) outcomes (customer satisfaction)
– Lagged indicators: important business outcomes (sales growth)
Use Reliable and Valid Business Facts

20. Best “Available” Business Facts Should
 Inform relevant decision makers
 Using frames, definitions, and logics they understand
 Focus of decision maker attention will largely be influenced by
available facts (data, metrics)
– Both unit-specific data and organization-wide
– If too narrow, other goals may be neglected
– If too broad, may be forced to simplify
– If not measured, it cannot be rewarded, if not reward, it won’t occur.
Use Reliable and Valid Business Facts

21. Use Reliable and Valid Business Facts
Focus of attention is influenced by metrics
Division A’s
Metrics
Budget Compliance
1st,2nd,3rd & 4th Qtr
Profitability
1st,2nd,3rd & 4th Qtr
Division B’s
Annual Metrics
Customer Satisfaction
Employee Retention
Staff Development
Return on Assets
Portion of Sales from
Recently Developed
Products

22. The QUESTION you are trying to answer
determines the ANALYSIS:
 How many?
 What works?
 Is it increasing? Decreasing?
 What’s it related to?
 Is there a trend?
22
• A count or percentage
• Always an estimate, not “truth”
• Report confidence interval (the
likely range the true number falls
within)
Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?

23. The QUESTION you are trying to answer
determines the ANALYSIS:
 How many?
 What works?
 Is it increasing? Decreasing?
 What’s it related to?
 Is there a trend?
23Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• Show if factor really led to
outcome change, using:
 Comparison of averages
 Time series -- % change
over time
 Before & After (pre/post)
tests
 Compare to groups without
factor – difference scores (t-
tests)

24. The QUESTION you are trying to answer
determines the ANALYSIS:
 How many?
 What works?
 Is it increasing? Decreasing?
 What’s it related to?
 Is there a trend?
24Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• Regression analysis tells which of a
set of factors are significantly
related
• Suitable for two kinds of data:
 Dichotomous (College YES 1
NO 0)
 Continuous (years of
education)

25. The QUESTION you are trying to answer
determines the ANALYSIS:
 How many?
 What works?
 Is it increasing? Decreasing?
 What’s it related to?
 Is there a trend?
25Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• Compare with past number
 Example: 2012 values divided
by 2011’s
• What controls do we need to really
know what is what?

26. The QUESTION you are trying to answer
determines the ANALYSIS:
 How many?
 What works?
 Is it increasing? Decreasing?
 What’s it related to?
 Is there a trend?
26Evidence-Based Management: Making Evidence-Based Decisions
What’s the Question?
• When and how would you act on it?

27. Turning evidence into practice
Evidence-Based Management:
Closing the gap between research and practice
Turning Evidence into Practice
& Practice into Evidence

28. Got Evidence? References
M. Blastland & A. Dilnot (2007) The Tiger that Wasn’t: Seeing through a World of Numbers. London, Profile
Books.
D. Kahneman (2011) Thinking, Fast and Slow. New York: Farrar, Straus & Giroux.
D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.
D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human Resource
Management Journal, 21, 221-235.
N.Silver (2012) The Signal and the Noise: Why So Many Predictions Fail but Some Don’t. New York: Penguin.

29. Illustration--Discuss with your seatmates…
 What indicators does your organization most
commonly use to make important decisions?
 Are these the “best business facts” you need to
make these decisions?
 What indicators would be more useful, if you
could get them??
Use Reliable and Valid Business Facts