Evidence-Based ManagementHelping Managers to Make Better DecisionsDenise M. RousseauH.J. Heinz II University Professor of Organizational BehaviorCarnegie Mellon Universitydenise@cmu.eduWhat It Means to be an Evidence-Based Manager
EBMgt is the practice of making organizationaldecisions based upon conscientious use of1. Science-based principles & knowledge2. Valid & relevant organizational and business facts3. Critical thinking aided by decision supports4. Ethical considerations (i.e., effects on stakeholders)What is Evidence-Based Management?
Better Decisions by Using Practices that Work (andavoiding those that don’t!) Defensible Decisions that Stand Up to Scrutiny (usingbest evidence and best process) Developing Expertise throughout a Career(experience can be a poor teacher--bad habits!)20 years of valid experience is different than 1 year of experiencerepeated 20 times!Why Should We Care about EBMgt?
Chesley Sullenberger, USAIR pilot, has been avisiting scholar at UC Berkeley’s Collaborativefor Catastrophic Risk Management since 2007 Does research on how to make decisions tomaintain safety despite technologicalcomplexity and crisis conditions1. Use of Scientific Findings
Has written and analyzed aviation accidentreports for over 20 years2. Reliance on Reliable and ValidOrganizational Facts
Used Decision Aids to Support Good Decision: As Sullyconsidered what decision to make that day, he had hiscopilot review and follow all checklists on boardrelevant to crash landings Formal Education to Prime His Skills: Sully is a graduateof the U.S. Air Force Academy and holds mastersdegrees from both Purdue University in IndustrialPsychology and the University of Northern Colorado inPublic Administration3. Mindful Decision Making:Becoming Decision Aware
The last person to leave the plane, ChesleySullenberger twice walked the plane’s aisle tocheck all passengers were off Sully’s last act onboard was to grab thepassenger list. Used on-shore to verify rescueof all passengers and crew4. Ethics and Responsibility toStakeholders
In Sullenberger’s Own Words…“One way of looking at this might be thatfor 42 years, Ive been making small,regular deposits in this bank of experience,education and training.And on January 15, the balance was sufficientso that I could make a very large withdrawal.”
How Is Sullenberger’s ExampleRelevant to Your Own LeadershipExperience?
EBMgt is a means toimprove decision quality.It’s a career, not a course.
Evidence-based practice movements abound inmedicine, education, and public policy Management research frompsychology, engineering, operations research (ETC.)yields 1000s of studies annually Internet (scholar.google.com) gives ready access Innovative companies now hiring “chief evidenceofficers” Public demands accountability (quality decisions thatare defensible)The Zeitgeist
EBMgt Overcomes Limits of Unaided Decisions Bounded Rationality The Small NumbersProblem of IndividualExperience Prone to See PatternsEven in Random Data Critical Thinking Decision Supports Research• Large Ns > individualexperience• Controls reduce biasThe “Human” Problem Evidence-Based Practice
1. Get Evidence into the Conversation2. Use Relevant Scientific Evidence3. Use Reliable and Valid Business Facts4. Become “Decision Aware” and UseAppropriate Processes5. Reflect on Decision’s Ethical and StakeholderImplicationsFive Good EBMgt Habits
#1 Get evidence into the conversationRegularly ask “what’s the evidence…?”Illustration- Discuss with your seatmates…What’s a practice in your organization that you suspect mightnot be NOT evidence-based?Five Good EBMgt Habits
Evidence is not the same as „proof‟ or „hard facts‟ ... can be- so strong that no one doubts its correctness, or- so weak that it is hardly convincing at allWhat is evidence?
Evidence of effect (do!) Evidence of no effect (don‟t!) No evidence of effect (research!)Don’t confuse
#2 Use Relevant Scientific Evidence Focus on Action Principles Where Science is Clear Rely on Science-based Sources Example: Locke’s Handbook of Organizational Behavior(access electronic copy in blackboard) Peer-reviewed research, especially meta-analysesReduce dysfunctional variations in practiceBuild effective routines, procedures, checklistsFive Good EBMgt Habits
Scientific EvidenceBest Scientific Evidence is based on large N (sample size ofpeople/organizations) well-controlled studies with comparison groups&/or longitudinal data peer-reviewedFive Good EBMgt Habits
#3 Use Reliable and Valid Business FactsBest Business Facts are large numbers sampled relative to population(not single or isolated cases, e.g. sales/# sales calls) linked to context (season, location, #users, etc.) provide key indicators for business decisionsFive Good EBMgt Habits
Illustration--Discuss with your seatmates… What indicators does your organization mostcommonly use to make important decisions? Are these the “best business facts” you need tomake these decisions? What indicators would be more useful, if youcould get them??Five Good EBMgt Habits
# Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patientsafety worse in Unit 1? Depends on number of unsafe incidents divided by #patients or # procedures—needs a control. Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10%turnover. Is retention better in one? Hard to determine. Small N’s have greaterbias and are more variable. McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem?Depends on industry comparison and business strategy. Company A managers focus decisions on monthly cost, downtime and revenues.Company B managers focus on service quality, employee retention andprofitability by customer category. So what? B’s more diverse performancecriteria can promote attention to longer-term and growth-oriented outcomes.A’s narrower economic focus can promote shorter-term thinking.Help Learner How to Interpret Business Facts
#4 Become “Decision Aware” Identify different kinds of decisions learnersface? What kinds of different approaches areused to them? Why? How can you determine whether you made a“good decision” when you cannot know theoutcome? (The answer to this question is what isknown as “decision quality”)Five Good EBMgt Habits
“Decision Awareness” Promotes Decision Quality To manage decisions, know what decisions must bemade. Map out decisions that affect key outcomes. Who is responsible? (Are they prepared?) What information is required? (Will it be available whenneeded?)Five Good EBMgt Habits
Five Good EBMgt HabitsAwareness Calls Attention to Decision Process.Proper Processes Improve Decision Quality What is the process for making the decision? Different processes work better…- for routine decisions (create validated checklists and action plans)- for decisions with known unknowns (systematic sequence ofconsiderations)- for decisions with unknown unknowns (pilot-tests and trial/experiment) Decisions have an “aftermath” and a “pre-math” that agood manager actively manages. Is the decision well-managed? Help make it so.
Using Evidence Well Requires YourOwn Critical Judgment
#5 Reflect on Decision’s EthicalImplications Who are stakeholders for this decision? Possible effects? How might the decision be altered to optimize positivestakeholder effects and reduce negative?Five Good EBMgt Habits
Scientific Principles for Effective Teaching Set learning goals (2-5) Pre-test: where does learner stand on learning goal before course Build opportunities for practicing those learnings throughoutcourse (curriculum) Post-test: Measure progress on each learning goal and providefeedback Feedback & Redesign: Use feedback to make course moreeffective over timeFive Good EBMgt Habits
Evidence based management:closing the gap between research and practiceTurning Evidence into Practice& Practice into Evidence
J. Ehrlinger, K. Johnson, M. Banner, D. Dunning, J. Kruger. (2008) Why the unskilled are unaware: Furtherexplorations of (absent) self-insight among the incompetent. Organizational Behavior and Human DecisionProcesses, 105,(1) pg. 98E.A. Locke (ed.), Handbook of Principles of Organizational Behavior, 2nd edition, 2009. Malden, MA: Blackwell.D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human ResourceManagement Journal, 21, 221-235.D.M. Rousseau, J. Manning & D. Denyer (2008) Evidence in Management and Organizational Science:Assembling the field’s full weight of scientific knowledge through reflective reviews. Annals of theAcademy of Management, 2, 475-515.R.C. Schank, D. Llyras & E. Soloway (2010) The future of decision making. New York: Palmgrave Macmillan.J.F. Yates. (2003). Decision management. San Francisco: Jossey-Bass.J.F. Yates & M.D. Tschirhart (2006). Decision making expertise. In K. A. Ericsson, N. Charness, P. J. Feltovich, &R. R. Hoffman. (Eds.). Cambridge handbook of expertise and expert performance (pp. 421-438). New York:Cambridge University Press.J.F. Yates, E.S. Veinott & A.L. Patalano (2003). Hard decisions, bad decisions: On decision quality and decisionaiding. In S. L. Schneider & J. C. Shanteau (Eds.), Emerging perspectives on judgment and decision research(pp. 13-63). New York: Cambridge University Press.Got Evidence? References