Evidence-Based Management:Valid and Reliable Organizational DataDenise M. RousseauH.J. Heinz II University Professor of Organizational BehaviorCarnegie Mellon Universitydenise@cmu.edu
EBMgt Overcomes Limits of Unaided Decisions Bounded Rationality The Small NumbersProblem of IndividualExperience Prone to See PatternsEven in Random Data Critical Thinking &Ethics Research• Large Ns > individualexperience• Controls reduce bias Relevant Org’l Facts• Reliable and valid• Interpreted usingappropriate logics Decision SupportsThe “Human” Problem Evidence-Based Practice
1. Get Evidence into the Conversation2. Use Relevant Scientific Evidence3. Use Reliable and Valid Business Facts4. Become “Decision Aware” and UseAppropriate Processes5. Reflect on Decision’s Ethical and StakeholderImplicationsFive Good EBMgt Habits
Different Ways of Thinking about Problems4Problem PrototypeProcess of ThoughtSolutionRearrangement ofElementsLocal OptimizationMechanical Thinking Intuitive Thinking Strategic ThinkingTransformation orchanged configurationAnalysisofEssence
5Example (1 of 4): the number of operations centresat a major bank were reduced by 82%Consolidation in Operating Units over 2 YearsCall CentersCredit GrantingCredit Management andRecoveriesBusiness Service CentersPersonal Service CentersOperations Service Centers351520501571424445532582% reductionCentres with over 200 FTEsCentres with under 200 FTEs65%35%1996 1998-9Example of Consolidation
6Example (2 of 4): intuition suggests that the smallercentres should be less efficient than the large centresdue to scale economieScale Curve for Traditional BusinessR2= 59%0%10%20%30%40%50%60%- 200 400 600 800FTEs in CenterExpectedscale curveIndicator ofefficiencyIndicator ofsize of centreExpected ResultNon-staff cost/All cost
7Example (3 of 4): however, the reverse was found to be true(1) total cost/FTE vs. FTEs shows same relationship, but with r2 of 44%Source: interviews, financial Q1 1999 operating results; call center economics excludes somecenters for which data was not availableContrary to Expectations, Units Don’t Show ExpectedScale Benefits (1)…R2= 59%0%10%20%30%40%50%60%- 200 400 600 800FTEs in CenterActualActual ResultExpectedscale curveNon-staff cost/All cost
8Example (4 of 4): this radically altered the course of the projectAssuming that ScaleEconomies ExistedConsolidate further - furtherworsening the situationAim for generalized headcountcuts to improve financials -further stretching staffKnowing that Reverse ScaleEconomies ExistedUnderstand why reverse scaleeconomies exist (reason:inconsistent policies, practicesand processes layered on top ofeach other from consolidatedcentres)Set in place programs to fix thecause of the problem (solution:standardize policies, practicesand processes and reduceheadcount and cost accordingly)Successful Engagement Possible DisasterImpacts of the Analysis
985An interesting example of the need for facts, correct analysis andpresentationDots indicate O-ring damage for 24 successful space shuttlelaunches prior to the Challenger failure.Challenger was launched on January 28, 1986 with 31 degreeforecast temperature.NumberofdamagedO-ringsperlaunch1230Temperature - Degrees Fahrenheit30 35 40 45 50 55 60 65 70 75 80Sources: 1) From Edward Tufte, Visual Explanations, MS 1991.2) Report of the Presedential Commission on the Space Shuttle Challenger ChallengerAccident (Washington DC, 1986) volume V, pages 895-896, ref. 2/26 1 of 3 and ref. 2/26 2 of 3.3) Siddhartha R. Dalal, Edward B. Fowlkes and Bruce Hoadley, “ Risk Analysis of the SpaceShuttle: Pre-Challenger Prediction of Failure.” Journal of the American StatisticalAssociation, December 1989, page 946.
Best “Available” Business Facts Should Present Representative Numbers Sampled FromEntire Population Be Interpretable In Context of Use Provide Key Indicators for Business Decisions Inform Well-thought Out Logics Make Sense to Relevant Decision MakersUse Reliable and Valid Business Facts
Best “Available” Business Facts Present Representative Numbers Sampled FromEntire Population (No Cherry Picking!) NOT biased single or isolated cases (Look atTOTAL successes & failures not just one or theother) NOT based on small sample sizes (Aggregate orcombine small units to increase reliability andreduce random variation)Use Reliable and Valid Business Facts
# Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patientsafety worse in Unit 1? Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10%turnover. Is retention better in one? McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem?How to Interpret Business Facts
Organizational FactsTake Many FormsData Information KnowledgeProblemSolutions
DATARaw Observations“Your score is six”“Eight medication errors occurred”Ignores total possible#items on testNeglects Base RateHow many total administrations of medication?#errors plus #non-errorsOrganizational ContextHow many in same department? By Same Person?
DATAChallenges: Are data reliable?Complete?Unbiased?
INFORMATIONMaking Data InterpretablePossible Totals or Percentages“6 out of 10” “60% correct”Base Rate Represented“8 errors out of 450 administrations”Organizational Context ConnectedWhen? Where? By Whom?4 errors made by same person are likely due to differentfactors than are 4 errors made by different individuals indifferent departments
Using Information EffectivelyInforms Well-thought Out Logics that Decision Makersare Skilled in UsingBased on critical thinking, consideration ofalternatives, and systematic evidence regardingappropriatenessInputsProcessesLead OutcomesLagged ResultsKNOWLEDGE
KNOWLEDGEKNOWLEDGE PROVIDES KEY INDICATORSFOR ORGANIZATIONAL DECISIONSDIAGNOSISIs 18% turnover a problem? A good thing?KNOWN IMPACT ON KEY OUTCOMESWhat’s the success rate of applicants scoring at orabove 60% on a test in the first year on the job?
Best “Available” Business Facts Should INFORM ON THE WELL-BEING, HEALTH ANDPERFORMANCE OF THE ORGANIZATION Lead and lagged indicators reflectingorganization’s performance pathways– Lead indicators: critical conditions to be managed in order toachieve important (lagged) outcomes (customer satisfaction)– Lagged indicators: important business outcomes (sales growth)Use Reliable and Valid Business Facts
Best “Available” Business Facts Should Inform relevant decision makers Using frames, definitions, and logics they understand Focus of decision maker attention will largely be influenced byavailable facts (data, metrics)– Both unit-specific data and organization-wide– If too narrow, other goals may be neglected– If too broad, may be forced to simplify– If not measured, it cannot be rewarded, if not reward, it won’t occur.Use Reliable and Valid Business Facts
Use Reliable and Valid Business FactsFocus of attention is influenced by metricsDivision A’sMetricsBudget Compliance1st,2nd,3rd & 4th QtrProfitability1st,2nd,3rd & 4th QtrDivision B’sAnnual MetricsCustomer SatisfactionEmployee RetentionStaff DevelopmentReturn on AssetsPortion of Sales fromRecently DevelopedProducts
The QUESTION you are trying to answerdetermines the ANALYSIS: How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?22• A count or percentage• Always an estimate, not “truth”• Report confidence interval (thelikely range the true number fallswithin)Evidence-Based Management: Making Evidence-Based DecisionsWhat’s the Question?
The QUESTION you are trying to answerdetermines the ANALYSIS: How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?23Evidence-Based Management: Making Evidence-Based DecisionsWhat’s the Question?• Show if factor really led tooutcome change, using: Comparison of averages Time series -- % changeover time Before & After (pre/post)tests Compare to groups withoutfactor – difference scores (t-tests)
The QUESTION you are trying to answerdetermines the ANALYSIS: How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?24Evidence-Based Management: Making Evidence-Based DecisionsWhat’s the Question?• Regression analysis tells which of aset of factors are significantlyrelated• Suitable for two kinds of data: Dichotomous (College YES 1NO 0) Continuous (years ofeducation)
The QUESTION you are trying to answerdetermines the ANALYSIS: How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?25Evidence-Based Management: Making Evidence-Based DecisionsWhat’s the Question?• Compare with past number Example: 2012 values dividedby 2011’s• What controls do we need to reallyknow what is what?
The QUESTION you are trying to answerdetermines the ANALYSIS: How many? What works? Is it increasing? Decreasing? What’s it related to? Is there a trend?26Evidence-Based Management: Making Evidence-Based DecisionsWhat’s the Question?• When and how would you act on it?
Turning evidence into practiceEvidence-Based Management:Closing the gap between research and practiceTurning Evidence into Practice& Practice into Evidence
Got Evidence? ReferencesM. Blastland & A. Dilnot (2007) The Tiger that Wasn’t: Seeing through a World of Numbers. London, ProfileBooks.D. Kahneman (2011) Thinking, Fast and Slow. New York: Farrar, Straus & Giroux.D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human ResourceManagement Journal, 21, 221-235.N.Silver (2012) The Signal and the Noise: Why So Many Predictions Fail but Some Don’t. New York: Penguin.
Illustration--Discuss with your seatmates… What indicators does your organization mostcommonly use to make important decisions? Are these the “best business facts” you need tomake these decisions? What indicators would be more useful, if youcould get them??Use Reliable and Valid Business Facts