Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evidence-Based Management presentation


Published on

A presentation delivered from Prof. Denise Rousseau in Athens, Greece in March 2012.

Published in: Business

Evidence-Based Management presentation

  1. 1. Evidence-Based ManagementHelping Managers to Make Better Decisions Denise M. Rousseau H.J. Heinz II University Professor of Organizational Behavior Carnegie Mellon University Pittsburgh, PA USA
  2. 2.
  3. 3. - Link to presentations- Teaching material- Checklists, tests- Chapters EBMgt handbook- Answer all your questions
  4. 4. What is Evidence-Based Management? EBMgt is the practice of making organizational decisions based upon conscientious use of 1. Science-based principles & knowledge 2. Valid & relevant organizational and business facts 3. Critical thinking aided by decision supports 4. Ethical considerations (i.e., effects on stakeholders)
  5. 5. Why Should We Care about EBMgt? Better Decisions by Using Practices that Work (and avoiding those that don’t!) Defensible Decisions that Stand Up to Scrutiny (using best evidence and best process) Developing Expertise throughout a Career (experience can be a poor teacher--bad habits!)20 years of valid experience is different than 1 year of experiencerepeated 20 times!
  6. 6. What does EBMgt Look Like?
  7. 7. Evidence-based Piloting?
  8. 8. 1. Use of Scientific Findings Chesley Sullenberger, USAIR pilot, has been a visiting scholar at UC Berkeley’s Collaborative for Catastrophic Risk Management since 2007 Does research on how to make decisions to maintain safety despite technological complexity and crisis conditions
  9. 9. 2. Reliance on Reliable and Valid Organizational Facts Has written and analyzed aviation accident reports for over 20 years
  10. 10. 3. Mindful Decision Making: Becoming Decision Aware Used Decision Aids to Support Good Decision: As Sully considered what decision to make that day, he had his copilot review and follow all checklists on board relevant to crash landings Formal Education to Prime His Skills: Sully is a graduate of the U.S. Air Force Academy and holds masters degrees from both Purdue University in Industrial Psychology and the University of Northern Colorado in Public Administration
  11. 11. 4. Ethics and Responsibility to Stakeholders The last person to leave the plane, Chesley Sullenberger twice walked the plane’s aisle to check all passengers were off Sully’s last act onboard was to grab the passenger list. Used on-shore to verify rescue of all passengers and crew
  12. 12. In Sullenberger’s Own Words… “One way of looking at this might be that for 42 years, Ive been making small, regular deposits in this bank of experience, education and training. And on January 15, the balance was sufficient so that I could make a very large withdrawal.”
  13. 13. How Is Sullenberger’s ExampleRelevant to Your Own Leadership Development?
  14. 14. EBMgt is a means toimprove decision quality. It’s a career, not a course.
  15. 15. The Zeitgeist Evidence-based practice movements abound in medicine, education, and public policy Management research from psychology, engineering, operations research (ETC.) yields 1000s of studies annually Internet ( gives ready access Innovative companies now hiring “chief evidence officers” Public demands accountability (quality decisions that are defensible)
  16. 16. EBMgt Overcomes Limits of Unaided Decisions Bounded Rationality  Critical Thinking The Small Numbers  Decision Supports Problem of Individual Experience  Research • Large Ns > individual Prone to See Patterns experience Even in Random Data • Controls reduce biasThe “Human” Problem Evidence-Based Practice
  17. 17. Five Good EBMgt Habits 1. Get evidence into the conversation 2. Acquire evidence in your area of Practice 3. Learn to find evidence 4. Become “Decision Aware” 5. Reflect on decisions’ ethical implications
  18. 18. Five Good EBMgt Habits #1 Get evidence into the conversation Regularly ask “what’s the evidence…?” Illustration- Discuss with your seatmates… What’s a practice in your organization that you suspect might not be NOT evidence-based?
  19. 19. Five Good EBMgt Habits #2 Acquire Evidence in Your Area of Practice  Focus on Action Principles Where Science is Clear (On-going Practice of A Professional Manager)  Focus on Business Facts based on valid metrics relevant to your decisions (On-going Practice of Responsible & Transparent Organization)
  20. 20. What is evidence?  Evidence is not the same as „proof‟ or „hard facts‟  ... can be - so strong that no one doubts its correctness, or - so weak that it is hardly convincing at all
  21. 21. Don’t confuse  Evidence of effect (do!)  Evidence of no effect (don‟t!)  No evidence of effect (research!)
  22. 22. Five Good EBMgt Habits#2a Learn Use of Scientific Evidence  Focus on Action Principles Where Science is Clear  Rely on Science-based Sources  Example: Locke’s Handbook of Organizational Behavior (access electronic copy for free)  Peer-reviewed research, especially meta-analyses Reduce dysfunctional variations in practice Build effective routines, procedures, checklists
  23. 23. Five Good EBMgt Habits#2a Learn Use of Scientific EvidenceBest Scientific Evidence is  based on large N (sample size of people/organizations)  well-controlled studies with comparison groups &/or longitudinal data  peer-reviewed
  24. 24. Peer Reviewed Journals
  25. 25. Five Good EBMgt Habits #2b Use Reliable and Valid Business Facts Best Business Facts are  large numbers sampled relative to population (not single or isolated cases, e.g. sales/# sales calls)  linked to context (season, location, #users, etc.)  provide key indicators for business decisions
  26. 26. Five Good EBMgt Habits Illustration--Discuss with your seatmates…  What indicators does your organization most commonly use to make important decisions?  Are these the “best business facts” you need to make these decisions?  What indicators would be more useful, if you could get them??
  27. 27. Help Learner How to Interpret Business Facts # Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patient safety worse in Unit 1? Depends on number of unsafe incidents divided by # patients or # procedures—needs a control. Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10% turnover. Is retention better in one? Hard to determine. Small N’s have greater bias and are more variable. McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem? Depends on industry comparison and business strategy. Company A managers focus decisions on monthly cost, downtime and revenues. Company B managers focus on service quality, employee retention and profitability by customer category. So what? B’s more diverse performance criteria can promote attention to longer-term and growth-oriented outcomes. A’s narrower economic focus can promote shorter-term thinking.
  28. 28. Five Good EBMgt Habits #3 Learn to Gather Evidence  Structure and pose a managerial question  Search for best available evidence (check out Google Scholar or CEBMa website)  Critically appraise information found  Apply relevant case information to decision  Write down the decision made, assumptions, and expected outcomes  Evaluate outcomes over time
  29. 29. Five Good EBMgt Habits Gathering Evidence is a 5-step approach 1. Formulate an answerable question (PICOC) 2. Search for the best available evidence 3. Critical appraise the quality of the found evidence 4. Integrate the evidence with managerial expertise and organizational concerns and apply 5. Monitor and evaluate the results
  30. 30. Getting the (Scientific) Evidence What kind of evidence are we looking for?  Studies with a design that best answers your question  Studies with the highest level of evidence
  31. 31. Levels of internal validity
  32. 32. Which study for which question?Explanation
  33. 33. Levels of internal validity It is shown that … It is likely that … There are signs that … Experts are of the opinion that …
  34. 34. Practice Searching for EvidenceUsing “Google Scholar” Learning through play !  Try all buttons  Make lots of mistakes  Have fun!  Just do it!
  35. 35. Five Good EBMgt Habits #4 Become “Decision Aware”  Identify different kinds of decisions you face. What kinds of approaches are used to them? Why?  How can you determine whether you made a “good decision” when you cannot know the outcome? (The answer to this question is what is known as “decision quality”)
  36. 36. Five Good EBMgt Habits Awareness Calls Attention to Decision Process. Proper Processes Improve Decision Quality  What is the process for making the decision?  Different processes work better… - for routine decisions (create validated checklists and action plans) - for decisions with known unknowns (systematic sequence of considerations) - for decisions with unknown unknowns (pilot-tests and trial/ experiment)  Decisions have an “aftermath” and a “pre-math” that a good manager actively manages. Is the decision well- managed? Help make it so.
  37. 37. Using Evidence Well Requires Your Own Critical Judgment
  38. 38. Five Good EBMgt Habits #5 Reflect on Decision’s Ethical Implications  Who are stakeholders for this decision?  Possible effects?  How might the decision be altered to optimize positive stakeholder effects and reduce negative?
  39. 39.  For teaching support  For access to journals  For help desk in support of evidence-searchesdenise@cmu.edumoral support and pointer knowledge!
  40. 40. Turning evidenceEvidence into Practice Turning into practice & Practice into Evidence Evidence based management: closing the gap between research and practice
  41. 41. Turning evidence into practiceGot Evidence? References J. Ehrlinger, K. Johnson, M. Banner, D. Dunning, J. Kruger. (2008) Why the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent. Organizational Behavior and Human Decision Processes, 105,(1) pg. 98 E.A. Locke (ed.), Handbook of Principles of Organizational Behavior, 2nd edition, 2009. Malden, MA: Blackwell. D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York. D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human Resource Management Journal, 21, 221-235. D.M. Rousseau, J. Manning & D. Denyer (2008) Evidence in Management and Organizational Science: Assembling the field’s full weight of scientific knowledge through reflective reviews. Annals of the Academy of Management, 2, 475-515. R.C. Schank, D. Llyras & E. Soloway (2010) The future of decision making. New York: Palmgrave Macmillan. J.F. Yates. (2003). Decision management. San Francisco: Jossey-Bass. J.F. Yates & M.D. Tschirhart (2006). Decision making expertise. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman. (Eds.). Cambridge handbook of expertise and expert performance (pp. 421-438). New York: Cambridge University Press. J.F. Yates, E.S. Veinott & A.L. Patalano (2003). Hard decisions, bad decisions: On decision quality and decision aiding. In S. L. Schneider & J. C. Shanteau (Eds.), Emerging perspectives on judgment and decision research (pp. 13-63). New York: Cambridge University Press.
  42. 42. Turning evidence into practice Appendix: How to conduct a CAT
  43. 43. CAT: Critically Appraised Topic
  44. 44. CAT: Critically Appraised Topic A Critically Appraised Topic (CAT) is a structured, short (3 pages max) summary of evidence on a topic of interest, usually focused around a practical problem or question. A CAT is like a “quick and dirty” version of a systematic review, summarizing the best available research evidence on a topic. Usually more than one study is included in a CAT. Examples:
  45. 45. CAT: structure 1) Question (PICOC) 2) Background / context 3) Search strategy 4) Results / evidence summary 5) Comments (limitations) 6) Conclusion 7) Practical relevance 8) References
  46. 46. Asking the Right Question?
  47. 47. Asking the Right Question? Does team-building work? Does leadership development training work? Does management development improve the performance of managers? Does employee participation prevent resistance to change? Is 360 degree feedback effective?
  48. 48. Answerable Question: PICOC P = Population I = Intervention (or success factor) C = Comparison O = Outcome / Objectives C = Context
  49. 49. Answerable Question: PICOC Scenario: You are a consultant, your client is an insurance company, there are plans for a merger, you have heard that the other company has a different culture, you want to know if this will effect the outcome P = Organizations with a different corporate culture I = Merger C = Organizations with a similar corporate culture O = Long term profitability C = Profit organizations, competitive market
  50. 50. Search terms Operationalise your Pico elements! O = long term profitability? Share holder value? Return on investment? Return on assets? EBIT? Employee productivity? Profit margin? Competitive position? Corporate image? Innovation power? Market share? Customer satisfaction?
  51. 51. Searching Evidence The problem with finding evidence: the abundance of literature
  52. 52. Searching Evidence evidencePostgraduate Course … and not unequivocal Article 1 Article 5 Article 2 Article 6 Article 3 Article 7 Article 4 Article 8
  53. 53. Searching Evidence Evidence-based Searching  In a systematic and transparent way searching for the “best” evidence  Part of EBMgt where decision maker is not a ‘subject matter expert’
  54. 54. Searching Evidence What kind of evidence are we looking for? 1. Studies with a design that best suits the research question 2. Studies with the highest level of evidence
  55. 55. Searching evidence Where do we search?
  56. 56. Databases  ABI/INFORM  Business Source Elite  PsycINFO  Web of Knowledge  ERIC  Google Scholar
  57. 57. Searching evidence How do we search? Search Strategy
  58. 58. Search Strategy 5. Run Search Query4. Select SuitableInformation 1. Determine subjectSources 2. Formulate 3. Select Keywords Answerable Question
  59. 59. Search Strategy Why do we need a search strategy?  Promotes deeper learning about your question  Leads to better yield of quality research.  Saves time in the long run.
  60. 60. Search strategy Two search strategies (Don’t forget to use pointer knowledge along the way) Snowball method Building blocks method
  61. 61. Snowball method Starting from one book or article, you search for other literature on the same topic.  Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article).  Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge or Google Scholar).
  62. 62. Building blocks method Keyword 1 Keyword 2 Keyword 3 Keyword 4 Synonyms or Synonyms or Synonyms or Synonyms or related terms related terms related terms related terms • …. • …. • …. • …. • …. OR • …. OR • …. OR • …. • …. • …. • …. • …. • …. • …. • …. • …. AND AND AND
  63. 63. Search terms Example: You are a consultant, your client is an insurance company, there are plans for a merger, you have heard that the other company has a different culture, you want to know if this will effect the outcome P = Organizations with a different corporate culture I = Merger C = Organizations with a similar corporate culture O = Long term profitability C = Profit organizations, competitive market
  64. 64. Select Keywords P = Organizations with a different corporate culture #2 I = Merger #1 C = Organizations with a similar corporate culture O = Long term profitability #3 C = Profit organizations, competitive market 1. Underline the most important keywords 2. Number the order of importance
  65. 65. Select keywords The keywords of your search query may be enough. If not, select more words by using:  synonyms  alternate spelling, translations  related terms / words / subjects  narrower or broader terms corporate culture: organizational behavior/character, corporate identity merger: acquisition, take-over, fusion, combination, unification profitability: profit, advantage, return on investment, shareholder value
  66. 66.
  67. 67.
  68. 68. Review “Becoming an Evidence-Based HR Manager” to learn more about interpreting the evidence