Evidence-Based Management: A New Approach to Teaching the Practice of Management


Published on

PDW Academy of Management, San Antonio, Texas USA, 2011
Eric Barends, Rob Briner, Blake Jelley, Lori Peterson, Densie Rousseau, Roye Werner

Published in: Education, Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • PRESENTATION ONE 14/08/11 Introduction to Evidence-Based Practice
  • [Be sure to acknowledge Wendy and Denise RE: the chapter upon which much of this presentation is based.]
  • [Mention an adapted version of Eric Barends’ five step “pull” EBMgt approach is key to our chapter on teaching EBMgt in which different courses from different masters programs are described.]
  • Burning Question: FINER PRP: “What don’t we know that matters?” Question Specification: CIMO
  • The question writing process was a back and forth between the students and myself. I utilized a discussion thread on blackboard for the students to post their questions as I did not want two students working on the exact same topic. Experience in previous semesters has shown that the questions evolve as the research is conducted. I did not want to limit the students and with the blackboard discussion thread, all students have access to the topics the other students have posted. While there were some physicians in the class, some of the non-clinical students attempted the clinical topics. Clinical topics are difficult as the students would have more trouble getting access to the research, so I encouraged these students to pick a less clinical topic. One example was Migraines, I encouraged the student away from the why of migraines to the impact of work time lost due to migraines which the student was able to find information on.
  • Individual student assignment, no groups All students selected a topic they were interested in and related to the course. No two students could use the same topic Scenario given: Student was newly hired into a high profile role. The executive leadership team has a problem they are trying to solve and they want you to research it and come back in three months with some background information and to help them make a decision. The deliverables are a 3-5 page written paper and a 4-6 minute presentation. As a new hire, this is a tremendous opportunity to make a good impression with the executives
  • The question writing process was a back and forth between the students and myself. I utilized a discussion thread on blackboard for the students to post their questions as I did not want two students working on the exact same topic. Experience in previous semesters has shown that the questions evolve as the research is conducted. I did not want to limit the students and with the blackboard discussion thread, all students have access to the topics the other students have posted. While there were some physicians in the class, some of the non-clinical students attempted the clinical topics. Clinical topics are difficult as the students would have more trouble getting access to the research, so I encouraged these students to pick a less clinical topic. One example was Migraines, I encouraged the student away from the why of migraines to the impact of work time lost due to migraines which the student was able to find information on.
  • This is a simple template I have prepared for the students to think about their research planning. Several students commented they wished they had something like this earlier in their university days, either as undergrads for research papers, or earlier in graduate school… It is a work in progress, but it does seem to help focus the students.
  • Student have access to university services, PubMed, OhioLink, some had research services through their employer, etc. Many utilized Google Scholar – with more success in 2nd semester, surprisingly. There are some tips I think the students could learn about library research to improve their process. The “pay services” included Elsevier, Springer, etc and journals published by these organizations, using Interlibrary loan many of their article retrieval issues would have been solved. Assignment required 10 sources, some reviewed 30+ articles and over time, got better at reading the abstracts to see if the paper was even helpful. Evolving topics is a problem. With the passage of health care reform, many of the students wanted to research this topic, however, there are fewer academic articles that will help them. Encouraging them toward topics they would be able to find information on. Of course, they can research anything they want, but some topics are better left for other classes…
  • First class received less information on the structure and struggled more with the elements. Second class received more information on what the sections mean and we spent time on what the summary research table should look like. Only one student in two semesters submitted a “research” report, the others compiled research that was more like an executive summary
  • The writing was not a problem as most students had much more research than they could utilize. Assignment called for 3-5 pages, all students utilized 5 pages both semesters. I am considering adding more pages to the next assignment 
  • Reflection suggested perhaps the in class review be earlier, I will try to fit this into the schedule for next semester In class is not graded, but not having a draft meant no points could be awarded for the activity. I reviewed these papers, scanned and emailed comments or summarized comments to students Students had experience with peer review earlier in semester through another assignment Often students write papers and the only person who sees them is the professor, this way, others see their work and perhaps they work harder when they know their peers see their work? Note, I do not use the peer grades when developing my evaluation as there is significant grade inflation among peer grades. No one wants to mark down their classmate…
  • Students delivered 2 Pecha Kucha during term, which helped with structure for the short presentation format With the 2 Pecha Kucha presentations, the 2 nd class presented much better than the first class. Pecha Kucha has gotten great reviews by both MBAs and undergrads. While a short presentation can be daunting, the preparation during the semester helped with nerves and content – first pecha kucha was much worse! These issues can be combated by more presentations, more weight on presentation, different audience (community leaders/alumni), any number of options, but none were so serious as to fail the student.
  • The Reflection Memo assignment questions are included in the assignment handout. Memos are the typical way the students communicate with me. The insights the students provide help me think about how I would change the assignment for the next time, if I offer the assignment at all. This is not an evaluation of the course, simply an insight into the assignment, which is an important distinction for some students.
  • Evidence-Based Management: A New Approach to Teaching the Practice of Management

    1. 1. Evidence-Based Management: A New Approach to Teaching the Practice of Management Roye Werner PDW, Annual AOM 2011, San Antonio Eric Barends Rob Briner Blake Jelley Denise Rousseau Lori Peterson
    2. 2. <ul><li>Rob Briner EBMgt: What ’ s stopping us? </li></ul><ul><li>Roye Werner Without a map </li></ul><ul><li>Eric Barends 5-step pull approach </li></ul><ul><li>Blake Jelley Strategy & Assignments </li></ul><ul><li>Lori Peterson Lessons Learned </li></ul><ul><li>- Subgroups Experiences, Feedback, Support </li></ul>PDW, Annual AOM 2011, San Antonio
    3. 3. PDW, Annual AOM 2011, San Antonio Rob Briner Part 1: EBMgt: What ’ s Stopping Us?
    4. 4. THE UNDERLYING ARGUMENT <ul><li>Four propositions: </li></ul><ul><ul><li>Research produced in business schools could be useful to organizations </li></ul></ul><ul><ul><li>Drawing on the best available evidence (including business school research) is likely to improve management decisions </li></ul></ul><ul><ul><li>Managers and organizations do not appear to be strongly aware of nor use research findings </li></ul></ul><ul><ul><li>We need to increase awareness of and access to research findings </li></ul></ul>
    5. 5.
    6. 6. WHAT ’ S STOPING US? <ul><li>The quick fix problem </li></ul><ul><li>The management fad (fashion) problem </li></ul><ul><li>Why don ’ t some academics and researchers like EBMgt? </li></ul><ul><li>Why don ’ t some managers and practitioners like EBMgt? </li></ul>
    7. 7. QUICK FIXES <ul><li>What is the quick fix? A ‘ solution ’ which </li></ul><ul><ul><li>Focuses on style and presentation not content </li></ul></ul><ul><ul><li>Is always slower than we hoped </li></ul></ul><ul><ul><li>Usually doesn ’ t work </li></ul></ul><ul><ul><li>Is followed by another quick fix </li></ul></ul><ul><li>So why do we do quick fixes? </li></ul><ul><ul><li>Can be career-enhancing for managers </li></ul></ul><ul><ul><li>Speed is often valued over accuracy </li></ul></ul><ul><ul><li>Do we crave quick and easy solutions? </li></ul></ul><ul><li>So who needs or wants academic research? </li></ul>
    8. 8.
    9. 9.
    10. 10.
    11. 11.
    12. 12.
    13. 13.
    14. 14.
    15. 15.
    16. 16. MANAGEMENT FADS (1) <ul><li>The nearly-forgotten fads </li></ul><ul><ul><li>Scientific Management/Taylorism </li></ul></ul><ul><ul><li>Business Process Reengineering </li></ul></ul><ul><ul><li>Management by results </li></ul></ul><ul><ul><li>Excellence </li></ul></ul><ul><ul><li>Total Quality Management </li></ul></ul><ul><ul><li>Learning Organizations </li></ul></ul><ul><ul><li>Knowledge Management </li></ul></ul>
    17. 17. <ul><li>The fads that haven ’t been forgotten (yet) </li></ul><ul><ul><li>Talent management </li></ul></ul><ul><ul><li>Management development </li></ul></ul><ul><ul><li>Executive coaching </li></ul></ul><ul><ul><li>Emotional intelligence </li></ul></ul><ul><ul><li>Employee engagement </li></ul></ul><ul><ul><li>Myers Briggs Type Indicator </li></ul></ul><ul><ul><li>Belbin Team Roles </li></ul></ul><ul><li>General concern about the destructive impact of fads from both practitioners and researchers </li></ul>MANAGEMENT FADS (2)
    18. 27.
    19. 28. FADS * SEEM TO BE ATTRACTIVE, COMPELLING AND IRRESISTIBLE <ul><li>Promise to deliver a lot and fast </li></ul><ul><li>Appear simple </li></ul><ul><li>New and shiny </li></ul><ul><li>Will make everything alright and help contain anxieties around intractable problems </li></ul><ul><li>Help user feel effective and cutting edge </li></ul><ul><li>Bits of some fads may work in some contexts </li></ul><ul><li>So who needs or wants academic research? </li></ul><ul><li>* Evidence-based management not a fad! </li></ul>
    20. 29.
    21. 30.
    22. 31.
    23. 33. WHY DON ’T SOME ACADEMICS LIKE EBMgt? <ul><li>Ambivalence about the value and applicability of management research </li></ul><ul><li>Few incentives to do systematic reviews or CATs </li></ul><ul><li>Primary research (collecting new data) valued more highly than secondary research (reviewing existing data) </li></ul><ul><li>EBMgt not academics ’ responsibility – this is about practice not research </li></ul><ul><li>Some concern that systematic reviews will expose the limited nature of management research </li></ul><ul><li>Some academics are like ‘gurus’ and feel that EBMgt might show their claims to be untrue </li></ul><ul><li>Few incentives to get involved </li></ul>
    24. 34. WHY DON ’T SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (1) <ul><li>Undermines formal authority </li></ul><ul><li>They feel it constrains freedom to make managerial decisions </li></ul><ul><li>Speed valued and rewarded more than accuracy </li></ul><ul><li>Feel they cannot use their own experience and judgment (not true) </li></ul><ul><li>Managers not necessarily rewarded for doing what works (organizations rarely evaluate) </li></ul><ul><li>Few incentives to get involved </li></ul>
    25. 35. <ul><li>Huge pressure to adopt fads </li></ul><ul><li>Have too high expectations of ‘evidence’ (e.g., that it should give the answer otherwise it’s not worth knowing, media representations of science – focus on ‘breakthrough’ single studies – causes distrust of scientific findings) </li></ul>WHY DON ’T SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (2)
    26. 36. <ul><li>Huge incentives and pressure to adopt fads </li></ul><ul><li>And there we see the power of any big managerial idea [or fad or fashion] . It may be smart, like quality, or stupid, like conglomeration. Either way, if everybody's doing it, the pressure to do it too is immense. If it turns out to be smart, great. If it turns out to be stupid, well, you were in good company and most likely ended up no worse off than your competitors. Your company's board consists mostly of CEOs who were probably doing it at their companies. How mad can they get? </li></ul>WHY DON ’T SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (2)
    27. 37. <ul><li>Huge incentives and pressure to adopt fads </li></ul><ul><li>The true value of conventional management wisdom is not that it's wise or dumb, but that it's conventional. It makes one of the hardest jobs in the world, managing an organization, a little easier. By following it, managers everywhere see a way to drag their sorry behinds through another quarter without getting fired. And isn't that, really, what it's all about? </li></ul><ul><li>(Colvin, 2004, Fortune ) </li></ul>WHY DON ’T SOME MANAGERS OR PRACTITIONERS LIKE EBMgt? (2)
    28. 38. SO WHAT ’ S STOPPING US? <ul><li>Few incentives for researchers, business school teachers, or practitioners </li></ul><ul><li>Not a case of removing barriers </li></ul><ul><li>Need to start doing it through our teaching and education to demonstrate the benefits of evidence-based approaches for individuals, professions, and organizations </li></ul><ul><li>‘ We ’ are stopping us! </li></ul>
    29. 39. Part 2: Without a Map Life in the (EBMgt) information universe Roye Werner
    30. 40. Finding the evidence: what I’ll cover: <ul><li>Where does this fit in the teaching of EBMgt? </li></ul><ul><li>How is information on how to manage structured? </li></ul><ul><li>How is research on how to manage structured? </li></ul><ul><li>Tips for teachers </li></ul><ul><li>Tips for searchers </li></ul><ul><li>Tips for managers </li></ul><ul><li>Future visions </li></ul>
    31. 41. Where this fits in the 5 Steps <ul><li>Formulate an answerable question </li></ul><ul><li>Search for the best available evidence </li></ul><ul><li>Critically appraise the evidence </li></ul><ul><li>Integrate the evidence with your managerial expertise and organizational concerns and apply </li></ul><ul><li>Evaluate the process </li></ul>
    32. 42. Peer-reviewed (vague estimate) Map: Management Advice What they see Found on Google? Articles in magazines, newspapers, trade publications, journals. Example: 1,471,332 under subject heading “Management” in ABI-Inform database - of those, 135,778 peer-reviewed. Print is almost gone. Books from academic, trade, and popular presses . Amazon lists 10,000 books on “Management Science” and 6,500 on “Organizational Behavior.” Also large sections in bookstores. Print still rules, but how long before ebooks take over? Websites, blogs, apps, social networks, open courseware . i.e. MIT Sloan courses, HRM Today network, “My MBA – Human Resources” app, HRSpace twitter, business professor blogs, British Library Management Portal Organizations. i.e. SHRM, Conference Board, CEBMa(!) Consultants. US has 361,206 management consultancies listed in Dun & Bradstreet.
    33. 43. Google Scholar ProQuest (ABI Inform, etc.) EBSCO (Business Source Premier, etc.) JSTOR Library discovery service, (Summon, Worldcat Local, Primo Central, etc.) or federated searching program Databases (3 popular ones) Academic books (soon to be mixed with databases) Map: Management Advice What you want them to see
    34. 44. Tips for Teachers: 1 Find out what databases your university library offers
    35. 45. Work with your librarians. They can teach classes, confer with students, create guides. They want to do this! Tips for Teachers: 2
    36. 46. Example of a librarian-created guide Tips for Teachers: 2
    37. 47. Offer hands-on training and practice Tips for Teachers: 3
    38. 48. Put links to resources on your course management system Tips for Teachers: 4
    39. 49. Google Scholar – excellent results, but text not part of the deal <ul><li>Good: </li></ul><ul><li>Powerful search </li></ul><ul><li>Recognized </li></ul><ul><li>Wider range </li></ul><ul><li>Easy, intuitive </li></ul><ul><li>Some things are free </li></ul><ul><li>More languages </li></ul><ul><li>Preferred route for many </li></ul><ul><li>Not so good: </li></ul><ul><li>Less control over search </li></ul><ul><li>Some non-scholarly items </li></ul><ul><li>Duplicates </li></ul><ul><li>No citation help </li></ul><ul><li>Connection to univ. unclear </li></ul><ul><li>Needs subscriptions </li></ul>Tips for Teachers: 5
    40. 50. Some basic principles <ul><ul><li>Databases are different from search engines – more structured, can take advantage of human-assigned categories </li></ul></ul><ul><ul><li>Boolean AND, OR, NOT, parentheses – combining sets </li></ul></ul><ul><ul><li>“ Quotation marks define a phrase” usually </li></ul></ul><ul><ul><li>Truncation symbols (often *) </li></ul></ul><ul><ul><li>Field searching – limiting to title, abstract, subject fields </li></ul></ul><ul><ul><li>Give “Advanced Search” a try </li></ul></ul><ul><ul><li>Look for a tutorial – 5 minutes there can save you much more in your search </li></ul></ul>Tips for Searchers
    41. 51. <ul><ul><li>Public libraries have more than you think </li></ul></ul><ul><ul><li>Is an abstract good enough? Google Scholar provides. (If not, public libraries do interlibrary loan also.) </li></ul></ul><ul><ul><li>Small Business Development Centers </li></ul></ul><ul><ul><li>Work with local universities and professors </li></ul></ul><ul><ul><li>Become a member of CEBMa </li></ul></ul>Tips for Managers
    42. 52. <ul><li>Open Access – please sign on </li></ul><ul><li>Systematic reviews: abundant, accessible and comprehensible </li></ul><ul><li>All in one place - example: ProQuest’s EBMedicine search model </li></ul><ul><li>Mobile access: get ready </li></ul>Future Visions
    43. 53. Eric Barends PDW, Annual AOM 2011, San Antonio Part 3: The 5-step pull approach
    44. 54. Pioneer <ul><li>Half of what you learn in medical school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half. </li></ul><ul><li>The most important thing to learn is how to learn on your own. </li></ul>
    45. 55. An EBM approach to education JAMA, 1992
    46. 56. Push vs Pull Push: teaching management principles based upon a convergent body of research and telling students what to do. Pull: teaching students how to find, appraise and apply the outcome of research (evidence) by themselves
    47. 57. The 5 steps of ‘ pull ’ EBM <ul><li>(0). Create awareness </li></ul><ul><li>Formulate an answerable question </li></ul><ul><li>Search for the best available evidence </li></ul><ul><li>Critically appraise the evidence </li></ul><ul><li>Integrate the evidence and apply </li></ul><ul><li>Monitor the outcome </li></ul>
    48. 58. The 5 steps of ‘ pull ’ EBM 0. A ware 1. A sk 2. A cquire 3. A ppraise 4. A pply 5. A nalyze & A djust
    49. 59. Objectives of the course <ul><li>1. What </li></ul><ul><ul><li>What is evidence-based management? </li></ul></ul><ul><ul><li>What does it look like in practice? </li></ul></ul><ul><li>2. How </li></ul><ul><ul><li>Formulate Answerable Questions </li></ul></ul><ul><ul><li>Search for evidence </li></ul></ul><ul><ul><li>Appraise a study </li></ul></ul><ul><ul><li>Integrate and apply </li></ul></ul>
    50. 60. 1. Asking the right questions
    51. 61. Answerable question I am a consultant, my client a large health-care organization. The board of directors has plans for a merger with a smaller healthcare organization. However, it’s been said that the organizational culture differs widely between the two organizations. The board wants to know if this can impede a successful outcome. Postgraduate Course
    52. 62. P = Population or problem I = Intervention or successfactor C = Comparison O = Outcome C = Context Answerable question: PICO(C) Postgraduate Course
    53. 63. Answerable question: PICOC P: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff? O: What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction? P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument? Postgraduate Course
    54. 64. P = back office employees, organizations with a different organizational culture I = merger, integration back office C = organizations with a similar organizational culture O = economy of scale C = healthcare organizations, unequal Answerable question: PICOC Postgraduate Course
    55. 65. Step 2: Finding the best available evidence Postgraduate Course
    56. 66. Where do we search? Postgraduate Course
    57. 67. Two types of search strategies Search strategie Postgraduate Course Building blocks method Snowball method
    58. 68. Search results
    59. 69. Hands on! Postgraduate Course
    60. 70. Making sense of evidence Step 3: Critically appraise the evidence Postgraduate Course
    61. 71. Critical appraisal How to read a research article? Postgraduate Course
    62. 72. Critical appraisal <ul><li>Study designs </li></ul><ul><li>Levels of evidence </li></ul><ul><li>Bias / confounding </li></ul><ul><li>Effect sizes </li></ul><ul><li>External validity </li></ul>Postgraduate Course
    63. 73. Research designs <ul><li>Systematic reviews & meta analysis </li></ul><ul><li>Randomised controlled study (experiment) </li></ul><ul><li>Non-randomised controlled study (quasi-experiment) </li></ul><ul><li>Correlational study </li></ul><ul><li>Before-after study </li></ul><ul><li>Qualitative research </li></ul>
    64. 74. Which design for which question? Research designs The “ best ” evidence depends on the question type !
    65. 75. Levels of evidence
    66. 76. Levels of Evidence: internal validity
    67. 77. Methodological pitfalls <ul><li>Bias </li></ul><ul><li>Confounding </li></ul>
    68. 78. Step 4: Turning evidence into practice
    69. 79. <ul><li>Is your organization / division / population so different from those in the study that its results cannot apply? </li></ul><ul><li>How relevant is the study to what you are seeking to understand or decide? </li></ul><ul><li>What are your organization ’ s potential benefits and harms from the intervention? </li></ul><ul><li>Is the intervention feasible in your setting? </li></ul>Organization concerns Always ask yourself to what extent the evidence is applicable in your situation:
    70. 80. P = back office employees vs middle managers I = integration back office vs integration board C = O = economy of scale vs shareholder value C = healthcare organization vs financial service organization Answerable question: PICOC Postgraduate Course
    71. 81. CAT: Critically Appraised Topic
    72. 82. CAT: Critically Appraised Topic A critically appraised topic (CAT) is a structured, short (3 pages max) summary of evidence on a topic of interest, usually focused around a practical problem or question. A CAT is like a “quick and dirty” version of a systematic review, summarizing the best available research evidence on a topic.
    73. 83. CAT: structure <ul><li>Background / context </li></ul><ul><li>Question (PICOC) </li></ul><ul><li>Search strategy </li></ul><ul><li>Results / evidence summary </li></ul><ul><li>Limitations </li></ul><ul><li>Conclusion </li></ul><ul><li>Recommendation </li></ul>
    74. 84. CAT-walk
    75. 85. An example
    76. 86. Part 3 Initial Strategies and Reflections R. Blake Jelley, Ph.D. PDW, Annual AOM 2011, San Antonio
    77. 87. Background <ul><li>Two faculty members leading EBMgt @ UPEI. </li></ul><ul><ul><li>PhD management (Wendy Carroll) </li></ul></ul><ul><ul><ul><li>Joined UPEI 2008; seasoned executive; inaugural EMBA co-director; award-winning instructor; active researcher. </li></ul></ul></ul><ul><ul><li>PhD industrial-organizational psychology (me) </li></ul></ul><ul><ul><ul><li>Joined UPEI 2007; “scientist-practitioner”; 5 years in applied research (Ontario Police College)  technical & peer-reviewed pubs. </li></ul></ul></ul><ul><li>UPEI EMBA program launched in 2008. </li></ul><ul><ul><li>My course assignment: Business Research Methods . </li></ul></ul>
    78. 88. <ul><li>Focus for EBMgt teaching (to date) is the EMBA. </li></ul><ul><ul><li>Some preliminary work with undergraduates. </li></ul></ul><ul><li>Current Components. </li></ul><ul><ul><li>EMBA Orientation. </li></ul></ul><ul><ul><li>Managing People & Organizations . </li></ul></ul><ul><ul><li>Business Research Methods . </li></ul></ul><ul><ul><li>Some other courses, too. </li></ul></ul><ul><ul><li>EMBA Signature Project. </li></ul></ul><ul><li>Alternative approaches to EBMgt Teaching. </li></ul><ul><ul><li>(Reasonably) integrated approach (ours). </li></ul></ul><ul><ul><li>An EBMgt-titled course (e.g., Rousseau). </li></ul></ul>Background
    79. 89. <ul><li>Based on Jelley, Carroll, & Rousseau (in press). </li></ul><ul><ul><li>Various exercises and assignments across different courses. </li></ul></ul><ul><li>Developing Basic Awareness of EBMgt. </li></ul><ul><ul><li>Define EBMgt. </li></ul></ul><ul><ul><li>Rynes et al. (2002) HR Knowledge Quiz. </li></ul></ul><ul><ul><li>Self-Guided Field Trip. </li></ul></ul><ul><ul><li>EBMgt as content (push) and process (pull). </li></ul></ul><ul><ul><li>Explicit consideration of EBMgt facets. </li></ul></ul>Teaching Strategies I
    80. 90. <ul><li>Learning to Ask the Right Questions. </li></ul><ul><ul><li>Decision awareness; practitioner focus. </li></ul></ul><ul><ul><li>Absenteeism Diagnosis exercise. </li></ul></ul><ul><ul><li>Burning Question assignment. </li></ul></ul><ul><ul><li>Preliminary Research Proposal assignment. </li></ul></ul><ul><ul><li>Question Specification exercise. </li></ul></ul><ul><ul><li>Surfacing Assumptions exercises. </li></ul></ul>Teaching Strategies II
    81. 91. <ul><li>Getting the Best Available Evidence. </li></ul><ul><ul><li>Rousseau’s rule #1. </li></ul></ul><ul><ul><ul><ul><li>“ Never trust truth claims made without references! It leads directly to Rule #1.1, Do your due diligence: Check (at least a sample of) those references. ” </li></ul></ul></ul></ul><ul><ul><li>Introduction to the business research landscape. </li></ul></ul><ul><ul><li>Literature Search Workshops. </li></ul></ul><ul><ul><li>Literature review; systematic review. </li></ul></ul><ul><ul><li>Snake Oil Symposium. </li></ul></ul><ul><ul><li>Primary research exercises/workshops. </li></ul></ul>Teaching Strategies III
    82. 92. <ul><li>Critically Appraising the Evidence. </li></ul><ul><ul><li>Brief intro. to philosophical issues/perspectives. </li></ul></ul><ul><ul><li>Limited or inclusive use of “evidence hierarchies”. </li></ul></ul><ul><ul><li>Critical appraisal of all four EBMgt elements. </li></ul></ul><ul><ul><li>Research Trinity: Design, measurement, & analysis. </li></ul></ul><ul><ul><li>Indicators of good research; reviewer’s checklists. </li></ul></ul><ul><ul><li>Variable Type and Hypothesis Example exercise. </li></ul></ul><ul><ul><li>Favorite Article Review exercises. </li></ul></ul><ul><ul><li>Myth of Market Share reflective paper. </li></ul></ul><ul><ul><li>Get the Evidence, Myth Busting, etc. group assignments. </li></ul></ul>Teaching Strategies IV
    83. 93. <ul><li>Integrating EBMgt Elements in support of Organizational Decisions and Actions. </li></ul><ul><ul><li>Classroom-based education provides a foundation. </li></ul></ul><ul><ul><ul><li>(For example, Rousseau’s & Carroll’s assignments…) </li></ul></ul></ul><ul><ul><li>That’s Incredible! assignment. </li></ul></ul><ul><ul><li>Persuasive Paper. </li></ul></ul><ul><ul><li>The Great Debate. </li></ul></ul><ul><ul><li>Learning Diary. </li></ul></ul><ul><ul><li>Letter to Self (e.g., www.futureme.org). </li></ul></ul>Teaching Strategies V
    84. 94. <ul><li>Evaluating the Process. </li></ul><ul><ul><li>After Action Review. </li></ul></ul><ul><li>Evaluating our work. </li></ul><ul><ul><li>Pre- and Post-course Assessments. </li></ul></ul><ul><ul><li>Need for measure development and formal evaluation research. </li></ul></ul>Teaching Strategies VI
    85. 95. <ul><li>Realization that evidence is not often used to support decision-making. </li></ul><ul><li>Claims of using learning at work; some specific demonstrations. </li></ul><ul><li>Reading Method sections of peer-reviewed articles… allegedly. </li></ul><ul><li>Meta-analyses may now be favorite reading materials. </li></ul><ul><li>Realization that information exists beyond personal experience and first page of Google hits. </li></ul><ul><li>A lot of material; not easy; make it more “down to earth”; practical assignments. </li></ul><ul><ul><li>- BUT realization that “easy may detract from learning.” </li></ul></ul><ul><li>Variance among students in uptake and understanding. </li></ul>Informal Feedback
    86. 96. <ul><li>Teaching strategies and resources. </li></ul><ul><ul><li>Continue to develop them. </li></ul></ul><ul><ul><li>Adapt existing ideas. </li></ul></ul><ul><ul><ul><li>(For me, more attention to the final two steps) </li></ul></ul></ul><ul><li>Measuring the EBMgt “mindset” and skills. </li></ul><ul><li>Formal research. </li></ul><ul><ul><li>EBMgt education evaluation. </li></ul></ul><ul><ul><li>EBMgt education “outcomes” as predictors (mediators) of organizational criteria? </li></ul></ul>Future Directions
    87. 97. Part 4 Lessons Learned Lori Peterson, PhD Cleveland State University PDW, Annual AOM 2011, San Antonio
    88. 98. CAT Assignment Components <ul><li>Professor Provides </li></ul><ul><li>Student Deliverables </li></ul><ul><li>EBMgt Lecture background (covered in class 1-3) </li></ul><ul><li>Research tips </li></ul><ul><li>Library tool overview </li></ul><ul><li>Real life example </li></ul><ul><li>Guidance at each step </li></ul><ul><li>Feedback </li></ul><ul><li>Question development </li></ul><ul><li>PICO(C) template </li></ul><ul><li>Draft written CAT Summary </li></ul><ul><li>Peer Review of draft CAT Summary </li></ul><ul><li>Final written CAT Summary </li></ul><ul><li>Peer Review of Final CAT Summary </li></ul><ul><li>CAT Walk (presentation) </li></ul><ul><li>Reflection Memo </li></ul>
    89. 99. Question Development <ul><li>Too broad </li></ul><ul><li>Too narrow </li></ul><ul><li>Too easily answered </li></ul><ul><li>Difficult to answer or find information </li></ul><ul><li>Topic too clinical for “Management” or “Administration” students </li></ul><ul><li>Picking a topic from the universe of health care related subjects </li></ul>
    90. 101. Research Tips and Pitfalls <ul><li>Procrastination… waiting, starting late, other classes </li></ul><ul><li>Unable to access articles through “pay services” </li></ul><ul><ul><li>Solution: Interlibrary loan and access through the off-campus library log-in </li></ul></ul><ul><li>Sorting “fact” from “opinion” pieces </li></ul><ul><li>Too much information or question too broad </li></ul><ul><li>Dated (prior to 2000) information for evolving topics </li></ul><ul><li>Learning tools or tactics earlier: </li></ul><ul><ul><li>Examples: search engine tactics and reference managers </li></ul></ul>
    91. 102. Writing Process <ul><li>Assignment limited page length (5 pages) and provided headings which helped with structure </li></ul><ul><li>Reading the articles was more time consuming than many expected, until they got better a selecting articles based on the abstract </li></ul><ul><li>Most questions were slightly revised in the process (as was expected and encouraged) </li></ul><ul><li>Writing has not been a problem, thanks, in part, to the research that is required (10 sources minimum) </li></ul><ul><li>Getting started or staying focused while writing may be another story… </li></ul>
    92. 103. CAT Summary Headings <ul><li>Question </li></ul><ul><li>Background / context </li></ul><ul><li>Results / evidence summary </li></ul><ul><li>Comments (limitations) </li></ul><ul><li>Recommendations </li></ul><ul><li>Conclusion </li></ul><ul><li>References </li></ul><ul><li>Appendix </li></ul><ul><ul><li>PICO(C) Table </li></ul></ul><ul><ul><li>Summary of Research Table </li></ul></ul><ul><ul><li>Other relevant information (tables, charts, etc) </li></ul></ul>
    93. 104. Peer Review <ul><li>In class, one week before written paper due </li></ul><ul><ul><li>Informal </li></ul></ul><ul><ul><li>Help with logic or gaps in content, not typos </li></ul></ul><ul><ul><li>Most papers were incomplete at this point </li></ul></ul><ul><li>Out of class, random assignment, 2 per student </li></ul><ul><ul><li>Formal, used a template </li></ul></ul><ul><ul><li>Template is a work in progress </li></ul></ul><ul><ul><li>Designed to be developmental for future writing </li></ul></ul><ul><ul><li>Many students accesses more papers than they were assigned </li></ul></ul><ul><ul><li>Helped with questions during the CAT Walk </li></ul></ul>
    94. 105. CAT Walk <ul><li>Presentation expected 4-6 minutes, with a maximum time of 10 minutes for questions </li></ul><ul><li>Limited slides to 6, slides were optional </li></ul><ul><li>Typical issues: </li></ul><ul><ul><li>Nervousness in presentation </li></ul></ul><ul><ul><li>Lack of preparation </li></ul></ul><ul><ul><li>Reading slides or notes </li></ul></ul><ul><ul><li>Back to audience (because of reading) </li></ul></ul>
    95. 106. Reflection Memo <ul><li>Number of article abstracts reviewed, articles downloaded, and articles used in summary </li></ul><ul><li>Why were most articles rejected? </li></ul><ul><li>What was your search strategy? (list the tools used. Also, consider how you would explain what you did if someone were to replicate your search.) </li></ul><ul><li>What would have made your search more effective? </li></ul><ul><li>What will you take away from this assignment? </li></ul><ul><li>What would you have liked to learn sooner to make this process easier for you? </li></ul>
    96. 107. Experiences, Barriers & Feedback Group 1: Denise & Lori Group 2: Blake & Rob Support, Teaching Material Group 3: Roye & Eric PDW, Annual AOM 2011, San Antonio Group Discussion
    97. 108. <ul><ul><li>The Center for Evidence-Based Management </li></ul></ul><ul><ul><li>WWW.CEBMa.org </li></ul></ul><ul><ul><li>Presentations </li></ul></ul><ul><ul><li>Handbook of EBMGt: </li></ul></ul><ul><ul><li>Jelley, R. B., Carroll, W. C., & Rousseau, D. M.: </li></ul></ul><ul><ul><li>Reflections on teaching evidence-based management </li></ul></ul>Further Information
    98. 110. CAT: experiences so far <ul><li>Positive </li></ul><ul><li>Clear (completely) new approach </li></ul><ul><li>Library search useful (much more evidence than expected) </li></ul><ul><li>Independency (managers can search for themselves) </li></ul><ul><li>Strong arguments, empowering (evidence speaks for itself) </li></ul><ul><li>Negative </li></ul><ul><li>Access to relevant databases (solved!) </li></ul><ul><li>Lack of aggregated evidence </li></ul><ul><li>Translating conclusions to own context </li></ul><ul><li>Quick and dirty: jumping to conclusion </li></ul>