Searching & Finding The best Available Scientific Evidence

715 views
621 views

Published on

VOV HR Network Belgium
Workshop Evidence-Based HR Management:
Searching & Finding The best Available Scientific Evidence
January 11, 2013, Anterp, Belgium

Published in: Education
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
715
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
28
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Searching & Finding The best Available Scientific Evidence

  1. 1. Pit stop EBMgt Evidence-Based Management (Searching & Finding Scientific Evidence) Pitstop Evidence-Based HR, VOV lerend netwerk, Antwerpen, 11 januari 2013
  2. 2. Pit stop EBMgt 1. Evidence-Based Management A (very) Short Introduction
  3. 3. Medicine: Founding fathersPit stop EBMgt David Sackett Gordon Guyatt McMaster University Medical School, Canada
  4. 4. How it all startedPit stop EBMgt
  5. 5. Problem I: too much information Pit stop EBMgt More than 1 million articles in 40,000 medical journals per year (= 1995; now probably more than 2 million). For a specialist to keep up this means reading 25 articles every day (for a GP more than 100!) Most of the new insights and treatment methods don‟t reach the target group
  6. 6. Problem I: too much information Pit stop EBMgt HRM: 1,350 articles in 2010 (ABI/INFORM). For an HR manager to keep up this means reading 3 to 4 articles every day (for a „general‟ manager more than 50!)
  7. 7. Problem II: persistent convictions Pit stop EBMgt if you’re breathe into a bag hyperventilating
  8. 8. Problem III: jumping to conclusions Pit stop EBMgt elderly people who have give them a drug that an irregular heartbeat are reduces the much more likely to die of number of coronary disease irregular beats
  9. 9. How 40,000 cardiologists can be wrong Pit stop EBMgt In the early 1980s newly introduced anti- arrhythmic drugs were found to be highly successful at suppressing arrhythmias. Not until a RCT was performed was it realized that, although these drugs suppressed arrhythmias, they actually increased mortality. The CAST trial revealed Excess mortality of 56/1000. By the time the results of this trial were published, at least 100,000 such patients had been taking these drugs.
  10. 10. Pit stop EBMgt David Sackett  Half of what you learn in medical school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half.  The most important thing to learn is how to learn on your own.  (Remember that your teachers are as full of bullshit as your parents)
  11. 11. Evidence-Based Practice Pit stop EBMgt 1991 Medicine 1998 Education 1999 Social care 2000 Nursing 2000 Criminal justice ???? Management?
  12. 12. Pit stop EBMgt 2. Evidence based management: What is it?
  13. 13. DefinitionPit stop EBMgt Evidence based practice: Improve information to support decision making
  14. 14. DefinitionPit stop EBMgt Evidence-based management means making decisions about the management of employees, teams or organizations through the conscientious, explicit and judicious use of four sources of information: 1. The best available scientific evidence 2. Organizational facts, metrics and characteristics 3. Stakeholders’ values and concerns 4. Practitioner expertise and judgment
  15. 15. Four sourcesPit stop EBMgt
  16. 16. Pit stop EBMgt Evidence-based practice:  Focuses on the decision making process  Thinks in terms of probability (instead of golden bullets).
  17. 17. Pit stop EBMgt 3. Evidence-based management: Why do we need it?
  18. 18. Four sourcesPit stop EBMgt
  19. 19. Pit stop EBMgtTrust me: 20 years of management experience!
  20. 20. Bounded rationalityPit stop EBMgt
  21. 21. Bounded rationalityPit stop EBMgt System 1  Fast  Intuitive, associative  heuristics & biases System 2  Slow (lazy)  Deliberate, „reasoning‟  Rational
  22. 22. System 1: prone to biases Pit stop EBMgt  Seeing order in randomness  Mental corner cutting  Misinterpretation of incomplete data  Halo effect  False consensus effect  Group think  Confirmation bias  Self serving bias  Authority bias  Sunk cost fallacy  Small numbers fallacy  Cognitive dissonance reduction  In-group bias  Recall bias  Anchoring bias  Inaccurate covariation detection  Distortions due to plausibility
  23. 23. Pit stop EBMgt 2. Searching & Finding The Best Available Scientific Evidence
  24. 24. Pit stop EBMgt Step 1: Formulate a focused question
  25. 25. 5-step approachPit stop EBMgt EBMgt is a 5-step approach 1. Formulate an answerable question (PICOC) 2. Search for the best available evidence 3. Critical appraise the quality of the found evidence 4. Integrate the evidence with managerial expertise and organizational concerns and apply 5. Monitor and evaluate the results
  26. 26. Asking the right question?Pit stop EBMgt Effect vs Non-effect
  27. 27. Types of questions: effectPit stop EBMgt Does it work? Does it work better than ....? Does it have an effect on ....?Effect What is the success factor for ....? What is required to make it work ...? Will it do more good than harm?
  28. 28. Types of questions: non-effectPit stop EBMgtNeeds: What do people want or need?Attitude: What do people think or feel?Experience: What are peoples’ experiences?Prevalence: How many / often do people / organizations ...?Procedure: How can we implement ...?Process: How does it work?Explanation: Why / how does it work?Economics: How much does it cost?
  29. 29. Focused question?Pit stop EBMgt  Does team-building work?  What are the costs and benefits of self-steering teams?  What are the success factors for culture change?  Does management development improve the performance of managers?  Does employee participation prevent resistance to change?  How do employees feel about 360 degree feedback?
  30. 30. Foreground question?Pit stop EBMgt  Does team-building work?  What is a „team‟?  What kind of teams?  In what contexts/settings?  What counts as „team-building‟?  What does „work‟ mean?  What outcomes are relevant?  Over what time periods?
  31. 31. Answerable question: PICOC Pit stop EBMgt P = Population I = Intervention (or success factor) C = Comparison O = Outcome C = Context
  32. 32. Example: merger Pit stop EBMgt
  33. 33. Answerable question: PICOC Pit stop EBMgt Imagine you are a consultant, your client is the board of directors of a large Canadian health-care organization. The board of directors has plans for a merger with a smaller healthcare organization in a nearby town. However, it‟s been said that the organizational culture differs widely between the two organizations. The board of directors asks you if this culture-difference can impede a successful outcome of a merger. Most of them intuitively sense that cultural differences matter, but they want an evidence-based advice.
  34. 34. Answerable question: PICOC Pit stop EBMgt What else would you like to know?
  35. 35. Answerable question: PICOC Pit stop EBMgt P: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff? O: What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction? P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument?
  36. 36. Answerable question: PICOCPit stop EBMgt According to the board the objective of the merger is to integrate the back-office of the two organizations (ICT, finance, purchasing, facilities, personnel administration, etc.) in order to create economy of scale. The front offices and primary process of the two organizations will remain separate. The cultural difference is not objectively assessed (it is the perception of the senior managers of both organizations).
  37. 37. Answerable question: PICOC Pit stop EBMgt P = back office employees in a healthcare organisation I = merger, integration back office C = status quo O = economy of scale C = different organizational culture, unequal
  38. 38. Exercise Pit stop EBMgt 1. Read the following five scenario‟s 2. Formulate on the basis of each scenario a focused question (use the PICOC format).
  39. 39. Scenario 2 Pit stop EBMgt “Ik werk als manager bij een grote financiële dienstverlener. Het concern is een holdingmaatschappij met daaronder een aantal divisies met een eigen P&L verantwoordelijkheid. Het accountmanagement van de divisies wordt aangestuurd op basis van een eigen set kritische prestatie indicatoren (kpi‟s) die niet onderling is afgestemd. Het gevolg is dat de accountmanagers elkaar regelmatig in de wielen rijden bij gezamenlijke klanten en de omzet wordt „weggekaapt‟ bij de ander. Kortom, er wordt onvoldoende samengewerkt. Nu wordt door de Executive Board van de holding een beroep gedaan op de divisiedirecties om meer met elkaar te gaan samenwerken. Mijn vraag is: staat het feit dat de divisies geen gedeelde kpi‟s hebben een goede interne samenwerking in de weg of hoeft dit geen probleem te zijn?”
  40. 40. Scenario 3 Pit stop EBMgt “I am a Dutch quality manager at a large international brewery. In the past 4 years I facilitated a lean management program that is running for more than 10 years in this organization. Since a couple of years, however, we see a decline in results, such as:  The number of problem analysis and improvement teams has been increased, but the amount of breakdowns and short stops in the production lines has not decreased.  There is slow progress for the total program for the Netherlands at this moment. According to the middle managers the improvement program does not address really important topics. I would therefore like to know how our employees feel about lean management.”
  41. 41. Pit stop EBMgt1. Neem een actueel probleem / vraag / issue dat speelt in de organisatie waar je stage hebt gelopen (of een vraagstuk dat je persoonlijke interesse heeft).2. Formuleer op basis daarvan een onderzoeksvraag3. Geef aan wat voor type vraag het is (effect of non-effect)4. Benoem de PICOC elementen.
  42. 42. Pit stop EBMgt Step 2: Search for the best available evidence
  43. 43. 5-step approachPit stop EBMgt EBMgt is a 5-step approach 1. Formulate an answerable question (PICOC) 2. Search for the best available evidence 3. Critical appraise the quality of the found evidence 4. Integrate the evidence with managerial expertise and organizational concerns and apply 5. Monitor and evaluate the results
  44. 44. Searching evidence Pit stop EBMgt What do we search?
  45. 45. Information sourcesPit stop EBMgt Type of Information Source Wall street Journal, Financial Times, Current Information Business week, Financieel Dagblad Overview of a subject Textbooks and popular books General background Encyclopedias, yearbooks & book reviews ABI/INFORM, Business Source Premier, Academic Information Emerald, PsychInfo, Science Direct Statistical Information CBS Statline, Eurostat Theories about a subject Textbooks and encyclopedias Company Annual Reports, Datastream, Company information Factiva.com, Amadeus
  46. 46. Articles in peer reviewed journals Pit stop EBMgt
  47. 47. Searching for evidencePostgraduate Course
  48. 48. DatabasesPostgraduate Course  ABI/INFORM  Business Source Elite  PsycINFO  Web of Knowledge  ERIC  Google Scholar
  49. 49. Searching for evidencePostgraduate Course
  50. 50. Searching evidencePostgraduate Course How do we search? Search Strategy
  51. 51. Search strategy Pit stop EBMgt Two types of search strategies Snowball method Building blocks method
  52. 52. Snowball methodPit stop EBMgt Starting from one book or article, you search for other literature on the same topic.  Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article).  Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge or Google Scholar).
  53. 53. Snowball methodPit stop EBMgt
  54. 54. Building blocks methodPit stop EBMgt Keyword 1 Keyword 2 Keyword 3 Keyword 4 Synonyms or Synonyms or Synonyms or Synonyms or related terms related terms related terms related terms • …. • …. • …. • …. • …. OR • …. OR • …. OR • …. • …. • …. • …. • …. • …. • …. • …. • …. AND AND AND
  55. 55. Answerable question: PICOC Pit stop EBMgt P = back office employees, health care I = merger, integration, back office C = status quo O = economy of scale C = different organizational culture, unequal 1. Underline the keywords 2. Number the order of importance from 1-4
  56. 56. Answerable question: PICOC Pit stop EBMgt P = back office employees 5. healthcare I = 1. merger, 3. integration, back office C = status quo O = 4. economy of scale C = different 2. organizational culture, unequal 1. Underline the keywords 2. Number the order of importance
  57. 57. Search terms Pit stop EBMgt Operationalise your Pico elements! O = long term profitability?
  58. 58. Construct validityPit stop EBMgt The Ambidextrous Organization (??)
  59. 59. Select keywords Pit stop EBMgt The keywords of your PICOC may be enough. If not, select more words by using:  synonyms  alternate spelling, translations  related terms / words / subjects  narrower or broader terms corporate culture: organizational behavior/character, corporate identity merger: acquisition, take-over, fusion, combination, unification profitability: profit, advantage, return on investment, shareholder value
  60. 60. Search in the following order Pit stop EBMgt 1. Thesaurus (SU) 2. Title (TI) 3. Abstract (AB) 4. Alternate spellings, synonyms
  61. 61. Search Query: an example Pit stop EBMgt I I O C 4. Health care 2. Corporate 1. Merger 3. Integration organization culture • Merger • Integration • Healthcare • Corporate culture • Fusion organization • Organizational • Non profit • Combination behavior • Not for profit • Take over OR OR OR • Organizational • Acquisition character • Unification • Corporate •… identity AND AND • Core beliefs • Shared values
  62. 62. Boolean operators Pit stop EBMgt apples AND oranges apples OR oranges apples oranges apples oranges
  63. 63. Start up Pit stop EBMgt
  64. 64. Always search for indexed topicsor thesaurus termsPit stop EBMgt
  65. 65. thesaurus termsPit stop EBMgt
  66. 66. Combining results Pit stop EBMgt Use the history function to combine results
  67. 67. Combining results Pit stop EBMgt Use the history function to combine results
  68. 68. Narrow your resultsPit stop EBMgt
  69. 69. Narrow your resultsPit stop EBMgt
  70. 70. Save your resultsPit stop EBMgt
  71. 71. If an article is not available in ABI/ Inform or BSE: try Google (advanced search)Pit stop EBMgt
  72. 72. Summary Pit stop EBMgt1. Formulate a focused question and break it down into keywords (PICOC),2. Underline the 2 to 4 most relevant keywords,3. Start with the most relevant keyword: is it listed in the thesaurus? If not: search in the title or abstract.4. Repeat your search with alternative terms, synonyms or alternate spellings.5. Combine the outcome of step 4 (thesaurus), 5 (title or abstract) and 6 (alternative terms) with OR (use the history function!)6. Repeat above steps for keyword 27. Combine the outcome of your search with keyword 1 and 2 with AND (use the history function!)8. Filter the articles with “studies” (subject > studies)9. Read the titles and abstracts of your results, select the relevant articles and safe them.10. No full text available in ABI/Inform or BS Elite? Try Google Advanced.
  73. 73. Pit stop EBMgt Learning through play !  Try all buttons  Make lots of mistakes  Have fun !
  74. 74. Assignment: search strategyPit stop EBMgt Search for research articles on teambuilding
  75. 75. Exercise: Search for evidence Pit stop EBMgt 2. Search in ABI/Inform:1. How many articles has Stephen Covey published in peer reviewed journals?2. How many of these articles are based on scientific research?3. Are there articles (by other authors) that are critical of Covey’s 7 Habits?4. How many of these critical articles are based on scientific research?
  76. 76. Pit stop EBMgt Evidence-based management Step 3: Critical appraisal of studies
  77. 77. 5-step approachPit stop EBMgt EBMgt is a 5-step approach 1. Formulate an answerable question (PICOC) 2. Search for the best available evidence 3. Critically appraise the quality of the found evidence 4. Integrate the evidence with managerial expertise and organizational concerns and apply 5. Monitor and evaluate the results
  78. 78. Pit stop EBMgt Effect vs Non-effect
  79. 79. Pit stop EBMgt Internal validity
  80. 80. Internal validityPit stop EBMgt internal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible.
  81. 81. Levels of internal validity Pit stop EBMgt
  82. 82. Which design for which Pit stop EBMgt question?Explanation
  83. 83. Pit stop EBMgt Best research design?
  84. 84. Critical appraisalPit stop EBMgt
  85. 85. Critical appraisalPostgraduate EBMgt Pit stop Course 1. Is the study design appropriate to the stated aims? 2. Was a control group used? 3. Was a pretest used? 4. Are the measurements likely to be valid and reliable? 5. Could bias or confounding have occurred? 6. How large was the effect size?
  86. 86. Levels of internal validityPit stop EBMgt 1. Were there enough subjects in the study? 2. Was a control group used? 3. Were the subjects randomly assigned? 4. Was a pretest used? 5. Was the study started prior to the intervention or event? 6. Was the outcome measured in an objective and reliable way? 6x yes = very high (A) 5x yes = high (A) 4-3x yes = limited (B) 2x yes = low (C) 1-0x yes = very low (D)

×