Evidence-Based HR Management & Systematic Reviews

1,710 views

Published on

Evidence-Based HR Management & Systematic Reviews
PhD Consortium of the 7th International Conference of the Dutch HRM network,
Rob Briner, Eric Barends

Published in: Education, Health & Medicine
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,710
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
97
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Evidence-Based HR Management & Systematic Reviews

  1. 1. Evidence-Based Management What is it? Why do we need it? How does it look like in practice? PhD Consortium of the 7th International Conference of the Dutch HRM network
  2. 2. Evidence-Based ManagementPostgraduate Course  What is it?  Where does it come from?  Why do we need it?  What is stopping us?  How does is look like in practice?
  3. 3. Postgraduate Course 1. Evidence based management: What is it?
  4. 4. Four propositionsPostgraduate Course  Research produced by management scholars could be useful to organizations  Drawing on available evidence (including research produced by academics) is likely to improve decisions  Managers and organizations do not appear to be strongly aware of nor use research findings  We need to increase awareness of and access to research findings
  5. 5. What is EBMgt?Postgraduate Course Evidence-based management is about making decisions through the conscientious, explicit, and judicious use of four sources of information: practitioner expertise and judgment, evidence from the local context, a critical evaluation of the best available research evidence, and the perspectives of those people who might be affected by the decision. (Briner, Denyer, Rousseau, 2009)
  6. 6. What is EBMgt?Postgraduate Course The problem is that management is not as evidence- based as it should be nor as it could be
  7. 7. What is EBMgt?Postgraduate Course  It is not a completely new idea – managers and organizations use evidence all the time  EBMgt is different because it‟s about:  Increasing the types of evidence we use  Using it more thoughtfully and carefully (conscientious, judicious, explicit)  Its only purpose is to help us make better decisions through more and more systematic use of evidence
  8. 8. Who says it’s a problem?Postgraduate Course  People in many other fields (medicine, social work, criminology, politicians) say it‟s a problem in their fields  Not everyone in management but some of people think it‟s a problem  Some academics and researchers  Some managers and organizations  Some professional associations  Some commentators and journalists
  9. 9. •9
  10. 10. Origins of EBMgtPostgraduate Course Management not the only field where there are these concerns. What field is this?  “a research-user gap”  “practitioners do not read academic journals”  “the findings of research into what is an effective intervention are not being translated into actual practice”  “academics not practitioners are driving the research agenda”  “the relevance, quality and applicability of research is questionable”  “practice is being driven more by fads and fashions than research”  “many practices are doing more harm than good”  “the collective wisdom from research is being lost”
  11. 11. Evidence-Based PracticePostgraduate Course 1991 Medicine 1998 Education 1998 Probation service 1999 Housing policy 1999 Social care 2000 Nursing 2000 Criminal justice ???? Management?
  12. 12. Academic interest in EBMgtPostgraduate Course Similar ideas around for a long time 2003 – Systematic reviews of evidence 2006 – Rousseau EBMgt Presidential Address and Pfeffer & Sutton book 2007-2009 – Rousseau EBMgt Collaborative 2008-2011 – Several conferences 2011 – Center for Evidence-Based Management (CEBMa) in Amsterdam 2011 – EBMgt Handbook But many researchers not interested at all in EBMgt
  13. 13. Manager & practitioner interest in EBMgt  Some HRM and professional bodies express interest (SHRM, SIOP, VOV Learning Network)  Some Universities running courses for practitioners (e.g., Amsterdam, Ghent)  Some publications for practitioners  But managers are:  Used to working in a different way  Under pressure to adopt fads and fashions  Have high expectations of evidence
  14. 14. Managers used to working in a different way: Need to act quickly: Speed more important than accuracy Organizational politics Formal authority and hierarchies Over-emphasize experience Rewarded for getting things done not doing what works
  15. 15. Sometimes we are evidence-based Try to gather data and information Invest time and effort in trying to understand and apply it Question our and others‟ assumptions and logic Are sceptical about what appear to be fads Resist the temptation to act quickly
  16. 16. Sometimes we are not so evidence-based Act on gut feeling (though intuition can be important for some decisions) Copy other people who appear successful (benchmarking) Think there is one ideal way (best practice) Let the „solution‟ frame and define our „problem‟ and create need (kitchen gadgets) Want to fit in and be as cool as everyone else (fashion)
  17. 17. Postgraduate Course 2. Evidence based management: Where does it come from?
  18. 18. Medicine: Founding fathersPostgraduate Course David Sackett Gordon Guyatt McMaster University Medical School, Canada
  19. 19. Problem I: too much informationPostgraduate Course  More than 1 million articles in 40.000 medical journals per year (= 1995; now probably more than 2 million). For a specialist to keep up this means reading 25 articles every day (for a GP more than 100!)  Most of the new insights and treatment methods don‟t reach the target group
  20. 20. Problem II: persistent convictionsPostgraduate Course if you’re breathe into a bag hyperventilating
  21. 21. Problem III: jumping to conclusionsPostgraduate Course people who have an give them a drug that irregular heartbeat are reduces the much more likely to die number of of coronary disease irregular beats
  22. 22. Postgraduate Course Examples of mechanistic reasoning gone wrong:  Oestrogen replacement therapy to reduce cardiac events and stroke in post-menopausal women.  Treatment of measles with antibiotics.  Rest for recovery.  Placing babies on their fronts to prevent Sudden Infant Death Syndrome (SIDS).  Debriefing after psychological trauma
  23. 23. Problem IVPostgraduate Course David Sackett Half of what you learn in medical school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half. The most important thing to learn is how to learn on your own. (Remember that your teachers are as full of bullshit as your parents)
  24. 24. Evidence based decisionPostgraduate Course David Sackett “Good doctors use both individual clinical expertise and the best available external evidence, and neither alone is enough. Without clinical expertise, practice risks becoming tyrannized by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient. Without current best evidence, practice risks becoming rapidly out of date, to the detriment of patients.”
  25. 25. Management: Founding MotherPostgraduate Course
  26. 26. Management: Founding FathersPostgraduate Course Jeffrey Pfeffer Robert Sutton
  27. 27. Postgraduate Course 2. Evidence-based management: Why do we need it?
  28. 28. EBMgt: some basic assumptionsPostgraduate Course  Research produced by management scholars could be useful to organizations  Drawing on available evidence (including research produced by academics) is likely to improve decisions  Organizations do not appear to be strongly aware of nor use research findings  EBMgt is a potentially useful way of thinking about how we can incorporate research evidence into decision-making
  29. 29. Postgraduate Course Reason 1: Errors and Biases of Human Judgment
  30. 30. Errors and Biases of Human JudgmentPostgraduate Course  Seeing order in randomness  Mental corner cutting  Misinterpretation of incomplete data  Halo effect  False consensus effect  Confirmation bias  Reinterpreting evidence  Authority bias  Group think  In-group bias  Self serving bias  Recall bias  Sunk cost fallacy  Anchoring bias  Cognitive dissonance reduction  Inaccurate covariation detection  Distortions due to plausibility
  31. 31. Errors and Biases of Human JudgmentPostgraduate Course  Seeing order in randomness  Mental corner cutting  Misinterpretation of incomplete data  Halo effect  Confirmation bias  False consensus effect  Authority bias  Reinterpreting evidence  In-group bias  Group think  Recall bias  Self serving bias  Anchoring bias  Sunk cost fallacy  Inaccurate covariation detection  Cognitive dissonance reduction  Distortions due to plausibility
  32. 32. Seeing order in randomnessPostgraduate Course We are predisposed to see order, pattern and causal relations in the world. Patternicity: The tendency to find meaningful patterns in both meaningful and meaningless noise.
  33. 33. Seeing order in randomnessPostgraduate Course We are pattern seeking primates: association learning
  34. 34. Points of impact of V-1 bombs in LondonPostgraduate Course
  35. 35. Points of impact of V-1 bombs in LondonPostgraduate Course
  36. 36. Errors and Biases of Human JudgmentPostgraduate Course  A Type I error or a false positive, is believing a pattern is real when it is not (finding a non existent pattern)  A Type II error or a false negative, is not believing a pattern is real when it is (not recognizing a real pattern) Dr. Michael Shermer (Director of the Skeptics Society)
  37. 37. Errors and Biases of Human JudgmentPostgraduate Course  A Type I error or a false positive: believe that the rustle in the grass is a dangerous predator when it is just the wind (low cost)
  38. 38. Errors and Biases of Human JudgmentPostgraduate Course  A Type II error or a false negative: believe that the rustle in the grass is just the wind when it is a dangerous predator (high cost)
  39. 39. Errors and Biases of Human JudgmentPostgraduate Course Pattern detection problem Assessing the difference between a Type I and Type II error is highly problematic (especially in split second „life and death‟ situations), so the default position is to assume all patterns are real.
  40. 40. Errors and Biases of Human JudgmentPostgraduate Course Jennifer Whitson, University of Texas Austin, corporate environments
  41. 41. Errors and Biases of Human JudgmentPostgraduate Course Erroneous beliefs plaque both experienced professionals and less informed laypeople alike. stress peptic ulcer
  42. 42. Oct 2005 Peptic ulcer – an infectious disease!This years Nobel Prize in Physiology or Medicine goes to Barry Marshall and RobinWarren, who with tenacity and a prepared mind challenged prevailing dogmas. Byusing technologies generally available (fibre endoscopy, silver staining ofhistological sections and culture techniques for microaerophilic bacteria), theymade an irrefutable case that the bacterium Helicobacter pylori is causing disease.By culturing the bacteria they made them amenable to scientific study.In 1982, when this bacterium was discovered by Marshall and Warren, stress andlifestyle were considered the major causes of peptic ulcer disease. It is nowfirmly established that Helicobacter pyloricauses more then 90% of duodenal ulcers.The link between Helicobacter pyloriinfection and peptic ulcer disease has beenestablished through studies of humanvolunteers, antibiotic treatment studies andepidemiological studies.
  43. 43. Errors and Biases of Human JudgmentPostgraduate Course Doctors, teachers, lawyers and managers hold many erroneous beliefs, not because they are ignorant or stupid, but because they seem to be the most sensible conclusion consistent with the available evidence. They hold such beliefs because they seem to be the irresistible products of their own professional experience. They are the products, not of irrationality, but of flawed rationality
  44. 44. Errors and Biases of Human JudgmentPostgraduate Course  Managers seem to be extremely good at generating ideas, theories, and explanations that have the ring of plausibility. They may be relatively deficient, however, in evaluating and testing those ideas once they are formed.  This requires that we think critically about experience, question our assumptions, and challenge what we think we know  (Show me the evidence!)
  45. 45. Postgraduate Course 3. Evidence-based management: What is stopping us?
  46. 46. Quick fixes (1)Postgraduate Course What is the quick fix? A „solution‟ which  Focuses on style and presentation not content  Is not evaluated  Is always slower than we hoped  Usually doesn‟t work  Is followed by another quick fix  Everybody forgets and becomes subject to organizational amnesia
  47. 47. Quick fixes (2)Postgraduate Course So why do we do quick fixes?  Can be career-enhancing for managers (e.g., issue selling, kick-ass CEOs)  Speed is often valued over accuracy  Heavily sold and marketed  Are we all looking for quick and easy solutions? So who needs or wants academic research?
  48. 48. •57
  49. 49. •58
  50. 50. •59
  51. 51. •60
  52. 52. •61
  53. 53. •62
  54. 54. •63
  55. 55. •64
  56. 56. Management Fads (1)Postgraduate Course The nearly-forgotten fads  Scientific Management/Taylorism  Business Process Reengineering  Management by results  Excellence  Total Quality Management  Learning Organizations  Knowledge Management
  57. 57. Management Fads (2)Postgraduate Course The fads that haven‟t been forgotten (yet)  Talent management  Management development  Executive coaching  Emotional intelligence  Employee engagement  Myers Briggs Type Indicator  Belbin Team Roles General concern about the destructive impact of fads from both practitioners and researchers
  58. 58. •67
  59. 59. •68
  60. 60. •69
  61. 61. •70
  62. 62. •71
  63. 63. •72
  64. 64. •73
  65. 65. •74
  66. 66. •75
  67. 67. •76
  68. 68. * FADS SEEM TO BE ATTRACTIVE, COMPELLING AND IRRESISTIBLE Promise to deliver a lot and fast Appear simple New and shiny Will make everything alright and help contain anxieties around intractable problems Help user feel effective and cutting edge Bits of some fads may work in some contextsSo who needs or wants academic research?*Evidence-based management not a fad!
  69. 69. •78
  70. 70. •79
  71. 71. dDwtNzI3 •80
  72. 72. •81
  73. 73. Errors and Biases of Human JudgmentPostgraduate Course  Seeing order in randomness  Mental corner cutting  Misinterpretation of incomplete data  Halo effect  False consensus effect  Confirmation bias  Reinterpreting evidence  Authority bias  Group think  In-group bias  Recall bias  Anchoring bias  Inaccurate covariation detection  Distortions due to plausibility
  74. 74. Under pressure to adopt fadsPostgraduate Course “And there we see the power of any big managerial idea (or fad). It may be smart, like quality, or stupid, like conglomeration. Either way, if everybodys doing it, the pressure to do it too is immense. If it turns out to be smart, great. If it turns out to be stupid, well, you were in good company and most likely ended up no worse off than your competitors. Your companys board consists mostly of CEOs who were probably doing it at their companies. How mad can they get?
  75. 75. Under pressure to adopt fadsPostgraduate Course The true value of conventional management wisdom is not that its wise or dumb, but that its conventional. It makes one of the hardest jobs in the world, managing an organization, a little easier. By following it, managers everywhere see a way to drag their sorry behinds through another quarter without getting fired. And isnt that, really, what its all about?” (Colvin, 2004, Fortune)
  76. 76. Postgraduate Course 4. Evidence based management: How does it look like in practice?
  77. 77. Four sourcesPostgraduate Course
  78. 78. Postgraduate Course JAMA, 1992
  79. 79. Push vs PullPostgraduate Course Push: teaching (management) principles based upon a convergent body of research and telling students what to do. Pull: teaching (managers) how to find, appraise and apply the outcome of research (evidence) by themselves
  80. 80. The 5 steps of ‘pull’ EBPPostgraduate Course 1. Formulate an answerable question 2. Search for the best available evidence 3. Critically appraise the evidence 4. Integrate the evidence with your managerial expertise and organisational concerns and apply 5. Monitor the outcome
  81. 81. The 5 steps of ‘pull’ EBPPostgraduate Course 1. Formulate an answerable question 2. Search for the best available evidence 3. Critically appraise the evidence 4. Integrate the evidence with your managerial expertise and organisational concerns and apply 5. Monitor the outcome
  82. 82. Answerable questionPostgraduate Course I am a consultant, my client a large health-care organization. The board of directors has plans for a merger with a smaller healthcare organization. However, it‟s been said that the organizational culture differs widely between the two organizations. The board want‟s to know if this can impede a successful outcome.
  83. 83. Answerable question: PICO(C)Postgraduate Course P = Population or problem I = Intervention or successfactor C = Comparison O = Outcome C = Context
  84. 84. Answerable question: PICOCPostgraduate Course P: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff? O: What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction? P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument?
  85. 85. The 5 steps of ‘pull’ EBPPostgraduate Course 1. Formulate an answerable question 2. Search for the best available evidence 3. Critically appraise the evidence 4. Integrate the evidence with your managerial expertise and organisational concerns and apply 5. Monitor the outcome
  86. 86. Where do we search?Postgraduate Course
  87. 87. Where do we search?•Postgraduate School
  88. 88. The 5 steps of ‘pull’ EBPPostgraduate Course 1. Formulate an answerable question 2. Search for the best available evidence 3. Critically appraise the evidence 4. Integrate the evidence with your managerial expertise and organisational concerns and apply 5. Monitor the outcome
  89. 89. Critical appraisalPostgraduate Course How to read a research article?
  90. 90. Critical appraisalPostgraduate Course 1. Study designs 2. Levels of evidence 3. Bias / confounding 4. Effect sizes 5. External validity
  91. 91. Research designsPostgraduate Course Which study for which question? The “best” evidence depends on the question type !
  92. 92. Levels of evidencePostgraduate Course
  93. 93. The 5 steps of ‘pull’ EBPPostgraduate Course 1. Formulate an answerable question 2. Search for the best available evidence 3. Critically appraise the evidence 4. Integrate the evidence with your managerial expertise and organisational concerns and apply 5. Monitor the outcome
  94. 94. Organization concernsPostgraduate Course Always ask yourself to what extent the evidence is applicable in your situation: 1. Is your organization / division / population so different from those in the study that its results cannot apply? 2. How relevant is the study to what you are seeking to understand or decide? 3. What are your organization‟s potential benefits and harms from the intervention? 4. Is the intervention feasible in your setting?
  95. 95. The 5 steps of ‘pull’ EBPPostgraduate Course 1. Formulate an answerable question 2. Search for the best available evidence 3. Critically appraise the evidence 4. Integrate the evidence with your managerial expertise and organisational concerns and apply 5. Monitor the outcome
  96. 96. Monitor the outcomePostgraduate Course  posttest?  pretest?  control group?
  97. 97. Do a trial!Postgraduate Course
  98. 98. Postgraduate Course (*). Why should academics be interested in EBMgt?
  99. 99. Postgraduate Course (5). Evidence based management: What can you do as a PhD student?
  100. 100. BarriersPostgraduate Course
  101. 101. BarriersPostgraduate Course Best available evidence = systematic reviews
  102. 102. Postgraduate Course
  103. 103. Postgraduate Course
  104. 104. Postgraduate Course
  105. 105. Postgraduate Course Better than a single study: a replication study Better than a replication study: a systematic review / meta analysis If there were 100 RCT‟s, 99 of which gave a „negative‟ result (where, say, the new intervention appeared to be harmful), while one had a „positive‟ result (were the intervention appeared helpful), it would obviously be a mistake to consider only the single positive RCT.
  106. 106. TYPES OF LITERATURE REVIEWExplicit systematic: Explicit use of rigorous method - can vary as least as much as the range of methods in primary researchImplicit systematic: rigorous method but not statedFalse systematic: described as systematic but with little evidence of explicit rigorous methodArgument/thematic: a review that aims to explore and usually support a particular argument or theme with no pretension to use an explicit rigorous method (though thematic reviews can be systematic)Expert or ad hoc review: informed by the skill and experience of the reviewer but no clear method so open to hidden bias.Rapid evidence assessment: a rapid review that may or may not be rigorous and systematic. If it is systematic then in order to be rapid it is likely to be limited in some explicit aspect of scope. (Gough 2007) •115
  107. 107. LITERATURE REVIEWS INMANAGEMENTHow many here have had training in reviewing literature?Are we really “standing on the shoulders of giants”?Do you recognize these sort of unqualified statements? • “Previous studies have shown that…” • “It has been demonstrated that…”But how many studies? Demonstrated how? Did other studies find something else?Very few systematic reviews in management •116
  108. 108. WHAT QUALITIES SHOULDLITERATURE REVIEWS HAVE?
  109. 109. WHAT QUALITIES SHOULDLITERATURE REVIEWS HAVE?Comprehensive? Focused?Reader-friendly? Exploratory?Informative? Inclusive? (of different typesBalanced? of evidence)Insightful? Transparent?Critical? Accurately referenced?Rigorous? Objective?Accessible? Replicable?User led? Interesting?Up-to-date? Standardized? •118
  110. 110. WHAT IS A SYSTEMATICREVIEW?It‟s research on existing researchWith a clear, explicit and replicable methodology • Clear review question • Search strategy • Quality criteriaAllows us to draw reliable conclusions about what we know and do not know about a given question or problem •119
  111. 111. Systematic reviewPostgraduate Course The intention behind a systematic review is to identify as fully as possible all the scientific studies of relevance to a particular subject and to assess the validity and authority of the evidence of each study separately. As the name indicates, a systematic review takes a systematic approach to identifying studies and has the methodological quality critically appraised by multiple researchers independently of each other, as a consequence of which the review is transparent and reproducible and can be monitored. The use of statistical analysis techniques in a systematic review to pool the results of the individual studies numerically in order to achieve a more accurate estimate of the effect is termed a “meta-analysis”.
  112. 112. THE QUESTIONS SRs ANSWER For any given specific problem:What do we know?What do we not know?What are we not sure about?How do we know we know or don‟t know or are not sure that…?What is the basis for our claims? (e.g., How much evidence? What quality?) •122
  113. 113. Stages of a SR SYSTEMATIC REVIEW 1. problem formulation; 2. locatingPractice- studies; What do Informs Existing research we practicerelevant studies 3. study know? selection Informsquestion What we future and evaluation; do not research know? questions 4. analysis and synthesis; 5. reporting of the results
  114. 114. Stages of a systematic reviewPostgraduate Course 1. Formulate a focussed review question 2. Search for the best available evidence 3. Select relevant studies 4. Critically appraise the evidence 5. Synthesise the findings 6. Report what is known
  115. 115. WHAT SORT OF QUESTIONSCAN BE ADDRESSED IN A SR? Each would require much more specificityDoes team-building work?Can you improve emotional intelligence?Do increases in EI lead to performance improvements?Does management development improve the performance of managers?Does employee engagement predict organizational performance?Is 360 degree feedback effective?Can potentially great leaders be identified?Is coaching effective? •125
  116. 116. Does team-building work?How would you make this question morespecific
  117. 117. PICOC & CIMOPostgraduate Course P = Population C = Context I = Intervention or factor I = Intervention or factor C = Comparison M = Mechanism O = Outcome O = Outcome C = Context
  118. 118. WHAT SORT OF QUESTIONSCAN BE ADDRESSED IN A SR?Does team-building work?• What is meant by „team‟? And what is not included as a „team‟?• What kind of teams?• In which particular contexts or settings?• What is „team building‟? And what is not „team building‟?• What does „work‟ mean?• „Work‟ compared to any other team intervention? No intervention?• What outcomes are relevant?• What are the mechanisms, processes and theory which might account for possible effects of team building on outcomes?• What time periods are relevant for observing any possible effects?• What about possible negative effects or harm?• What types of data from what sorts of designs would in principle provide good quality, medium quality and poor quality evidence? •128
  119. 119. Which study for which question?Postgraduate School Qualitative Observational Controlled Research question Surveys studies studies studies Effectiveness: does it work?, does A work better than B? + ++ Process: how does it work, why does it work? ++ + Context: in what circumstances does it work, for whom? ++ + + Safety: will it do more good than harm? + + ++ Acceptability: will the target group accept the intervention / new method of working? ++ + + Cost effectiveness: does it reduce costs? is A cheaper than B? ++ Appropriateness: is this the right intervention / method for this target group? ++ ++ Satisfaction: is the target group satisfied with the new method of working? ++ ++ +
  120. 120. Stages of a systematic reviewPostgraduate Course 1. Formulate a focussed review question 2. Search for the best available evidence 3. Select relevant studies 4. Critically appraise the evidence 5. Synthesise the findings 6. Report what is known
  121. 121. MAKING DECISIONS ABOUTSEARCH STRATEGYWhat sources of evidence?What sources will you include or exclude and why?How iterative can you be? • Test the question doing some simple searches • Does the question work? • Does the search strategy work? •131
  122. 122. Search strategyPostgraduate School Two types of search strategies Snowball method Building blocks method
  123. 123. Snowball methodPostgraduate School Starting from one book or article, you search for other literature on the same topic. Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article). Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge).
  124. 124. Snowball methodPostgraduate School ISI Web of Knowledge
  125. 125. Building blocks methodPostgraduate School Keyword 1 Keyword 2 Keyword 3 Keyword 4 Synonyms or Synonyms or Synonyms or Synonyms or related terms related terms related terms related terms • …. • …. • …. • …. • …. OR • …. OR • …. OR • …. • …. • …. • …. • …. • …. • …. • …. • …. AND AND AND
  126. 126. Postgraduate School
  127. 127. Building blocks methodPostgraduate School
  128. 128. Building blocks methodPostgraduate School
  129. 129. Stages of a systematic reviewPostgraduate Course 1. Formulate a focussed review question 2. Search for the best available evidence 3. Select relevant studies 4. Critically appraise the evidence 5. Synthesise the findings 6. Report what is known
  130. 130. Stages of a systematic reviewPostgraduate Course 1. Formulate a focussed review question 2. Search for the best available evidence 3. Select relevant studies 4. Critically appraise the evidence 5. Synthesise the findings 6. Report what is known
  131. 131. Standard appraisal questions•Postgraduate School 1. Did the study adress a clearly focused issue? 2. Is the sample size justified? 3. Is the design appropriate to the stated aims? 4. Are te measurements likely to be valid and reliable? 5. Are the statistical methods described? 6. Did untoward events occur during the study? 7. Were the basic data adequately described? 8. Do the numbers add up? 9. Was the statistical significance assessed? 10. What do the findings mean? 11. Are important effects overlooked? 12. What implications does the study have for your practice?
  132. 132. Stages of a systematic reviewPostgraduate Course 1. Formulate a focussed review question 2. Search for the best available evidence 3. Select relevant studies 4. Critically appraise the evidence 5. Synthesise the findings 6. Report what is known
  133. 133. Once a body of evidence has beencollated….How relevant is this to what we are seeking to understand or decide?How representative is this of the population that concerns us?How reliable, how well-founded theoretically, empirically is it? „These are tough but necessary tests for evidence based policy and practice‟ Solesbury 2004
  134. 134. Stages of a systematic reviewPostgraduate Course 1. Formulate a focussed review question 2. Search for the best available evidence 3. Select relevant studies 4. Critically appraise the evidence 5. Synthesise the findings 6. Report what is known
  135. 135. What does synthesis mean in MOS?Analysis, • “…is the job of systematically breaking down something into its constituent parts and describing how they relate to each other – it is not random dissection but a methodological examination” • Is the aim is to extract key data, ideas, theories, concepts [arguments] and methodological assumptions from the literature?Synthesis, • “…is the act of making connections between the parts identified in analysis. It is about recasting the information into a new or different arrangement. That arrangement should show connections and patterns that have not been produced previously” (Hart, 1998: p.110)
  136. 136. Summarizing evidenceWhat does the evidence say?Consistency of evidence?Quality of evidenceQuantity of evidence (avoiding double counting) •146
  137. 137. The example of management andleadership developmentDoes leadership development work?What do we know?What is the „best‟ research evidence available?How can this evidence be „put together‟?What are the strengths and weaknesses of different approaches to synthesis?
  138. 138. What methods of synthesis are available? (1/2) Aggregated synthesis Meta analysis Analytic induction Meta ethnography Bayesian meta analysis Meta narrative mapping Meta needs assessment Case Survey Meta synthesis Comparative case study Metaphorical analysis Constant targeted comparison Mixed method synthesis Content analysis Narrative synthesis Critical interpretive synthesis Quasi statistics Realist synthesis Cross design synthesis Reciprocal analysis Framework analysis Taxonomic analysis Grounded theory Thematic synthesis Hermeneutical analysis Theory driven synthesis Logical analysis
  139. 139. What methods of synthesis are available? (2/2) Synthesis by aggregation • extract and combine data from separate studies to increase the effective sample size. Synthesis by integration • collect and compare evidence from primary studies employing two or more data collection methods. Synthesis by interpretation • translate key interpretations / meanings from one study to another. Synthesis by explanation • identify causal mechanisms and how they operate. •(Rousseau, Manning, Denyer, 2008)
  140. 140. Van Buren & Erskine, 2002 (building onKirkpatrick)Organizations reported collecting data on:78% reaction (how participants have reacted to the programme)32% learning (what participants have learnt from the programme)9% behaviour (whether what was learnt is being applied on the job)7% results (whether that application is achieving results)
  141. 141. Stages of a systematic reviewPostgraduate Course 1. Formulate a focussed review question 2. Search for the best available evidence 3. Select relevant studies 4. Critically appraise the evidence 5. Synthesise the findings 6. Report what is known
  142. 142. Postgraduate School What do you do with this?
  143. 143. Does management and leadershipdevelopment workOverall, the results suggest a medium to large effect size for learning and behaviour (largely based on self report)Absence of evidence of impact of business impact
  144. 144. Do the results of synthesis create clarity -or confusion, conflict and controversy? “A wide variety of program outcomes are reported in the literature – some that are effective, but others that are failing. In some respects the lessons for practice can be found in the wide variance reported in these studies. The range of effect sizes clearly shows that it is possible to have very large positive outcomes, or no outcomes at all” (p. 240/241) (Collins and Holton, 1996: 240/241)
  145. 145. Do the results of synthesis create clarity -or confusion, conflict and controversy? “Organizations should feel comfortable that their managerial leadership development programmes will produce substantial results, especially if they offer the right development programs for the right people at the right time. For example, it is important to know whether a six-week training session is enough or the right approach to develop new competencies that change managerial behaviours, or it is individual feedback from a supervisor on a weekly basis regarding job performance that is most effective?” (Collins and Holton, 1996: 240/241)
  146. 146. systematic reviewPostgraduate Course Example
  147. 147. Flexible working conditions and their effects onemployee health and wellbeing (Joyce et al, 2010)Background: Flexible working conditions are increasingly popular in developed countries but the effects on employee health and wellbeing are largely unknown.Objectives: To evaluate the effects (benefits and harms) of flexible working interventions on the physical, mental and general health and wellbeing of employees and their families.Search strategy: Our searches (July 2009) covered 12 databases including the Cochrane Public Health Group Specialized Register, CENTRAL; MEDLINE; EMBASE; CINAHL; PsycINFO; Social Science Citation Index; ASSIA; IBSS; Sociological Abstracts; and ABI/Inform. We also searched relevant websites, hand searched key journals, searched bibliographies and contacted study authors and key experts. •157
  148. 148. Flexible working conditions and their effects onemployee health and wellbeing (Joyce et al, 2010)Selection criteria: Randomized controlled trials (RCT), interrupted time series and controlled before and after studies (CBA), which examined the effects of flexible working interventions on employee health and wellbeing. We excluded studies assessing outcomes for less than six months and extracted outcomes relating to physical, mental and general health/ill health measured using a validated instrument. We also extracted secondary outcomes (including sickness absence, health service usage, behavioral changes, accidents, work-life balance, quality of life, health and wellbeing of children, family members and co-workers) if reported alongside at least one primary outcome.Data collection and analysis: Two experienced review authors conducted data extraction and quality appraisal. We undertook a narrative synthesis as there was substantial heterogeneity between studies. •158
  149. 149. Flexible working conditions and their effects on employee health and wellbeing (Joyce et al, 2010)Main results: Ten studies fulfilled the inclusion criteria. Six CBA studies reported on interventions relating to temporal flexibility: self-scheduling of shift work (n = 4), flexitime (n = 1) and overtime (n = 1). The remaining four CBA studies evaluated a form of contractual flexibility: partial/gradual retirement (n = 2), involuntary part-time work (n = 1) and fixed-term contract (n = 1). The studies retrieved had a number of methodological limitations including short follow-up periods, risk of selection bias and reliance on largely self-reported outcome data. Four CBA studies on self- scheduling of shifts and one CBA study on gradual/partial retirement reported statistically significant improvements in either primary outcomes (including systolic blood pressure and heart rate; tiredness; mental health, sleep duration, sleep quality and alertness; self-rated health status) or secondary health outcomes (co-workers social support and sense of community) and no ill health effects were reported. Flexitime was shown not to have significant effects on self-reported physiological and psychological health outcomes. Similarly, when comparing individuals working overtime with those who did not the odds of ill health effects were not significantly higher in the intervention group at follow up. The effects of contractual flexibility on self-reported health (with the exception of gradual/partial retirement, which when controlled by employees improved health outcomes) were either equivocal or negative. No studies differentiated results by socio-economic status, although one study did compare findings by gender but found no differential effect on self- reported health outcomes. •159
  150. 150. Flexible working conditions and their effects onemployee health and wellbeing (Joyce et al, 2010)Authors’ conclusions: The findings of this review tentatively suggest that flexible working interventions that increase worker control and choice (such as self scheduling or gradual/partial retirement) are likely to have a positive effect on health outcomes. In contrast, interventions that were motivated or dictated by organizational interests, such as fixed-term contract and involuntary part-time employment, found equivocal or negative health effects. Given the partial and methodologically limited evidence base these findings should be interpreted with caution. Moreover, well- designed intervention studies are needed to delineate the impact of flexible working conditions on health, wellbeing and health inequalities. •160

×