Downloaded 63 times
Uploaded byCenter for Evidence-Based Management
Using CATs and REAs to inform decision-making
This document discusses critically appraised topics (CATs) and rapid evidence assessments (REAs) as tools for evidence-based practice in management. It provides an overview of what CATs and REAs are, which are shorter versions of systematic literature reviews. The document outlines the core principles of CATs, REAs, and systematic reviews, including being systematic, transparent, replicable, and synthesizing evidence. It also discusses how to structure a CAT and provides examples of CAT topics. Finally, it discusses how CATs and REAs can be used to inform decision making by considering practitioners' expertise, organizational data, scientific literature, and stakeholder values when identifying problems and solutions.
More Related Content
Similar to Using CATs and REAs to inform decision-making
More from Center for Evidence-Based Management
Using CATs and REAs to inform decision-making
- 1. 1 Using CATs andREAs to inform decision-making CEBMa Annual Meeting: How to Teach and apply Evidence-Based Practice theWit Hotel Chicago 9 August 2018 Rob B Briner www.cebma.org
- 2. 2 What is it? The conscientious (effort), explicit (clarity) and judicious (critical of quality) use of the best available evidence from multiple sources to increase the likelihood of a favourable outcome – It’s about the process – It’s not about certainties (this will work) – It’s is about probabilities and likelihoods – It is about reducing uncertainty (given our context this is more likely to lead to the outcome we want than doing something else or doing nothing) 2
- 3.
- 4.
- 5.
- 6. 6 What are CATsand REAs? Critically Appraised Topics Rapid Evidence Assessment They are shorter versions of systematic literature reviews of the literature They are the “scientific literature” part of EBMgt 6
- 7. 7 Possible purposes andproperties of literature reviews Critical? Comprehensive? Detailed? Exhaustive? Objective? Focused? Descriptive? Practically relevant? Analytic? Theoretical? Gap-identifying? Summarizing? Selective? Interesting? Thematic? Systematic? 7
- 8. 8 What types ofliterature review are conducted in management? Literature reviews motivating empirical studies Formal full-length literature reviews by academics Meta analyses Reviews or overviews in textbooks Reviews in popular management books Reviews in management usually narrative not systematic 8
- 9. 9 Nature of claimsmade by most of these reviews For example: – Previous research has shown that team building improves performance – It has been demonstrated that management development is effective – Many studies have shown that employee engagement increases performance – There is much evidence that job stress causes ill health 9
- 10. 10 Nature of claimsmade by most of these reviews BUT! – Did all previous research show this? – What proportion of previous research? – How many studies? – How strongly or clearly or consistently was this shown? – Were the study designs such that the conclusions reached could be justified? – What did the authors do to avoid the biases of pre-existing beliefs? 10
- 11. 11 Disadvantages of traditional managementliterature reviews Do not usually focus on a specific (or practice-relevant) question Have diverse aims and purposes Adopt a wide variety of approaches and structure Do not use particular methods or do not explicitly state the methods used to conduct the review Are more prone to bias because, for example, authors can select studies that support their views and ignore those that do not Are less likely to identify the best available evidence Do not aim to be comprehensive by including all relevant and available research Are not replicable nor easy to update Do not tell us what we know and do not know and the bases for that ignorance/knowledge Cannot ‘stand on the shoulders of giants’ (and all research findings) 11
- 12. 12 What are systematicreviews? They are research on existing research Precise question (like a research question) Explicit methodology Replicable Make it clear: – What is known – What is not known – And the basis for those claims 12
- 13. 13 What are they? “systematic reviews never provide ‘answers’. What they do is report as accurately as possible what is known and not known about the questions addressed in the review” (Briner, Denyer, & Rousseau, 2009, p. 27). 13
- 14. 14 Core principles ofREAs, CATs and SRs Systematic/organized: Systematic reviews are conducted according to a system or method which is designed in relation to and specifically to address the question the review is setting out to answer. Transparent/explicit: The method used in the review is explicitly stated. Replicable/updatable: As with many forms of primary research, the method and the way it is reported should be sufficiently detailed and clear such that other researchers can repeat the review, repeat it with modifications or update it. Synthesize/summarize: Systematic reviews pull together in a structured and organized way the results of the review in order to summarize the evidence relating to the review question. 14
- 15.
- 16. 16 Structure of aCAT? Section 1. Background: What is the context of the CAT question? Section 2. What does the CAT answer? Section 3. Defining inclusion criteria: Which studies will be taken into account? Section 4. Search strategy: How should the studies be sought? Section 5. Study selection: How should you select the studies? Section 6. Data extraction: What information should be extracted? Section 7. Critical appraisal: How should the quality of the studies be judged? Section 8. Results: What did you find? Section 9. Conclusion Section 10. Limitations Section 11. Implications and recommendations 16
- 17.
- 18. 18 Review questions Whatis known in the scientific literature about the impact of (1) goal setting and (2) performance appraisal on workplace performance? What is meant by (1) goal setting and (2) performance appraisal? (What are they?) What are the assumed causal mechanisms? (How are they supposed to work?) What is the effect of (1) goal setting and (2) performance appraisal on workplace performance? What is known about possible moderators and/or mediators that affect the relationship between (1) goal setting and (2) performance appraisal and workplace performance? What is known about the reliability and validity of performance appraisal? 18
- 19. 19 Example CAT topicsfrom Masters in HRM What is known in the scientific literature about – the effects of flexible working interventions on: (a) Work-nonwork conflict? And, (b) Psychological well-being? – interventions to increase innovative behaviour at work? – (a) The validity and reliability of measures of unconscious bias? and, (b) The effects of unconscious bias training on unconscious bias? – the effects of mindfulness training on the well-being and performance of employees? – the effects of interventions aimed at improving safety culture? – the effects of interventions aimed at reducing sexual harassment? – the effects of training interventions designed to improve multicultural working in teams – interventions to increase the percentage of women in senior management positions in large organizations? – interventions and techniques which aim to improve the performance of employees identified as poor performers? 19
- 20. 20 Using CATs andREAs to inform decision- making 20
- 21. 21 Example of employeeengagement (EE) Suppose you or your organization believe that low EE is somehow a problem in your organization And because it is you should measure EE and attempt to increase ‘low’ scores somehow How would you (a) approach this problem and (b) identify a solution from an evidence-based practice approach? 21
- 22. 22 22 Element 1: Practitioners’professional expertise Identifying the problem – Have I/we seen EE problems before? What happened? – Based on our experience, is the level of EE a problem? – What do I/we believe about causes and consequences of low EE? Identifying solution (only if EE is a problem) – Have I/we seen EE interventions before? What happened? – What do I/we believe about EE interventions? – Based on our experience, is the level of EE here a problem? What are costs & benefits of intervening? How relevant and applicable and trustworthy is my/our expertise?
- 23. 23 23 Element 2: Organizationaldata Identifying the problem – What actually is the EE level? – Are there patterns or trends in EE? – Do data show how EE is a problem? – Do data show that low EE is causing problems? Identifying solution (only if EE is a problem) – What attempts to enhance EE are currently in place and are they working? – What else is happening that might be affecting EE? – Are there relationships between EE and other data? Employee type? Shift? Department? How relevant and applicable and trustworthy are our organizational data?
- 24. 24 24 Element 3: Scientificliterature Identifying the problem – What are the average rates of EE in my sector and location – is the level here ‘low’? – What does systematically reviewed research evidence suggest are the problems with low EE? Identifying solution (only if EE is a problem) – What does research evidence from systematic reviews suggest are major causes of low EE? – What does research evidence from systematic reviews suggest as effective interventions to increase EE? How relevant and applicable and trustworthy are the scientific findings?
- 25. 25 25 Element 4: Stakeholdersvalues and concerns Identifying the problem – How do employees feel about and view the EE ‘problem’? – Do they see negative consequences? – What do managers think about the problem? – Do customers or clients or service users have a view? Identifying solution (only if EE is a problem) – How do employees feel about and view the solutions? – What do managers think about the solutions? – What alternative explanations and proposed solutions do others have? How relevant and applicable and trustworthy are stakeholder concerns evidence?
- 26. 26 How would youuse the scientific evidence to help make a decision? 26
- 27. 27 The value ofaggregation: Scientific evidence in context Is there in principle likely to be a practical problem? Is the proposed solution in principle likely to be effective Scientific evidence is not just about findings – it elaborates understanding of the problem and potential solution Helps tell you extent to which scientific evidence is relevant to your context Helps to redefine nature of problem and solution – we thought it was X but more likely is Y May give insight into causal pathways Indications about how important problem/solution is compared to others (X seems to influence Y but effect size much smaller than A, B, and C) 27
- 28. 28 Questions? Comments? Thoughts? r.briner@qmul.ac.uk RobB Briner www.cebma.org