Critical Appraisal of the Medical Education Literature 2012

Critical appraisal in the health professions: a session for UBC-Maastricht MPHE programme, 2012Dean Giustini, UBC Biomed Librarian | MHPE Teleconference, 13 July 2012

Learning goals• To provide overview of critical appraisal & key principles• Map out the literature in medical education• Critically appraise relevance of published literature in medical education• To define BEME: Best evidence in medical education

Revisit evidence-based practiceEducational research starts with scenario • Formulate a focused question • Search for best evidence • Appraise evidence • Apply evidence in practice • Critically evaluate whole“…research, including qualitative studies, form the basis of evidence-based practice for health professionals. …however, many practicing health educators do not feel fully confident in their ability to critically appraise qualitative research… Jeanfreau, 2011

Mapping medical education literature A widely-disseminated literature Masses of information Rapid expansion & change of literature Essential to engage with scholarly contributions of others in teaching & learning Reflection on one’s own teaching & best practices in general BEME: basing your decisions about teaching practices on the available evidence …more later!

Other reflections on educational research Quantitative methods • Gold standard is RCT, systematic review • Numbers used to describe & analyze • Precise answers are found for defined questions Qualitative methods • No real gold standard • Words are used to describe & analyze • Seek to understand how people make sense of & interpret • Focus on perspectives & attitudes of individual • Enquiry is based in a culture or natural setting of individual(s)

Dr. Kevin Eva says “…we need to look for fatal flaws in research methodologies thatmight invalidate the conclusions being drawn in any given study and, yes we want to ensure that research in the field maintains a focus on helping educators solve the practical dilemmas they face on a day-to-day basis. ….still, the extent to which wedefine our efforts purely based on any particular methodological criteria is the extent to which we miss a great opportunity to truly stake claims to progression in our thinking and our educational innovations…” Eva KW. Broadening the debate about quality in medical education research. Med Educ. 2009 Apr;43(4):294-6

Types of qualitative methodologies used• No ‘gold standard’ or hierarchy • researchers match method to research question (or should) • no evidence-based pyramid • triangulation of methods/’mixed’ is popular• Popular methodologies • Ethnography • Grounded theory • Phenomenology • other? Daly J, et al. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007 Jan;60(1):43-9.

More differences between Qualitative/Quantitative ResearchQualitative Quantitative • A focus on text, depth • A focus on numbers, breadth • Answers ‘how / why’ • Answers ‘what / when / how much’ • Flexible, open-ended, seeks to • Structured, seeks to confirm hypotheses explore phenomena about phenomena • Richness & context • Precision & focus • Develops emerging theories or • Tests existing theories or models models (grounded theory) • Smaller # of subjects • Large # of subjects • Larger amount of information • Smaller amount of information collected collected from each subject from each subject Cost? ….which methodologies are likely to cost more?

Making sense of qualitative research / 10 key questions1. Was there a clear statement of the aims of the research?2. Is a qualitative methodology appropriate?3. Was the research design appropriate to address the aims of the research?4. Was the recruitment strategy appropriate to the aims of the research?5. Were the data collected in a way that addressed the research issue?6. Has the relationship between researcher and participants been adequately considered?7. Have ethical issues been taken into consideration?8. Was the data analysis sufficiently rigorous?9. Is there a clear statement of findings?10. How valuable is the research? Many of these questions recur across checklists:• Cochrane Qualitative Research Methods Group. Chapter 6 – Critical appraisal of qualitative research. http://cqrmg.cochrane.org/supplemental-handbook-guidance• Critical Appraisal Skills Programme (CASP): making sense of evidence website: http://www.casp-uk.net/

Why does evidence fail to get into practice? …because 75% of researchers cannot understand basic statistics ….and, 70% cannot critically appraise & apply research ~ Dunn et al. Using research for practice: a UK experience of the BARRIERS scale Critical appraisal is iterative, heuristic & perhaps even social

Qualitative methods in the literature• Used when ‘context of teaching’ is needed • To identify & describe contexts • To draw out experiences, feelings, attitudes, beliefs, perceptions• Diagnostic • To explain something ‘why’ • To examine reasons for certain behaviours• Evaluative • To evaluate the effectiveness of something • To measure satisfaction, needs, barriers, improvements

Data collection, qualitative research• In-depth & unstructured interviews • Some see as the ‘gold standard’ • open-ended, respondent-led • very detailed, loose structure, • emergent issues (‘inductive’) • recorded and transcribed for analysis• Focus groups • Around 6-10 people; discuss common issues • Facilitated; non-directive questioning, observation • Range of opinions sought • Remember group dynamics, depth & validity

Other methods used in educational research• Case studies• Descriptive survey• Ethnography• Phenomenological research• Narrative review• Historical research• Randomized controlled study• Systematic review Dr. Michael Wesch,• Other types? Educator, digital ethnographer

Recruitment & sampling• Purposive sampling • Purposeful, strategic, active, systematic • Chosen for potential to obtain information to inform research• Theoretical sampling • Is guided by emerging theories• Sampling strategies • Revised & modified during research • Not a statistical sample • Does not need to represent group • Saturation point

Best Evidence in Medical EducationBEME is defined as:“…the implementation, by teachers in their practice, of methods andapproaches to education based on the best evidence available…” Harden RM, Grant J, et al BEME guide no. 1: best evidence medical education. Med Teach. 1999;21:553–562.

The six steps of BEME 1. Frame the question 2. Develop a search strategy 3. Produce the raw data 4. Evaluate the evidence 5. Implement change 6. Evaluate the change

BEME encourages teachers to:• Comprehensively & critically appraise medical literature in education• Categorize the power of that evidence• Identify the gaps & flaws in the existing literature• Suggest appropriately planned studies to optimize the evidence BEME aims to “…assist education practitioners to become increasingly skilled at using evidence to optimise student learning, shape their teaching, and in designing curricula and education policies…”

BEME appraisal checklist Comprehensively, critically appraise literature • systematic • transparent Categorizing power of evidence • realist • epistemological openness http://www2.warwick.ac.uk/fac/med/beme/ Identify gaps & flaws in existing literature • published • grey literature • hand searching Use appropriately-planned studies • optimize the evidence • education intervention more evidence-based

BEME Coding Sheet

Summary• Qualitative/quantitative/’mixed’ methods are common• Experiences, attitudes, beliefs, perceptions, behaviours interrogated…• No gold standard methodology• Ethnography, grounded theory, phenomenology most common• Purposive sampling used; not a statistically valid one• Inductive & iterative process – also true of critical appraisal• Explicit, systematic, justified & reproducible – all apply

References• BEME: Best evidence in medical education website• Bligh J, Anderson M. Medical teachers and evidence. Med Ed. 2000;34:162-163.• Cochrane Qualitative Research Methods Group. Chapter 6 – Critical appraisal of qualitative research. http://cqrmg.cochrane.org/supplemental-handbook-guidance• Cook TD. Qualitative and quantitative methods in evaluation research. Beverly Hills, CA: Sage.• Côté L. Appraising qualitative research articles in medical education. Med Teach. 2005 Jan;27(1):71-5.• Critical Appraisal Skills Programme (CASP): making sense of evidence website: http://www.casp-uk.net/• Daly J. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007 Jan;60(1):43-9.• Denzin N, Lincoln LS. Handbook of qualitative research. Sage Publications, 1994.• Eva KW. Broadening the debate about quality in medical education research. Med Educ. 2009 Apr;43(4): 294-6.• Eva KW. On the limits of systematicity. Med Educ. 2008 Sep;42(9):852-3.• Gosall N, Gosall G. The doctors guide to critical appraisal. PasTest Ltd, 2009.• Greenhalgh T. How to read a paper. BMJ books. London: Blackwell, 1995.• Guidelines for evaluating papers on educational interventions. BMJ. 1999 May 8;318(7193):1265-7.• Gunderman RB. Achieving excellence in medical education. Springer, 2008.• Haig A, Dozier M. Systematic searching for evidence in medical education. BEME Guide No. 3. Med Teach 2003;25(4):352–363 and 25(5):463–484.• Harden RM, Grant J. BEME guide no. 1: best evidence medical education. Med Teach. 1999;21:553–562.• Harden RM, Crosby JR. An introduction to outcome‐based education. Med Teach. 1999;22(1)7–14.• International Centre for Allied Health Evidence Critical Appraisal Tools http://www.unisa.edu.au/cahe/Resources/CAT/default.asp• Johnson B, Johnson L. Educational research: quantitative, qualitative and mixed approaches. Sage, 2007.

http://blogs.ubc.ca	395
http://hlwiki.slais.ubc.ca	17
https://si0.twimg.com	2
http://dashboard.bloglines.com	1
https://twitter.com	1
http://www.linkedin.com	1

Critical Appraisal of the Medical Education Literature 2012

by Dean Giustini on Jul 11, 2012

Accessibility

Categories

Upload Details

Usage Rights

6 Embeds 417

Statistics

Critical Appraisal of the Medical Education Literature 2012 Presentation Transcript