Methods conference 11_edwin_van_teijlingen

350 views

Published on

Published in: Business, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
350
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • I am going to dismiss this view, since we are argue as practical health service researcher that comparisons are possible. BUT that does not mean that we should ignore the views of some of the purists in qualitative research who are that comparing two qualitative studies is not possible (or desirable).
  • I am going to dismiss this view, since we are argue as practical health service researcher that comparisons are possible. BUT that does not mean that we should ignore the views of some of the purists in qualitative research who are that comparing two qualitative studies is not possible (or desirable).
  • Validity refers to the degree to which a study accurately reflects or assesses the specific concept that the researcher is attempting to measure. While reliability is concerned with the accuracy of the actual measuring instrument or procedure, validity is concerned with the study's success at measuring what the researchers set out to measure .
  • I am going to dismiss this view, since we are argue as practical health service researcher that comparisons are possible. BUT that does not mean that we should ignore the views of some of the purists in qualitative research who are that comparing two qualitative studies is not possible (or desirable).
  • Methods conference 11_edwin_van_teijlingen

    1. 1. Synthesising & Integrating Qualitative Research Evidence Methods @ Plymouth 2011 Friday 20 May E. van Teijlingen, A. Avenell, F. Douglas & P. Hoddinottwww.bournemouth.ac.uk
    2. 2. Road map• Introduction• Rise of qualitative synthesis;• Methodological conundrums (tensions & questions raised);• Tools use to address some of those;• Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment. www.bournemouth.ac.uk
    3. 3. Introduction• Research team• Substantive topic• HTA fundingSynthesising & Integrating Qual. Res.• A growing methodological field• Different approaches have grown organically over past two decades• Our ideas to attempt synthesis.www.bournemouth.ac.uk
    4. 4. HTA Project IResearch team:• Alison Avenell• Flora Douglas• Pat Hoddinott• Edwin van TeijlingenTopic areas:• Men’s Health• Obesity Prevention & TreatmentFunding body:• HTA Project 09/127/01www.bournemouth.ac.uk
    5. 5. HTA Project IMultidisciplinary Research TeamTeam member: Expertise:Alison Avenell Health Service Research; Nutrition/Obesity Research; Quantitative Sys. Review.;Flora Douglas Health Prom.; Men’s Health; Qualitative Research; Public Health Nutrition;Pat Hoddinott Public Health Nutrition; General Practice; Qualitative Research;Edwin van Teijlingen Sociology; Health Promotion; Systematic Review; Qualitative Research; Men’s Health www.bournemouth.ac.uk
    6. 6. Men’s Health IMen die younger than womenMen generally have less healthy lifestyles than women, more likely to take risk, etc.Men tend to seek help later in disease stage (i.e. wait longer before going to get medical care)www.bournemouth.ac.uk
    7. 7. Men’s Health IIPerhaps a gender-sensitive approach is needed to better understand mens health seeking behaviour.If existing services were more “male friendly”, i.e. convenient and anonymous they might be better used by men.(Schofield et al., 2000; Banks, 2001; Lloyd & Forrest, 2001).www.bournemouth.ac.uk
    8. 8. Weight managementObesity & associated diseases are huge challenge to NHS and UK.Lack of engagement of men in services for obesity is a serious hindrance to mitigating its effects.Obesity interventions, especially weight management interventions, often seen as ‘for women’.www.bournemouth.ac.uk
    9. 9. The funded studyOverarching objective is to integrate quantitative & qualitative evidence base for engagement and management of men with obesity in weight loss services:(1) Systematically reviewing the effectiveness and cost- effectiveness of interventions for obesity in men, and men in contrast to women(2) Systematically reviewing the effectiveness and cost- effectiveness of interventions to engage men in their weight reduction(3) Systematically reviewing qualitative research with men about obesity management, and providers of such services for men www.bournemouth.ac.uk
    10. 10. Road map• Introduction• Rise of qualitative synthesis;• Methodological conundrums (tensions & questions raised);• Tools use to address some of those;• Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment. www.bournemouth.ac.uk
    11. 11. Background I• Field of synthesising evidence from qualitative primary studies has witnessed the emergence of several different approaches, each is based on different epistemological viewpoint. • These approaches not only develop research questions, but also conduct a critique of the literature included in the review. www.bournemouth.ac.uk
    12. 12. Background IIBUT, end product of synthesis can be: • too conceptual/ complex for policy-makers, managers & practitioners;  • Too costly, some methods are time and resource intensive.Policy makers often / usually want a rapid answer, which limits the choice of analysis procedures. www.bournemouth.ac.uk
    13. 13. Background III• We reflect on approaching synthesis from ‘realist’ perspective.  We are trying to find out not only “what works” for men in terms of weight management, but also “for which men, and under what circumstances”. • We are developing an eclectic study design, drawing from several methodological approaches, employing deductive and inductive analytical approaches throughout review process.www.bournemouth.ac.uk
    14. 14. Why do we do it?When to use Qualitative Methods:•Focusing on the ‘why’ question.•Preparing for quantitative studies•In parallel with quantitative studies: • explain / examine in-depth meaning findings of quantitative studies. • as part of triangulation.www.bournemouth.ac.uk
    15. 15. Range of Methods In-depth interviews Focus groups Observation Content analysis Critical incident technique Case studieswww.bournemouth.ac.uk
    16. 16. Road map• Introduction• Rise of qualitative synthesis;• Methodological conundrums (tensions & questions raised);• Tools use to address some of those;• Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment. www.bournemouth.ac.uk
    17. 17. Can we compare? IIn order to be able to synthesise one has to agree that we actually can compare studies based on very different qualitative research methods in different populations based on different methodological approaches within qualitative research.www.bournemouth.ac.uk
    18. 18. Can we compare? IIWhich leads to next step that one has to agree that we can assess the quality of qualitative research across studies using different methods and/or approaches .www.bournemouth.ac.uk
    19. 19. Range of views1. All research perspectives are unique, each valid in its own right, i.e. no general quality criteria;2. Qualitative and quantitative methods can be assessed by the same (or very similar) criteria;3. Qualitative and quantitative methods can be assessed, but not by the same criteria (checklists for qualitative research). www.bournemouth.ac.uk
    20. 20. NO to comparing I• Some people argue that we can’t apply quantitative / Positivist ideas of systematic reviewing and meta- analysis to methods as diverse as qualitative ones.www.bournemouth.ac.uk
    21. 21. NO to comparing IISAMPLING Purposive / theoretical Criterion / quota Purposive sampling: Random • Extreme Case Snowball • Maximum etc Variation • Homogeneous • Typical Case • Critical Case • Etc. www.bournemouth.ac.uk
    22. 22. No to comparing IIICan we really mix qualitative methodologies?Ethnographers study shared experiences and their construction within a specific environment and culture, whilst Grounded Theorists uses the data more to explain (i.e. build theory around) a phenomenon.www.bournemouth.ac.uk
    23. 23. No common criteria• All research perspectives are unique and each is valid in its own terms. This view rejects the notion of establishing general quality criteria;• View of some purists.Mays N, Pope C. Assessing quality in qualitative research. BMJ 2000; 320:50-52.www.bournemouth.ac.uk
    24. 24. Same or similar criteria• Qualitative and quantitative methods can be assessed by the same (or very similar) criteria, such as: • acceptability, cost, validity, reliability, generalisability (or relevance) and objectivity. (Ryan et al. 2001)www.bournemouth.ac.uk
    25. 25. ChecklistsPredefined criteria in assessing quality ofqualitative studies: ‘checklists’Blaxter, M. Criteria for qualitative research. Med. Sociol. News 2000; 26: 34-7.Mays & Pope. Rigour and qualitative research. BMJ 1995; 311:109-112.Mays & Pope. Assessing quality in qualitative research. BMJ 2000; 320:50-2.Critical Appraisal Skills Programme (CASP) 10 questions to help you makesense of qualitative research,www.sph.nhs.uk/sph-files/casp-appraisal-tools/Qualitative%20Appraisal%20Tool.pdfMalterud K. Qualitative research: standards, challenges & guidelines, TheLancet 2001 358: 483-88 www.bournemouth.ac.uk
    26. 26. Typical checklist I• Worth or relevance of research?• Clarity of research question: able to set aside preconceptions?• Appropriateness of research design to question?• Is the context or setting adequately described?• Sampling: full range possible cases/settings so that conceptual rather than statistical generalisations could be made? Cont. www.bournemouth.ac.uk
    27. 27. Typical checklist II• Analysis: analysis incorporating all observations? Could it explain key processes or respondents’ accounts or observations? Search for disconfirming cases?• Reflexivity of account: Self-conscious assessment impact of methods used on data obtained? Are sufficient data included in report to assess whether analytical criteria had been met? (Mays & Pope, BMJ 2000: 50-52) www.bournemouth.ac.uk
    28. 28. Case for checklists•Checklist give outsiders a chance to evaluate thequality of our work.•At the same time it is important to stress that there amany different ways to conduct qualitative research. www.bournemouth.ac.uk
    29. 29. Against checklists I•Are qualitative methods really different in general outline?•Are there no common criteria for a good piece of researchusing either qualitative or quantitative methods?•There are different ways to conduct quantitative research: • (e.g. parametric versus non-parametric statistics). www.bournemouth.ac.uk
    30. 30. Against checklists II• Checklists focus on how research is conducted rather than on the quality of the method itself.• There is a risk that criteria become ‘bureaucratic’ or ‘prescriptive’. Whilst flexibility is exactly the strength of qualitative methods! Barbour (2001) www.bournemouth.ac.uk
    31. 31. Not same criteria• Qualitative and quantitative methods can be assessed, but not by the same criteria.• E.g. Lincoln & Guba (1985) suggested four criteria specifically for the evaluation of qualitative methods: • credibility, transferability, consistency (or dependability) and confirmability. www.bournemouth.ac.uk
    32. 32. Case for common criteria I• Main quality criteria: (1) acceptability; (2) validity; (3) reliability; or (4) generalisability (or relevance); (5) objectivity.• Criteria apply to the valuation of quantitative and qualitative techniques, perhaps some criteria require minor modification for qualitative research. www.bournemouth.ac.uk
    33. 33. Common criteria: example ‘validity’(2) Validity• Triangulation• Respondent validation• Reflexivity (relevant to validity & reliability)• Data analysis: •Computer packages •Grounded theory •Analytical induction (e.g. identify ‘deviant’ cases) www.bournemouth.ac.uk
    34. 34. Common criteria: example ‘objectivity’Methods employed to maximise objectivity :•Reflexivity: a process where researcherscontinually reflect upon how their own interestsand potential biases could alter theinterpretation of the results;•intersubjectivity (interrater reliability). www.bournemouth.ac.uk
    35. 35. Grading studiesA: No, or few flaws. The study credibility, transferability, dependability and confirmability is high.B: Some flaws, unlikely to affect the credibility, transferability, dependability and/or confirmability of the study.C: Some flaws that may affect the credibility, transferability, dependability and/or confirmability of the study.D: Significant flaws that are very likely to affect the credibility, transferability, dependability and/or confirmability of the study. (Downe et al. 2007) www.bournemouth.ac.uk
    36. 36. SynthesizingTable 1 Process of meta-synthesisSteps in process of doing a qualitative meta-synthesis:1. aim of study: establish appropriate research question2. set inclusion and exclusion criteria methodologies3. identify and locate relevant papers/publications4. appraise studies located through process above.5. synthesize findings reported in papers above. www.bournemouth.ac.uk
    37. 37. 1. Research question• Some of earlier meta-synthesis have been criticised for being too broad, e.g. synthesis by Sherwood (1997) on ‘clients perspective of caring’• Walsh and Downe (2005) concluded that the debate of a broader more inclusive stance versus a narrower more precise stance is still ongoing, although they seem to favour a narrower focus. www.bournemouth.ac.uk
    38. 38. 2 Inclusion/ exclusion criteria• Include all qualitative research, all methods and all approaches.• Include only selected methods• Include only selected approaches• Etc.www.bournemouth.ac.uk
    39. 39. 3 Identifying and locating studiesQualitative researchers are less likely to publish in journals. Most electronic databases comprise mainly journal articles, not reports and book chapters which often accepts longer manuscripts needed for reporting qualitative research in detail.Hence need to search outwith the standard electronic databases, and check references of articles found in the initial trawl.Methods sections in papers are often too short for authors to give detailed information on how the study was exactly conducted, this creates the need to contact the original authors for further details.www.bournemouth.ac.uk
    40. 40. 4 Appraising qualitative literature• Arguably the most crucial step in meta-analysis, but most uncertain.• Some meta-synthesists refer to existing appraisal methods (‘tools’), such as CASP (as mentioned before) or the UK Cabinet Office quality framework (Spencer et al. 2003).www.bournemouth.ac.uk
    41. 41. 5 Approaches to meta analysis• Meta narrative• Meta synthesis• Thematic synthesis• Grounded Theory• Critical interpretive synthesis• Meta ethnography• Ecological Triangulation• Qualitative Metasummary• Content Analysis• Framework synthesis• Etc.(review of range of methods by Barnett-Page & Thomas 2009)www.bournemouth.ac.uk
    42. 42. 5a Meta-narrativeNarrative inductive method of analysisGreenhalgh et al. (2005) on diffusion of innovation in health services.Bring together research which contains many different theories from many different disciplines & study designs.“In the best narrative work, descriptive/ interpretative analysis is a story about stories.” (Jones 2004: 107)www.bournemouth.ac.uk
    43. 43. 5b Thematic synthesisAdopts approaches from meta- ethnography and grounded theory. Method developed to assist reviews on interventions.Thomas & Harden (2008) advocated thematic analysis in three stages: (a) coding ‘line by line’ of included studies; (b) develop ‘descriptive’ themes and (c) generate analytical themes.www.bournemouth.ac.uk
    44. 44. 5c Criticalinterpretive synthesisAdopts approaches from meta- ethnography and grounded theory. Critical interpretive synthesis is approach to doing the whole reviewing process not just the synthesis part of it (Dixon Woods et al. 2007). E.g. iterative approach redefining research question and searching and selecting from literature.www.bournemouth.ac.uk
    45. 45. 5d Meta ethnographyA comparative textural analysis using three ways to order them:• in terms of one another;• set against one another;• tied to one another.Meta-ethnography aims to recover the social and theoretical context in which substantive findings emerge (Noblit & Hare, 1988, p. 5-6). It is driven by interpretation, not analysis.www.bournemouth.ac.uk
    46. 46. But complex & expensiveBarnett-Page and Thomas (2009) argued that products of some qualitative synthesis reviews can be complex and conceptual and (in their opinion), tend to require further interpretation by policy makers and practitioners.• Some synthesis methods can be very time and resource intensivewww.bournemouth.ac.uk
    47. 47. Road map• Introduction• Rise of qualitative synthesis;• Methodological conundrums (tensions & questions raised);• Tools use to address some of those;• Our thinking around qualitative synthesis for a HTA project on men’s health and obesity prevention & treatment. www.bournemouth.ac.uk
    48. 48. Realist Approach IRealist evaluation has as its underlying principles of what works best, for whom, and under what circumstances (Kazi 2003:1).Realistic or Realist evaluation has particular strengths in assessing interventions delivered in context; (Pawson & Tilley 1997). www.bournemouth.ac.uk
    49. 49. Realist Approach IIRealist evaluation is particularly useful “in the evaluation of practice in an open system that consists of a constellation of interconnected structures, mechanisms and contexts” (Kazi 2003:158). www.bournemouth.ac.uk
    50. 50. Realist Approach III• We aim to uncover how effective interventions work (if they exist), and describe a recipe for the intervention ingredients, processes, environmental and contextual factors that contribute to effectiveness (Pawson 1997).www.bournemouth.ac.uk
    51. 51. Realist Approach IV• A ‘realist’ perspective (Pawson, 2006; Spencer et al., 2003), as we are concerned with trying to find out not only “what works” for men in terms of weight management, but also “for which men, and under what circumstances”.• Also we as researchers span a range of clinical, sociological and public health disciplines, which represents a broad range of perspectives.www.bournemouth.ac.uk
    52. 52. Realist Approach VWe shall try an eclectic study design, drawing from range of methodological literature and employ deductive and inductive analytical approaches throughout proposed review process.We will define a priori research questions, but also generate new research questions inductively from the data to build theory (about what works, for whom, etc.) using for example, constant comparative method, and searching for disconfirming evidence to falsify any emerging theory or theories.www.bournemouth.ac.uk
    53. 53. The Plan: Step 1Step 1: A priori research questionsWe aim to uncover how effective interventions work (if they exist), and describe intervention ingredients, processes, environmental and contextual factors that contribute to effectiveness (Pawson, 2006). To identify barriers/ facilitators men experience in weight management interventions. Specific research questions are:• What are the best evidence-based management strategies for treating obesity in men?• How can mens engagement in obesity services be improved? www.bournemouth.ac.uk
    54. 54. Steps 2 & 3Step 2: Literature searchUse multiple strategies to identify qualitative research (Shaw et al. (2004; Grant 2004).Step 3: Apply inclusion /exclusion criteria to study abstractsStudies include males, must have overweight and weight management as the prime focus of study. Studies in developed countries only, which are relevant to the UK.www.bournemouth.ac.uk
    55. 55. Step 4Purposive sampling to group studies into three categoriesA. Qualitative & mixed method studies linked to RCTs, which include any qualitative data as part of reporting quantitative outcomes, as recommended in critical interpretive synthesis (Dixon Woods et al., 2007)B. Qualitative & mixed method studies linked to non-randomised intervention studies including any qualitative data reported as part quantitative outcomes.C.Qualitative studies that are not linked to any specific intervention.www.bournemouth.ac.uk
    56. 56. Step 4 (cont.)An iterative, sequential analysis will then proceed. During qualitative stage A, qualitative data from (effective & non-effective) RCTs will be analysed using constant comparative method to illicit aspects of intervention processes and contexts that contributed to trial outcomes.Using theoretical sampling, category B studies (non- randomised intervention studies) will be indentified and analysed in the same way. During this stage emergent themes and patterns between the two groups will be compared and contrasted, and emergent theory modified accordingly.In qualitative stage C, non-intervention studies will be theoretically sampled and analysed to further refine & validate our emerging analysis. www.bournemouth.ac.uk
    57. 57. Step 5Quality assessment of included studiesApply Critical Appraisal Skills Programme (CASP) quality appraisal criteria.We will assess each article identified, but which would be excluded on purely quality grounds, on the basis of its potential conceptual contribution to review (‘sensitivity analysis’).  www.bournemouth.ac.uk
    58. 58. Step 6Final thematic synthesis step is proposed as a staged (but overlapping) process which involves: coding data, developing initial descriptive themes, with a final stage that involves the development of higher order analytical themes and concepts.This output will identify promising ‘ingredients’ of interventions more likely to be effective in male weight reduction, both in terms of essential and necessary contextual/environmental variables, and intervention processes.www.bournemouth.ac.uk
    59. 59. Men’s involvementAdvisory group from Men’s Health Forum to:• ensure review is conducted in systematic and transparent manner;• check for over-interpretation of data;• provide expert opinion on credibility and plausibility of thematic index and other emergent issues; and potentially, be able to provide sources for disconfirming data.Will report our findings as a narrative synthesis combining and juxtaposing quantitative and qualitative findings.www.bournemouth.ac.uk
    60. 60. • Figure 1 - Graphical representation of the proposed review processwww.bournemouth.ac.uk
    61. 61. References IBanks, I. (2001) No man’s land: Men illness and the NHS, BMJ; 323: 1058-1060.Barbour R. (2001) Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ 322:1115-17Barnett-Page, E., Thomas, J. (2009) Methods for the synthesis of qualitative research: a critical review BMC Med Res Meth, 9 (59) (www.biomedcentral.com/content/pdf/1471-2288-9-59.pdf)Campbell, R., et al. (2003) Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med 56 (4), 671-684. Critical Appraisal Skills Programme : www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf Dixon-Woods, M., et al. (2007) Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods. J Health Serv Res Pol 12: 42–47.Downe S,. et al. (2007) Expert intrapartum maternity care: a metasynthesis. J Adv Nurs 57: 127-40.Downe S. (2008) Metasynthesis: a guide to knitting smoke. Evidence Based Midwifery 6(1): 4-8 Egger G, et al. (2001) Systematic reviews in health care: Meta-analysis in context . London: BMJ PublEPPI-Centre (2007) EPPI-Centre methods for conducting systematic reviews . London: Inst. of Educ.Grant, MJ. (2004) How does your searching grow? A survey of search preferences and use of optimal search strategies in identification of qualitative research. Health Info Libraries J 21: 21-32. www.bournemouth.ac.uk
    62. 62. References IIGreenhalgh T, et al. (2005) Storylines of research in diffusion of innovation: a metanarrative approach to systematic review. Soc Sci Med 2005, 61:417-30.Jones, K., (2004) Mission drift in qualitative research, or moving toward a systematic review of qualitative studies, moving back to a more systematic narrative review. The Qualitative Report, 9 (1): 95-112.Kazi, M.A.F. (2003) Realist Evaluation in Practice: Health and Social Work . London: Sage.Lloyd, T, Forrest, S. (2001) Boys’ and Young Men’s Health. A Literature & Practice Review: An Interim Report. London, UK: Health Development Agency.Lincoln & Guba. Naturalistic enquiry, Sage, Beverley Hills, CA, 1985.Noblit GW., Hare, RD. (1988) Meta-ethnography: synthesizing qualitative studies . London: Sage.Pawson, R, Tilley, N. (1997) Realistic Evaluation. London: Sage.Pawson, R. (2006) Evidence-based policy: A realist perspective. London: Sage.Pope, K, et al. (2007) Synthesising qualitative and quantitative health evidence. Open Univ. Press.Rose, H.D., Cohen, K. (2010) The experiences of young carers: a meta-synthesis of qualitative findings, J Youth Stud 13 (4): 473- 487.Ryan M, et al. (2001) Eliciting public preferences for healthcare: a systematic review of techniques. Health Technol Assess, 5(5). http://www.hta.nhsweb.nhs.uk/fullmono/mon505.pdfRobertson LM, et al. (2008) BMC Health Serv Res; 8: 141. www.bournemouth.ac.uk
    63. 63. References IIISchofield T. et al. (2000). Understanding mens health and illness: a gender-relations approach to policy, research, and practice. J Am Coll Health. 48(6): 247-256.Shaw, RL et al. (2004) Finding qualitative research: evaluation of search strategies. BMC Med Res Meth 5; 4  Sherwood, G. (1997) Meta-synthesis: merging qualitative studies to develop nursing knowledge. IntJ Human Caring 3 (1): 37-42. Spencer, L. et al. (2003) Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence. Centre for Social Research. London: National Centre for Social Research/UK Cabinet Office. Strauss A., Corbin, J. (1994) Grounded theory methodology: An overview. In Handbook of Qualitative Research. (Eds) Denzin, N.K. & Lincoln, Y.S. Sage: Thousand Oaks.Thomas, J., Harden, A. (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Meth 8(45) http://www.biomedcentral.com/1471-2288/8/45.Walsh D, Downe, S (2005) Meta-synthesis method for qualitative research: a lit. rev J Advanc Nurs 50: 204-11. www.bournemouth.ac.uk
    64. 64. Thank you! Prof. Edwin van Teijlingen Centre for Midwifery, Maternal & Perinatal Health Research, Bournemouth University vanteijlingen@bournemouth.ac.ukwww.bournemouth.ac.uk

    ×