Methods conference 11_edwin_van_teijlingen

Synthesising &
Integrating Qualitative
Research Evidence

Methods @ Plymouth 2011
Friday 20 May
E. van Teijlingen, A. Avenell, F. Douglas & P. Hoddinott

www.bournemouth.ac.uk

Road map
• Introduction
• Rise of qualitative synthesis;
• Methodological conundrums (tensions &
questions raised);
• Tools use to address some of those;
• Our thinking around qualitative synthesis
for a HTA project on men’s health and
obesity prevention & treatment.


Introduction

• Research team
• Substantive topic
• HTA funding

Synthesising & Integrating Qual.
Res.
• A growing methodological field
• Different approaches have grown
organically over past two decades
• Our ideas to attempt synthesis.

HTA Project I

Research team:
• Alison Avenell
• Flora Douglas
• Pat Hoddinott
• Edwin van Teijlingen
Topic areas:
• Men’s Health
• Obesity Prevention & Treatment
Funding body:
• HTA Project 09/127/01

HTA Project I

Multidisciplinary Research
Team
Team member: Expertise:
Alison Avenell Health Service Research;
Nutrition/Obesity Research;
Quantitative Sys. Review.;
Flora Douglas Health Prom.; Men’s Health;
Qualitative Research; Public Health
Nutrition;
Pat Hoddinott Public Health Nutrition; General
Practice; Qualitative Research;
Edwin van Teijlingen Sociology; Health Promotion;
Systematic Review; Qualitative
Research; Men’s Health

Men’s Health I

Men die younger than women
Men generally have less healthy
lifestyles than women, more likely
to take risk, etc.
Men tend to seek help later in
disease stage (i.e. wait longer
before going to get medical care)


Men’s Health II

Perhaps a gender-sensitive
approach is needed to better
understand men's health seeking
behaviour.
If existing services were more “male
friendly”, i.e. convenient and
anonymous they might be better
used by men.
(Schofield et al., 2000; Banks, 2001; Lloyd & Forrest, 2001).


Weight
management
Obesity & associated diseases are
huge challenge to NHS and UK.
Lack of engagement of men in
services for obesity is a serious
hindrance to mitigating its effects.
Obesity interventions, especially
weight management interventions,
often seen as ‘for women’.


The funded study

Overarching objective is to integrate quantitative &
qualitative evidence base for engagement and
management of men with obesity in weight loss
services:
(1) Systematically reviewing the effectiveness and cost-
effectiveness of interventions for obesity in men, and
men in contrast to women
(2) Systematically reviewing the effectiveness and cost-
effectiveness of interventions to engage men in their
weight reduction
(3) Systematically reviewing
qualitative research with men about
obesity management, and providers
of such services for men

Background I

• Field of synthesising evidence from
qualitative primary studies has witnessed
the emergence of several different
approaches, each is based on different
epistemological viewpoint.
• These approaches not only develop
research questions, but also conduct a
critique of the literature included in the
review.


Background II

BUT, end product of synthesis can be:
• too conceptual/ complex for policy-makers,
managers & practitioners;
• Too costly, some methods are time and
resource intensive.

Policy makers often / usually want a rapid
answer, which limits the choice of
analysis procedures.

Background III

• We reflect on approaching synthesis
from ‘realist’ perspective. We are trying
to find out not only “what works” for
men in terms of weight management,
but also “for which men, and under
what circumstances”.
• We are developing an eclectic study
design, drawing from several
methodological approaches, employing
deductive and inductive analytical
approaches throughout review process.

Why do we do it?

When to use Qualitative Methods:
•Focusing on the ‘why’ question.
•Preparing for quantitative studies
•In parallel with quantitative studies:
• explain / examine in-depth meaning
findings of quantitative studies.
• as part of triangulation.


Range of Methods

 In-depth interviews
 Focus groups
 Observation
 Content analysis
 Critical incident technique
 Case studies


Road map
• Introduction
questions raised);
• Tools use to address some of
those;
• Our thinking around qualitative synthesis
for a HTA project on men’s health and
obesity prevention & treatment.


Can we compare? I

In order to be able to synthesise one
has to agree that we actually can
compare studies based on very
different qualitative research
methods in different populations
based on different methodological
approaches within qualitative
research.


Can we compare? II

Which leads to next step that one
has to agree that we can assess
the quality of qualitative research
across studies using different
methods and/or approaches .


Range of views

1. All research perspectives are unique, each valid
in its own right, i.e. no general quality criteria;
2. Qualitative and quantitative methods can be
assessed by the same (or very similar) criteria;
3. Qualitative and quantitative methods can be
assessed, but not by the same criteria
(checklists for qualitative research).


NO to comparing I

• Some people argue that we can’t
apply quantitative / Positivist ideas
of systematic reviewing and meta-
analysis to methods as diverse as
qualitative ones.


NO to comparing II

SAMPLING
 Purposive / theoretical
 Criterion / quota
Purposive sampling:
 Random
• Extreme Case
 Snowball
• Maximum
 etc Variation
• Homogeneous
• Typical Case
• Critical Case
• Etc.


No to comparing III

Can we really mix qualitative
methodologies?
Ethnographers study shared experiences
and their construction within a specific
environment and culture, whilst
Grounded Theorists uses the data
more to explain (i.e. build theory
around) a phenomenon.


No common criteria

• All research perspectives are unique
and each is valid in its own terms. This
view rejects the notion of establishing
general quality criteria;
• View of some purists.

Mays N, Pope C. Assessing quality in qualitative research. BMJ 2000;
320:50-52.


Same or similar
criteria

• Qualitative and quantitative methods
can be assessed by the same (or very
similar) criteria, such as:
• acceptability, cost, validity, reliability,
generalisability (or relevance) and
objectivity.
(Ryan et al. 2001)

Checklists

Predefined criteria in assessing quality of
qualitative studies: ‘checklists’
Blaxter, M. Criteria for qualitative research. Med. Sociol. News 2000; 26: 34-7.
Mays & Pope. Rigour and qualitative research. BMJ 1995; 311:109-112.
Mays & Pope. Assessing quality in qualitative research. BMJ 2000; 320:50-2.
Critical Appraisal Skills Programme (CASP) 10 questions to help you make
sense of qualitative research,
www.sph.nhs.uk/sph-files/casp-appraisal-tools/Qualitative%20Appraisal%20Tool.pdf

Malterud K. Qualitative research: standards, challenges & guidelines, The
Lancet 2001 358: 483-88


Typical checklist I

• Worth or relevance of research?
• Clarity of research question: able to set aside
preconceptions?
• Appropriateness of research design to question?
• Is the context or setting adequately described?
• Sampling: full range possible cases/settings so that
conceptual rather than statistical generalisations could
be made? Cont.


Typical checklist II

• Analysis: analysis incorporating all observations? Could it
explain key processes or respondents’ accounts or
observations? Search for disconfirming cases?
• Reflexivity of account: Self-conscious assessment impact
of methods used on data obtained? Are sufficient data
included in report to assess whether analytical criteria
had been met? (Mays & Pope, BMJ 2000: 50-52)


Case for checklists

•Checklist give outsiders a chance to evaluate the

quality of our work.

•At the same time it is important to stress that there a

many different ways to conduct qualitative research.


Against checklists I

•Are qualitative methods really different in general outline?

•Are there no common criteria for a good piece of research

using either qualitative or quantitative methods?

•There are different ways to conduct quantitative research:

• (e.g. parametric versus non-parametric statistics).


Against checklists II

• Checklists focus on how research is conducted

rather than on the quality of the method itself.

• There is a risk that criteria become ‘bureaucratic’

or ‘prescriptive’. Whilst flexibility is exactly the

strength of qualitative methods!

Barbour (2001)


Not same criteria

• Qualitative and quantitative methods can
be assessed, but not by the same criteria.

• E.g. Lincoln & Guba (1985) suggested
four criteria specifically for the evaluation
of qualitative methods:
• credibility, transferability, consistency
(or dependability) and confirmability.


Case for common
criteria I

• Main quality criteria: (1) acceptability; (2)
validity; (3) reliability; or (4) generalisability
(or relevance); (5) objectivity.

• Criteria apply to the valuation of quantitative and qualitative
techniques, perhaps some criteria require minor
modification for qualitative research.


Common criteria:
example ‘validity’

(2) Validity
• Triangulation
• Respondent validation
• Reflexivity (relevant to validity & reliability)
• Data analysis:
•Computer packages
•Grounded theory
•Analytical induction (e.g. identify ‘deviant’ cases)


Common criteria:
example ‘objectivity’

Methods employed to maximise objectivity :

•Reflexivity: a process where researchers
continually reflect upon how their own interests
and potential biases could alter the
interpretation of the results;

•intersubjectivity (interrater reliability).

Grading studies

A: No, or few flaws. The study credibility, transferability,
dependability and confirmability is high.
B: Some flaws, unlikely to affect the credibility,
transferability, dependability and/or confirmability of
the study.
C: Some flaws that may affect the credibility,
transferability, dependability and/or confirmability of
the study.
D: Significant flaws that are very likely to affect the
credibility, transferability, dependability and/or
confirmability of the study.
(Downe et al. 2007)

Synthesizing

Table 1 Process of meta-synthesis

Steps in process of doing a qualitative meta-synthesis:
1. aim of study: establish appropriate research
question
2. set inclusion and exclusion criteria methodologies
3. identify and locate relevant papers/publications
4. appraise studies located through process above.
5. synthesize findings reported in papers above.


1. Research question

• Some of earlier meta-synthesis have been
criticised for being too broad, e.g. synthesis
by Sherwood (1997) on ‘clients perspective
of caring’
• Walsh and Downe (2005) concluded that
the debate of a broader more inclusive
stance versus a narrower more precise
stance is still ongoing, although they seem
to favour a narrower focus.


2 Inclusion/
exclusion criteria
• Include all qualitative research, all
methods and all approaches.
• Include only selected methods
• Include only selected approaches
• Etc.


3 Identifying and
locating studies
Qualitative researchers are less likely to
publish in journals. Most electronic
databases comprise mainly journal articles,
not reports and book chapters which often
accepts longer manuscripts needed for
reporting qualitative research in detail.
Hence need to search outwith the standard
electronic databases, and check references
of articles found in the initial trawl.
Methods sections in papers are often too short
for authors to give detailed information on
how the study was exactly conducted, this
creates the need to contact the original
authors for further details.

4 Appraising
qualitative literature
• Arguably the most crucial step in
meta-analysis, but most uncertain.

• Some meta-synthesists refer to
existing appraisal methods
(‘tools’), such as CASP (as
mentioned before) or the UK
Cabinet Office quality framework
(Spencer et al. 2003).

5 Approaches to
meta analysis
• Meta narrative
• Meta synthesis
• Thematic synthesis
• Grounded Theory
• Critical interpretive synthesis
• Meta ethnography
• Ecological Triangulation
• Qualitative Metasummary
• Content Analysis
• Framework synthesis
• Etc.
(review of range of methods by Barnett-Page & Thomas 2009)


5a Meta-narrative

Narrative inductive method of analysis
Greenhalgh et al. (2005) on diffusion of
innovation in health services.
Bring together research which contains
many different theories from many
different disciplines & study designs.
“In the best narrative work, descriptive/
interpretative analysis is a story about
stories.” (Jones 2004: 107)


5b Thematic
synthesis
Adopts approaches from meta-
ethnography and grounded theory.
Method developed to assist reviews
on interventions.
Thomas & Harden (2008) advocated
thematic analysis in three stages: (a)
coding ‘line by line’ of included
studies; (b) develop ‘descriptive’
themes and (c) generate analytical
themes.

5c Critical
interpretive synthesis
Adopts approaches from meta-
ethnography and grounded theory.
Critical interpretive synthesis is
approach to doing the whole reviewing
process not just the synthesis part of it
(Dixon Woods et al. 2007). E.g.
iterative approach redefining research
question and searching and selecting
from literature.


5d Meta
ethnography
A comparative textural analysis using
three ways to order them:
• in terms of one another;
• set against one another;
• tied to one another.
Meta-ethnography aims to recover the
social and theoretical context in
which substantive findings emerge
(Noblit & Hare, 1988, p. 5-6). It is
driven by interpretation, not analysis.

But complex &
expensive
Barnett-Page and Thomas (2009)
argued that products of some
qualitative synthesis reviews can
be complex and conceptual and
(in their opinion), tend to require
further interpretation by policy
makers and practitioners.
• Some synthesis methods can be
very time and resource intensive

Road map
• Introduction
questions raised);
• Tools use to address some of those;
• Our thinking around
qualitative synthesis for a HTA
project on men’s health and obesity
prevention & treatment.

Realist Approach I

Realist evaluation has as its underlying
principles of what works best, for
whom, and under what
circumstances (Kazi 2003:1).
Realistic or Realist evaluation has
particular strengths in assessing
interventions delivered in context;
(Pawson & Tilley 1997).


Realist Approach II

Realist evaluation is particularly useful
“in the evaluation of practice in an
open system that consists of a
constellation of interconnected
structures, mechanisms and
contexts”
(Kazi 2003:158).


Realist Approach III

• We aim to uncover how effective
interventions work (if they exist),
and describe a recipe for the
intervention ingredients,
processes, environmental and
contextual factors that contribute
to effectiveness (Pawson 1997).


Realist Approach IV

• A ‘realist’ perspective (Pawson,
2006; Spencer et al., 2003), as we
are concerned with trying to find out
not only “what works” for men in
terms of weight management, but
also “for which men, and under what
circumstances”.
• Also we as researchers span a
range of clinical, sociological and
public health disciplines, which
represents a broad range of
perspectives.

Realist Approach V

We shall try an eclectic study design,
drawing from range of methodological
literature and employ deductive and
inductive analytical approaches
throughout proposed review process.
We will define a priori research questions,
but also generate new research
questions inductively from the data to
build theory (about what works, for
whom, etc.) using for example, constant
comparative method, and searching for
disconfirming evidence to falsify any
emerging theory or theories.

The Plan: Step 1

Step 1: A priori research questions
We aim to uncover how effective interventions work
(if they exist), and describe intervention
ingredients, processes, environmental and
contextual factors that contribute to effectiveness
(Pawson, 2006). To identify barriers/ facilitators
men experience in weight management
interventions. Specific research questions are:
• What are the best evidence-based
management strategies for treating
obesity in men?
• How can men's engagement in obesity
services be improved?

Steps 2 & 3

Step 2: Literature search
Use multiple strategies to identify
qualitative research (Shaw et al.
(2004; Grant 2004).
Step 3: Apply inclusion
/exclusion criteria to study
abstracts
Studies include males, must have
overweight and weight management
as the prime focus of study. Studies
in developed countries only, which
are relevant to the UK.

Step 4

Purposive sampling to group studies
into three categories
A. Qualitative & mixed method studies linked to
RCTs, which include any qualitative data as
part of reporting quantitative outcomes, as
recommended in critical interpretive
synthesis (Dixon Woods et al., 2007)
B. Qualitative & mixed method studies linked to
non-randomised intervention studies
including any qualitative data reported as
part quantitative outcomes.
C.Qualitative studies that are not linked to any
specific intervention.


Step 4 (cont.)
An iterative, sequential analysis will then proceed.
During qualitative stage A, qualitative data from
(effective & non-effective) RCTs will be analysed
using constant comparative method to illicit
aspects of intervention processes and contexts
that contributed to trial outcomes.
Using theoretical sampling, category B studies (non-
randomised intervention studies) will be indentified
and analysed in the same way. During this stage
emergent themes and patterns between the two
groups will be compared and contrasted, and
emergent theory modified accordingly.
In qualitative stage C, non-intervention studies will
be theoretically sampled and analysed to further
refine & validate our emerging analysis.

Step 5

Quality assessment of included
studies
Apply Critical Appraisal Skills Programme
(CASP) quality appraisal criteria.
We will assess each article identified, but
which would be excluded on purely
quality grounds, on the basis of its
potential conceptual contribution to
review (‘sensitivity analysis’).


Step 6

Final thematic synthesis step is proposed
as a staged (but overlapping) process
which involves: coding data, developing
initial descriptive themes, with a final
stage that involves the development of
higher order analytical themes and
concepts.
This output will identify promising
‘ingredients’ of interventions more likely
to be effective in male weight reduction,
both in terms of essential and necessary
contextual/environmental variables, and
intervention processes.

Men’s involvement

Advisory group from Men’s Health Forum to:
• ensure review is conducted in systematic
and transparent manner;
• check for over-interpretation of data;
• provide expert opinion on credibility and
plausibility of thematic index and other
emergent issues; and potentially, be able to
provide sources for disconfirming data.

Will report our findings as a narrative synthesis
combining and juxtaposing quantitative and
qualitative findings.


• Figure 1 - Graphical
representation of the
proposed review process


References I
Banks, I. (2001) No man’s land: Men illness and the NHS, BMJ; 323: 1058-1060.

Barbour R. (2001) Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ
322:1115-17

Barnett-Page, E., Thomas, J. (2009) Methods for the synthesis of qualitative research: a critical review BMC Med
Res Meth, 9 (59) (www.biomedcentral.com/content/pdf/1471-2288-9-59.pdf)

Campbell, R., et al. (2003) Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences
of diabetes and diabetes care. Soc Sci Med 56 (4), 671-684.

Critical Appraisal Skills Programme : www.phru.nhs.uk/Doc_Links/Qualitative%20Appraisal%20Tool.pdf

Dixon-Woods, M., et al. (2007) Appraising qualitative research for inclusion in systematic reviews: a quantitative
and qualitative comparison of three methods. J Health Serv Res Pol 12: 42–47.

Downe S,. et al. (2007) Expert intrapartum maternity care: a metasynthesis. J Adv Nurs 57: 127-40.

Downe S. (2008) Metasynthesis: a guide to knitting smoke. Evidence Based Midwifery 6(1): 4-8

Egger G, et al. (2001) Systematic reviews in health care: Meta-analysis in context . London: BMJ Publ

EPPI-Centre (2007) EPPI-Centre methods for conducting systematic reviews . London: Inst. of Educ.

Grant, MJ. (2004) How does your searching grow? A survey of search preferences and use of optimal search
strategies in identification of qualitative research. Health Info Libraries J 21: 21-32.

References II

Greenhalgh T, et al. (2005) Storylines of research in diffusion of innovation: a metanarrative
approach to systematic review. Soc Sci Med 2005, 61:417-30.
Jones, K., (2004) Mission drift in qualitative research, or moving toward a systematic review of
qualitative studies, moving back to a more systematic narrative review. The Qualitative
Report, 9 (1): 95-112.
Kazi, M.A.F. (2003) Realist Evaluation in Practice: Health and Social Work . London: Sage.
Lloyd, T, Forrest, S. (2001) Boys’ and Young Men’s Health. A Literature & Practice Review: An
Interim Report. London, UK: Health Development Agency.
Lincoln & Guba. Naturalistic enquiry, Sage, Beverley Hills, CA, 1985.
Noblit GW., Hare, RD. (1988) Meta-ethnography: synthesizing qualitative studies . London: Sage.
Pawson, R, Tilley, N. (1997) Realistic Evaluation. London: Sage.
Pawson, R. (2006) Evidence-based policy: A realist perspective. London: Sage.
Pope, K, et al. (2007) Synthesising qualitative and quantitative health evidence. Open Univ.
Press.
Rose, H.D., Cohen, K. (2010) The experiences of young carers: a meta-synthesis of qualitative
findings, J Youth Stud 13 (4): 473- 487.
Ryan M, et al. (2001) Eliciting public preferences for healthcare: a systematic review of
techniques. Health Technol Assess, 5(5).
http://www.hta.nhsweb.nhs.uk/fullmono/mon505.pdf
Robertson LM, et al. (2008) BMC Health Serv Res; 8: 141.

References III

Schofield T. et al. (2000). Understanding men's health and illness: a gender-relations approach
to policy, research, and practice. J Am Coll Health. 48(6): 247-256.
Shaw, RL et al. (2004) Finding qualitative research: evaluation of search strategies. BMC Med
Res Meth 5; 4
Sherwood, G. (1997) Meta-synthesis: merging qualitative studies to develop nursing knowledge.
IntJ Human Caring 3 (1): 37-42.
Spencer, L. et al. (2003) Quality in Qualitative Evaluation: A Framework for Assessing
Research Evidence. Centre for Social Research. London: National Centre for Social
Research/UK Cabinet Office.
Strauss A., Corbin, J. (1994) Grounded theory methodology: An overview. In Handbook of
Qualitative Research. (Eds) Denzin, N.K. & Lincoln, Y.S. Sage: Thousand Oaks.
Thomas, J., Harden, A. (2008) Methods for the thematic synthesis of qualitative research in
systematic reviews. BMC Med Res Meth 8(45)
http://www.biomedcentral.com/1471-2288/8/45.
Walsh D, Downe, S (2005) Meta-synthesis method for qualitative research: a lit. rev J Advanc
Nurs 50: 204-11.


Thank you!
Prof. Edwin van Teijlingen
Centre for Midwifery, Maternal & Perinatal Health Research,
Bournemouth University

vanteijlingen@bournemouth.ac.uk


Methods conference 11_edwin_van_teijlingen

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Methods conference 11_edwin_van_teijlingen

Similar to Methods conference 11_edwin_van_teijlingen (20)

Recently uploaded

Recently uploaded (20)

Methods conference 11_edwin_van_teijlingen

Editor's Notes