This document provides an overview of critical appraisal of qualitative research. It discusses what characterizes qualitative research, including that it aims to provide an in-depth understanding of human experiences through inductive and contextual methods. Key aspects of rigor in qualitative research are described, such as using triangulation, reflexivity, multiple coding, and respondent validation. Checklists for critically appraising qualitative research focus on questions like whether the sample and data collection were appropriate, if data analysis was rigorous, and if ethical issues were adequately addressed. The importance of assessing relevance and coherence is also highlighted for evaluating qualitative studies.

1. Critical Appraisal of
Qualitative Research
Dr Kritika Samsi
King’s College London

2. Outline for today
 What is qualitative research and why use it?
 Theoretical perspectives
 What this means for qualitative research methods
 Typical research process
 Rigour in qualitative research – what researchers
do
 Checklists and how to use them to critically
appraise
 Assessing qualitative research in mixed-method
studies

3. What characterises qualitative research?
 Concerned with nuances of meaning and in-depth
understanding
 Predominantly inductive method of enquiry, i.e. bottom-up
data-driven approach (Bryman, 2004, p.9)
 Contextual: Importance of understanding human experience
at an individual perspective (Denzin and Lincoln, 2008)
 Flexible research strategy: Commitment to iterative process
 Rich data: Depth versus breadth
 Small sample but high detail
 Analysis is descriptive and interpretative
 Researcher’s standpoint acknowledged and
questioned (reflexivity)

4. Why use qualitative research?
 To understand social phenomena from individual
perspective
 E.g. experience of stigma & discrimination
 To understand behaviour where relevant variables are not
apparent
 Non-adherence to treatment in high risk group
 To study human experience in
natural settings
 Cultural or social context
 To generate plausible theory to
explain experiences and
behaviour

5. Qualitative Research Questions
? To explore
? To understand…
Barriers and facilitators
Role relationships
Why do…?
How do…?
Attitudes and
beliefs
Perceptions
Coping strategies

6. Qualitative Research Questions
To explore African American Families' Expectations and Intentions for
Mental Health Services. (Thompson et al., 2012)
To understand the attitudes, the social pressures (subjective norms) and the enabling
factors (environment: services access and quality, time, money etc.,) that influenced the
decisions and ability of pregnant women and their families to utilize available safe birthing.
(Khan et al., 2012)
To explore barriers and facilitators to cancer education. (Louis-Nance et al., 2012)
How do mentor mothers living with HIV in South Africa cope with
potential impact on their role? (Dhlamini et al., 2012)
What is the lived experience of mothers and families aiming for the clinical
ideal of breastfeeding their new born for 6 months? (Hoddinott et al.,
2012)

7. Theoretical perspectives in qualitative
research
 Based on an episetomology of social
constructionism (Denzin and Lincoln, 2008)
 No single concrete reality, every individual’s
perception of it contributes to understanding the
larger phenomenon
 No one’s reality is more important or “more real”
than another’s
 “Subtle realism” (Hammersley, 1992; Snape & Spencer, 2003)
 There is a single concrete reality, and we can know it
– to a degree – by using appropriate methods

8. Common methodological approaches
Ethnography (Hammersley & Atkinson,
2007)
Grounded theory (Charmaz, 2006)
Discourse analysis (Potter, 1996)
Interpretative Phenomenological
Analysis (IPA) (Smith et al., 2009)
Content Analysis (Weber, 1990)
Framework analysis (Pope et al.,
2000)
Inductive
Deductive
Abstract
Concrete

9. Theory & Methods –
What do similarities/differences mean?
Ethnography Grounded theory Interpretative
Phenomenological
Analysis
Discourse analysis Content Analysis Framework alysis
Aim &
Research
Question
Immersion in
natural setting to
gain insider
experience; e.g.
service evaluations,
uptake of
medications
To generate theory
from empirical data;
e.g. stigma in mental
health; beliefs and
perceptions of
mental in different
ethnic groups
To understand
individual in-depth
experience; e.g. lived
experience of coping
with dementia
To capture nuances
of text or public
discourse; e.g.
understanding
political theory,
social change,
cultural contexts
To capture the
meaning at
descriptive level; e.g.
why do carers access
services for their
relatives with
anorexia?
Mainly used for
“problem-oriented”
methods in health
services research;
e.g. what are the
training needs for
primary care staff?
Sampling
&
Methods
Observational
studies
•Range of
perspectives and
stay true to research
question;
unstructured
questionnaire
•Theoretical
sampling
•Homogenous
sample and stay true
to participants’
stories;
unstructured
questionnaire
•Purposeful
sampling
•Documents,
speeches,
newspapers, mass
media
•Purposeful/
Theoretical
sampling
•Documents, e.g.
newspapers, mass
media
•Purposeful/
Convenience
sampling
•Interviews with
semi- structured
questionnaire
•Purposeful/
Convenience
sampling
Analysis Data-driven; but no
fixed commitment to
developing new
theory
•Data-driven
•Constant
comparison and
iterative approach
Identification of
descriptive and
interpretative
themes, that actively
engages the
researcher and
participants
Detailed, thorough
analysis of
discourses –
speeches,
conversations,
written text
Deductive approach
as categories are
identified in advance;
involves counting of
frequencies, i.e.
number of times a
topic arises
•Theory driven
•Deductive approach
starting with
questions on
interview topic
guide
Researche
r’s
position
R's skill and neutral
position vital
R’s position or
potential ‘bias’ is
managed
R’s position is
paramount;
importance of
reflexivity
High level of
interpretation or
abstraction expected
•R’s position neutral
•Inter-rater reliability
often calculated
Neutral position of
researcher; limited
interpretation

10. Theory & Methods –
What do similarities/differences mean?
“Thematic analysis” is method of data analysis – used more
commonly in pragmatic research
(e.g. service development, control-group studies of RCTs)
(Braun & Clarke, 2008)

11. Typical Research Process
1. Literature review
 What are the gaps in understanding/evidence/theory?
2. Formulate research question
3. Choose methodology
4. Choose appropriate methods
5. Select populations & settings appropriate to sampling
strategy
6. Consult experts/stakeholders
 Consider cultural issues, sensitivity, access; prepare and
test topics for interview
7. Iterative data analysis according to appropriately chosen
methodology

12. Rigour in qualitative research
What do researchers do to impose rigour?

13. Principles of rigour
 Credibility and authenticity
 Critical appraisal and integrity
How can the researcher remained faithful
to participants’ experience? (Hammersley, 1992)
Has the researcher explore alternative explanations,
discrepant data, examine bias & etc.?
(Graneheim and Lundman, 2004)

14. Techniques for imposing rigour
Triangulation Reflexivity
Multiple coding Respondent validation
Deviant case analysis

15. Techniques for imposing rigour
Triangulation
“The use of more than one method or source of
data in the study of a social phenomenon”
Types:
•Multiple methods
•Focus groups/ interviews/ journals/ observations
•Multiple data sources
•GPs/ patients/ carers/ social care staff
•Multiple researchers
•To get different observations on same situation

16. Techniques for imposing rigour
Reflexivity
•Position of researcher in research
•Interaction with participants
•Characteristics of researcher:
•e.g. age, gender, profession, relationship,
personal experience
•Consider what this might imply and try to
limit the effects

17. Techniques for imposing rigour
Multiple coding
•2 or 3 independent researchers
•Different disciplinary backgrounds
•Calculate inter-rater reliability
•Process rather than value
•Discuss and resolve discrepancies
•Generates ideas

18. Techniques for imposing rigour
Respondent validation
•Taking findings back to participants
•Getting their opinions on interpretations and
implications drawn from their interviews
•Incorporate this into analysis
•Must be aware of the limitations
•Researcher aiming to provide overview
•Generates further data to be interpreted

19. Techniques for imposing rigour
Deviant case analysis
•Process of exploring experiences of those
participants who appear to be ‘deviant’ from
responses of the norm
•Encourages the researcher to examine, question,
develop and refine the emergent theory further
•Most prevalent in grounded theory

20. Summary – of what researchers do
 Attempt to control subjectivity while staying true to
participants’ story
 Acknowledge their own relationship with
study/participants/data and question implications
on study findings
 Capture as many (subjective!) viewpoints as
possible to gain a greater range of perspectives
 Specifically go after and examine “outliers”, through
‘deviant’ participants, and multiple coding exercises
that challenge assumptions

21. Checklists

22. Caveat!
Readers and researchers should be wary of using
criteria checklists as “cook books” to enhance rigour
of their study
Understanding and appreciating qualitative principles
and methodological implications more important
(Barbour, 2001; Elliott et al., 1999; Kuper et al., 2008a/b)
Individual researcher skill still important (necessary?)
for good qualitative research – upto readers to
ascertain whether that has been done

23. Questions to ask (from Kuper et al, 2008a)
1. Was the sample used in the study
appropriate to its research question?
2. Were the data collected appropriately?
3. Were the data analysed appropriately?
4. Can I transfer the results of this study to my
own setting?
5. Does the study adequately address potential
ethical issues, including reflexivity?
6. Overall: is what the researchers did clear?

24. Was the sample used appropriate to its
research questions?
 How were participants selected and
recruited?
 Were they relevant to the research question?
 Was sampling strategy justified?
 Was the sampling purposive/ theoretical?
 Was it a convenience sample?

25. Were the data collected appropriately?
 Were the data collection
methods appropriate for
research objectives and
settings?
 Field observation
(participant/ non-
participant)
 Interviews (in-depth; focus
groups)
 Document analysis (diaries;
letters; newspaper articles)
 Was there explicit
consideration of how this
might have influenced
findings?

26. Sample size dilemma?
 Not an issue of sample size
in statistical sense
 Data collection needs to be
comprehensive enough in
breadth and depth to
generate and support
interpretation
 Adequacy depends on
emerging findings
 Need for iterative process
 Can the data be audited?
 Paper trail

27. Were the data analysed appropriately?
 Transparency of analytical methods – especially for
interpretation
 Systematic approach should have been used
 Data-driven or theory-driven (e.g. thematic analysis;
grounded theory)
 Efforts made to describe contradictory data and
divergent findings
 Multiple coding – so findings are corroborated by
more than one researcher
 Do results look credible and justify conclusions?

28. Can I transfer results of this study to my
own setting?
 Qualitative research is contextual – i.e. not seeking to be
generalisable
 “Transferability” more important principle (Kuper et al.
2008a)
 How do these apply in other contexts/ situations/
real world?
 Onus for assessing transferability rests with reader
 Author has to describe setting and context
transparently and honestly
 Real-world implications for practice should be clearly
described, if not obvious

29. Does the study adequately address potential
ethical issues, including reflexivity?
Essentially, balancing moral actions of the researcher and virtues of
research aims and outcomes
Principles to follow:
1. Autonomy
2. Beneficence/non-maleficence
3. Justice
4. Misrepresentation
 In qualitative research, components of ethical research relationship
 Acknowledgement of bias
 Rigour
 Rapport and managing distress
 Respect for autonomy
 Confidentiality, especially in research reports
 Avoidance of exploitation (being aware of power relationships)
Research
Ethics
Committee

30. The importance of relevance
A form of transferability, ‘relevance’ refers to
emphasizing the value of the study and the wider
implications of it (Mays & Pope, 2000)
 “So what?”
 What contribution does the study make to existing
knowledge?
 Are the limitations thoughtfully discussed?
 How do the findings fit with existing theory?
 Does it contribute by developing new theory?
 What are the implications for practice/service?
 Has the study been disseminated responsibly?

31. Coherence & Transparency:
Is what the researchers did
clear?

32. Assessing qualitative research in mixed-
method studies
 Integrating qualitative and quantitative methods should be
done from conception, design to conclusion
 Often ensures triangulation
 Has study been adequately justified?
 Does it explain descriptive, reductionist quantitative
study
 Rationale presented for chosen methodology
 Sampling conducted appropriately?
 Ethical balance – power relationships considered?
 Recruiting participants from control group
 Have findings from both studies been presented jointly?
 Is it necessary?

33. Qualitative + Quantitative
Research study and
methods should be
designed and
chosen on the basis
of the research
questions.

34. CASP Quality checklist
Screening questions
1. Was there a clear statement of the aims of the research?
2. Is a qualitative methodology appropriate?
Detailed questions
3. Was the research design appropriate to address the aims of the
research?
4. Was the recruitment strategy appropriate to the aims of the research?
5. Were the data collected in a way that addressed the research issue?
6. Has the relationship between researcher and participants been
adequately considered?
7. Have ethical issues been taken into consideration?
8. Was the data analysis sufficiently rigorous?
9. Is there a clear statement of findings?
10. How valuable is the research?
Is it worth continuing?

35. Thank you!
kritika.1.samsi@kcl.ac.uk

36. Questions?

37. References
 Barbour R (2001) Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ,
322, 115-117.
 Braun V & Clarke V (2008) Using thematic analysis in psychology, Qualitative Research in Psychology, 3 (2): 77-101.
 Bryman A (2004) Social Research Methods, Oxford: Oxford University Press.
 Charmaz K (2006) Constructing grounded theory: A practical guide through qualitative analysis. Sage, London.
 Critical Appraisal Skills Programme: http://www.casp-uk.net/
 Denzin N & Lincoln Y (2003) Introduction: The discipline and practice of qualitative research. In Denzin & Lincoln
(editors), Strategies of qualitative inquiry (2nd dition). Thousand Oaks, CA: Sage.
 Dhlamini L, Knight L, van Rooyen H, van Heerden A & Rotheram-Borus M (2012) Qualitative interviews with
mentor mothers living with HIV: potential impacts of role and coping strategies, J Int AIDS Soc, 11;15(4):1-4.
 Elliott R, Fischer C & Rennie D (1999) Evolving guidelines for publication of qualitative research studies in
psychology and related fields. British Journal of Clinical Psychology, 38, 215-229.
 Graneheim U & Lundman B (2004) Qualitative content analysis in nursing research: Concepts, procedures and
measures to achieve trustworthiness, Nurse Education Today, 24, 105-112.
 Hammersley M (1992) What's Wrong with Ethnography?, London, Routledge.
 Hammersley M & Atkinson P (2007) Ethnography: Principles in Practice – 3rd edition; New York: Routledge.
 Hoddinott P, Craig L, Britten J & McInnes R (2012) A serial qualitative interview study of infant feeding experiences:
idealism meets realism. BMJ Open;2:e000504 doi:10.1136/bmjopen-2011-000504.
 Khan M, Mirza S, Ahmed M, Rasheed A, Khan A, Walley J & Nisar N (2012) Making birthing safe for Pakistan women:
a cluster randomized trial, BMC Pregnancy Childbirth, 12(1):67. [Epub ahead of print]

38.  Kuper A, Reeves S & Levinson W (2008b) An introduction to reading and appraising qualitative research. BMJ, 337:
a288.
 Kuper A, Reeves S & Levinson W (2008a) Critically appraising qualitative research, BMJ, 337: a1035.
 Louis-Nance T, Flournoy M, Clinton K, Hightower K, Sebastian N, Wilkinson L & Glover S (2012) The females against
cancer educational series: a qualitative evaluation of mother/daughter knowledge and perceptions of human
papillomavirus and its related cancers, J Natl Med Assoc, 104(3-4):194-8.
 Mays N & Pope C (2000) Qualitative research in healthcare: Assessing quality in qualitative research. BMJ, 320, 50-
52.
 Pope C, Ziebland S & Mays N (2000) Qualitative research in health care: Analysing qualitative data, BMJ, 320(7227):
114–116.
 Potter J (1996) Discourse Analysis and Constructionist Approaches: Theoretical Background. In: Richardson J (Ed.)
Handbook of qualitative research methods for psychology and the social sciences. Leicester: BPS Books.
 Smith J, Flowers P & Larkin M (2009) Interpretative Phenomenological Analysis: Theory, Method and Research.
­London: Sage Publications.
 Snape D & Spencer L (2003) The Foundations of Qualitative Research. In: Ritchie J & Lewis J (Eds.) Qualitative
Research Practice. London: Sage.
 Thompson R, Dancy B, Wiley T, Najdowski C, Perry S, Wallis J, Mekawi Y & Knafl K (2012) African American Families'
Expectations and Intentions for Mental Health Services, Adm Policy Ment Health, 2012 Jul 13. [Epub ahead of print]
 Weber (1990) Basic Content Analysis. London: Sage Publications.