This presentation analyses how to critically appraise a randomised control trail paper through CASP checklist.

1. An Introduction to Critical Appraisal of
Quantitative Research
Shampa Sen
Clinical Support Librarian
Princess Royal University Hospital
Claire Jones
Ex Clinical Support Librarian
Princess Royal University Hospital

2. Introductions
• Your name​
• Your reasons for undertaking critical appraisal​
• What experience do you have with quantitative
research?​

3. Learning Objectives
Understanding the main concepts of critical appraisal
Becoming familiar with the major features and
terminology of quantitative research
Apply critical appraisal to a quantitative paper using a CASP
checklist

4. Quantitative Research: A refresher on the essentials
Quantitative research is a
systematic investigation of
phenomena by gathering
quantifiable data and
performing statistical,
mathematical, or
computational techniques

5. What is quantitative, or measurable,
data used for in research?​
• Describing treatment effect​
• Predicting results if applied
to the general population​
• Testing for causality
• Analysing averages and
patterns​
What kind of study designs
generate quantitative data?
• Observational studies, for example cohort studies or
case control studies​
• Experimental studies, for example randomised control
trials (RCTs)​

6. What is Bias in
Research ?
Bias in research refers to a
systematic error that can occur
during the design, conduct,
or interpretation of a study,
leading to inaccurate conclusions.
It can occur at any stage of
the research process and can have
a significant impact on the
reliability and validity of
the findings​.

7. Types of
Bias
Design
Sample
Selection
Performance
Language
Attrition
Detection
Confounding Reporting

8. What is Design Bias?
when the research design, survey questions, and
research method is influenced by the
preferences of the researcher rather than its
suitability to the research work
when volunteers are chosen to represent your
research population, but those with different
experiences are ignored
when survey respondents are given insufficient
time to complete surveys
A
B
C

9. Look at the Picture and Identify The Type of Bias
when the selection of participants for a research study isn't
representative of the whole population. The skewed sample
could lead to a misrepresentation of the data and flawed
conclusions.
A Selection Bias
Sample Bias
Performance Bias
B
C

10. Occurs when the method of selecting
participants or groups for a study
produces an outcome that is not
representative of the total population.
For instance, if the sample group is not
randomised or certain groups are
excluded, it could produce skewed or
incomplete results.
A
B
C
Performance Bias
Selection Bias
Sample Bias
Find Out The Type Of Bias From The Definition

11. If the investigators know that an experimental group have been given
an active drug, they may focus their attention on this group. The
participants might receive more frequent exams and more diagnostic
tests. This could result in the experimental group having a greater
chance of a positive outcome — not because they have been given an
active drug, but because they received very focused attention.
Identify The Scenario and Choose The Bias
Detection Bias Reporting Bias Performance Bias

12. A sort of bias that influences research is
publication bias. It is also known as reporting bias.
It refers to a condition in which favorable outcomes
are more likely to be reported than negative or
empty ones. Analysis bias can also make it easier
for reporting bias to happen.
Reporting
Design
Performance
A
B
C
Look at the Picture and Identify The Type of Bias

13. City Living
(exposer)
Heart Disease
(outcome)
Stress
Can You Identify The Bias?
A
B
C
Selection Bias
Confounding Bias
Sample Bias
Confounding bias due to a third factor that distorts the
association between exposure and outcome.

14. _________________ bias refers to systematic differences between
groups of a study in how the outcome is assessed. In other
words, _________ bias occurs when the patient’s
characteristics influence the probability of detecting the
outcome.
Performance Design Detection
Fill In The Gaps

15. Attrition bias is the selective dropout of some participants who systematically differ
from those who remain in the study. Almost all longitudinal studies will have some
dropout, but the type and scale of the dropout can cause problems.
Attrition is participant dropout over time in research studies
Design Bias Performance Bias Attrition Bias

16. Take 3 minutes to skim through
the article again
• Do you have any initial thoughts regarding
potential strengths or weaknesses in this trial?
• Right: Photo of the helmet described in this
article​.
• From: https://www.uchicagomedicine.org/patie
nt-care/helmet-based-ventilation-is-superior-to-
face-mask-for-patients-with-respiratory-distress

17. How do I know what to look out for?
Two examples of freely available checklists you could use to structure your appraisal:
CASP -
https://casp-uk.net/casp-tools-checklists/

18. Critically appraising for antiracism
Critically Appraising for Antiracism - Ramona
Naicker's presentation at the UHMLG Spring Forum 2022
Critically Appraising for Antiracism – Mission
statement and quality assessment tool

19. Critically appraising for antiracism
• Minoritised ethnic populations are underrepresented in clinical research
• Disparities in health outcomes for minoritized ethnic populations are
sometimes attributed to notions of self-identified 'race' or 'cultural differences' rather
than genetic ancestry and social inequality:
"Blaming culture, as with blaming race, blames the patient and assumes that there is
little that the health or political system can do to alleviate these inequalities"
R NaickerCritically Appraising for Antiracism (criticallyappraisingantiracism.org)

20. CASP Q1. Did the study address a clearly focused
research question?
Consider PICO:
• Who is the population under study?
• What is the intervention/exposure?
• What is the comparison?​
• What is the outcome?

21. CASP Q2. Was the assignment of participants to
interventions randomised?
• How was randomisation carried out?​
• Was randomisation sufficient to eliminate bias?​
• Was the allocation sequence concealed
from researchers and participants?

22. CASP Q3. Were all participants who entered the
study accounted for at its conclusion?
• Were losses to follow-up and exclusions
after randomisation accounted for?​
• Were participants analysed in the study
groups to which they were randomised?​
• Was the trial stopped early? ​If so, what
was the reason?

23. CASP Q7. Were the effects of the intervention reported
comprehensively?
• What outcomes were measured?
• Were they clearly specified?
• How were the results expressed?
• Was there any missing or incomplete data?
• Were potential sources of bias identified?
• Which statistical tests were used?
From the critically appraising for
anti-racism checklist​:
• Were the differences in
study outcomes for minoritised
ethnic populations
appropriately addressed
and interpreted?

24. Ways to describe treatment effects using the data
Outcome event​ ENDOTRACHEAL INTUBATION
Total​
Yes​ No​
Experimental group​
​(HELMET)
a​ b​ a + b​ =
Control group​
​(FACE MASK)
c​ d​ c + d​ =
Total​ a + c b + d​ a + b + c + d​ =
(Raw data
taken from
the top of
Table 2 in
the article)
Using equations to turn this raw data into expressions of measurement
8
24
32
36
15
51
44
39
83
Experimental Event Rate (EER) =
𝑎
𝑎+𝑏
8
44
= 0.182 or 18.2% (of patients with helmet had to be intubated)
Control Event Rate (CER) =
𝑐
𝑐+𝑑
24
39
= 0.615 or 61.5% (of patients with face mask had to be intubated)
Absolute Risk Reduction (ARR) = CER – EER = 43.3% (the measure of the risk of intubation with helmet)
Relative Risk Reduction (RRR) =
𝐶𝐸𝑅 −𝐸𝐸𝑅
𝐶𝐸𝑅
= 70.4% (measure of the risk of intubation when compared
to the face mask)

25. CASP Q8. Was the precision of the estimate of the
intervention or treatment effect reported?
• Are the results statistically significant?
• Is there a true difference between outcomes in the control and
intervention groups (or was it down to chance)?​
• Three standard tests you can expect to see:​
• Power
• P-values
• Confidence intervals
CASP Q9. Do the benefits of the experimental intervention
outweigh the harms and costs?

26. Power
• Power determines an adequate sample size
• To aim for a higher power figure = to aim for a larger sample size
= a more robust trial
Q: What was the power in this paper?
Q: How many participants were needed to achieve this power?
A: Aimed for 80% (p.3)
A: 206 participants but trial was stopped early. The trial is therefore ‘underpowered’

27. P-values = probability values
• Assesses probability that the results have arisen by chance
• Assesses the likelihood of a causal relationship between the intervention
being tested and the outcome event
Q: What are the p-values in this paper?
Q: Are they statistically significant?
A: 0.001 for primary outcome (table 2).
A: Yes they are; there is only a 1 in 1000 chance that the results are due to pure chance.

28. Confidence Intervals (CI)
• “Any result obtained in a sample of patients can only give an estimate of the result
which would be obtained in the whole population. The real value will not be known,
but the confidence interval can show the size of the likely variation from the
true figure" (Dr Kate O'Donnell, University of Glasgow for NHS SE London Health Libraries)
• Important for clinicians when basing decisions on likelihood of effectiveness
Q: What confidence interval is given for the primary outcome in this paper?
A: 95% CI. In other words, there is a 95% likelihood that the true result
lies within the range specified (in this case, the range is -62.4 to -24.3)

29. Example CIs presented in a forest plot graph
A: They cross the line of no effect. For that confidence interval the result will sometimes be that
there is no detectable difference in health outcome from the treatment, therefore we cannot
reject the null hypothesis / cannot categorically say this treatment will always work.
Q: Why are some of the confidence intervals marked as 'Not Significant'?
Absolute difference (0 represents the line of no effect) Relative difference (1 represents the line of no effect)

30. Cardiac deaths –
Less = good
31

31. Smoking cessation –
More = good
32

32. How might we summarise these results?
33
• Results are clinically significant
• 0.001 P-value = 1 in 1000 probability the outcomes are down to chance
• The CI does not cross the line of no effect: this suggests we could reject the
null hypothesis
• Underpowered trial; the researchers acknowledge the positive results may
be magnified
• Further similar trials are needed to narrow down the confidence interval
Weaknesses
Strengths

33. CASP Q4. Were patients, health workers and study personnel
‘blind’ to treatment?
• Preventing those involved in a trial from knowing
which group a participant belongs to so that
behaviour doesn’t change as a result of that knowledge
• Is the description of the blinding clear?
• Who was blinded?
• Blinding of certain groups is not always possible

34. CASP Q5. Were the study groups similar at the start of the trial?
35
• Were the baseline characteristics of each study group
clearly set out?
• Were there any differences between the study groups
that could affect the outcome?
• If the groups are not balanced, is this acknowledged
and what steps have been taken to overcome the
problem?
From the critically appraising for anti-racism checklist​:
• Were minoritised ethnic participants recruited?
• Is the sample of minoritised ethnic participants
representative?

35. CASP Q6. Aside from the experimental intervention did each study
group receive the same level of care (were they treated equally)?
36
• Was there a clearly defined study protocol?
• If any additional interventions were given, were they similar between the study
groups?
• The groups should be treated the same in every respect apart from the difference
in interventions so that differences in outcomes are due to the intervention being
investigated.
More info: Glasziou et al. (2007). Evidence-based Practice Workbook (2nd Ed.), p.83

36. Tips for success
37
• Group work
• Read all the paper
• Review and feedback
‘Undertaking a critical appraisal is really using your everyday skills,
and applying them in a more structured and systematic way’
Dawes (2005). Evidence-based Practice: a Primer for Healthcare Professionals (2nd Ed.)

37. Thank you for attending this session
Reflection and feedback
Email: Shampa.sen@nhs.net
OR
kch-tr.br-library@nhs.net
Telephone: 01689 864306