RCT Critical Appraisal - Results

Critique of Therapy
Article – Results
Dr. Majdi N. Al-Jasim
SBFM, ABFM
Consultant Family Physician
PCFCM - AlAhsa

OBJECTIVES
To Discuss the Quantitative Measures Used in Therapy Articles:
▪ P-value
▪ Confidence Interval
▪ Event Rates
▪ Relative Risk
▪ Relative Risk Reduction
▪ Absolute Risk Reduction
▪ Number Needed to Treat or to Harm
Dr. Majdi Al-Jasim; SBFM, ABFM

Validity
Results
Apply it
Is the study done in a correct way?
(Methodology Section)
Are results significant?
(Results Section)
What are benefits and risks?
Is this study generalizable?
Randomized Controlled Trial Critique

P-value
▪ Used to declare Statistical Significant results.
▪ It emphasizes that how the results occur by chance.
▪ The commonly accepted cut point for calling a result
“statistically significant” is p<0.05
Example:
In a multiple choice question with 5 choices, there is a chance by
20% that you can answer the question correctly even if you did
not study well.!!!
So we can say the p-value here is 0.2

▪ A range of values that is almost sure to contain the true
population parameter.
▪ 95% CI means: if we repeat the study 100 times, the results in
95 studies will be within the CI range.
Example:
If 95% CI (5 – 13), then we are sure by 95% that if we repeat the
study again and again, the result will be between 5 and 13.
Confidence Interval (CI)

▪ A narrow range CI, means more precision; and it means a
large sample size most likely used in that study.
Example:
95% CI (3 – 25)
Wide range CI  Small sample
95% CI (0.7 – 1.2)
Narrow range CI  Large sample

▪ In studies that use any ratio in their outcome (like Relative
Risk, Odds Ratio), if CI range crosses the value “1”, then the
result is not statistically significant.
▪ In studies that use difference in their outcome (like weighted
mean difference), if CI range crosses the value “0”, then the
result is not statistically significant.

95% CI (0.2 – 0.7)
“Significant”
95% CI (0.9 – 6.7)
“Not significant”
95% CI (6.8 – 9.7)
“Significant”

Which of the following 95% CI represent significant results and
large sample size, assuming the result is a ratio:
 95% CI (12 – 105)
Significant but small sample size
 95% CI (0.8 – 1.2)
Not significant but large sample size
 95% CI (0.1 – 0.4)
Significant with large sample size

P-value & 95% CI

Event Rate
The rate of having new outcomes (i.e. incidence) in the group
who took the new treatment (Experiment Event Rate) or the
other group who took old treatment or placebo (Control Event
Rate).
Formula:
Experiment Event Rate (EER) = Events / Group Total
Control Event Rate (CER) = Events / Group Total

Event Rate
EER = 4/8
CER = 2/8

Relative Risk (RR)
The risk of having the outcome (harm or benefit) among
experiment group to those in control group.
➢ Some authors call it Risk Ratio.
➢ If it is over a period of time, it is called Hazard Ratio (HR).
Formula:
RR = Experiment Event Rate / Control Event Rate
RR = EER / CER

EER = 4/8
CER = 2/8
RR = EER/CER
RR = (4/8)/(2/8)
RR = 2
Relative Risk (RR)

Interpretation:
▪ Harmful outcome:
➢ > 1  the experiment increases the harm.
➢ < 1  the experiment decreases the harm.
➢ = 1  no difference between experiment or control group.
▪ Beneficial outcome:
➢ > 1  the experiment increases the benefit .
➢ < 1  the experiment decreases the benefit.
➢ = 1  no difference between experiment or control group.
Relative Risk (RR)

Kaplan Meier Hazard Function Plot
Relative Risk (RR)
Death from Cardiovascular
causes in Empagliflozin
group is 5 within 4 years.
Death from Cardiovascular
causes in placebo group is 9
within 4 years; p<0.001
Conclusion:
Empagliflozin is better than
placebo in decreasing the
death from cardiovascular
causes within 4 years.
Always…
In Hazard
Function Plot,
the lower curve
is better than
the upper curve
if the outcome is
harmful and vice
versa for
beneficial
outcome.

In an article, the risk of having stroke in group taking aspirin is 0.6 times
compared to the group who takes placebo; 95% CI (0.4 - 0.7).
What do you understand from the result?
Relative Risk (RR)

Analysis:
RR = 0.6
We noticed that RR < 1
The outcome is stroke (harmful outcome).
So, the aspirin will decrease the stroke events.
95% CI (0.4 - 0.7) is not crossing the value “1”, so the result is statistically
significant. Also it is narrow which means large sample size used in this
study.
But for how much?
Relative Risk (RR)

Here RR is 0.6 which means the risk of stroke in aspirin group is 60%
of that in control group.
In other ward, if control group has risk to develop stroke by 1000
cases in million then the aspirin group will be 60% of that (i.e. 600
cases in million). So the actual decrease is 40% (i.e. 400 cases). That
is why they came up with another measuring parameter called
Relative Risk Reduction (RRR).
Relative Risk (RR)

Relative Risk (RR)
▪ Relative Risk (RR) just tell if there is
increase or decrease in the outcome
in experiment group; but it doesn’t
tell for how much.
▪ The actual decrease or increase is
calculated as the rest of RR and so
here is where we need to use
Relative Risk Reduction (RRR).

Relative Risk Reduction (RRR)
Used to see how much the experiment treatment is
relatively reducing the chance of having outcome in
treated patient.
➢If it measures a beneficial outcome, it is called in some
articles Relative Benefit Increment (RBI).
Formula:
RRR = 1 - RR

Interpretation:
Using experiment treatment will relatively reduce the risk of having
the outcome by (%) compared to control treatment.
Example:
If RRR = 70% in comparing ACEI vs placebo in decreasing IHD. This
means in a person who is treated with ACEI, his chance of having IHD
will be relatively reduced by 70%.

In an article, the rate of developing IHD in the group using aspirin was
40%, while it was 54% in the control group.
By how much aspirin will relatively reduce the events
of IHD?

Answer:
EER = 40%
CER = 54%
RR = EER / CER = 40/54 = 0.74
RRR = 1 – RR = 1 – 0.74 = 0.26
So RRR = 0.26 = 26%
So using aspirin will relatively reduce the IHD events by 26%.

Same!!

▪ The problem with RRR is that it is bonded to RR.
▪ You can assume wrongly that the treatment is effective
in decreasing the risk of a disease whether it decreases
the risk from 4 per 1000,000 to 2 per 1000,000 or from
4 per 10 to 2 per 10 (both will have RRR = 50%).
▪ Here is where they came up with Absolute Risk
Reduction (ARR).

Absolute Risk Reduction (ARR)
Used to see the magnitude of benefit between experiment
treatment and control treatment.
➢ Some authors call it Attributable Risk or Risk Difference.
➢ If it measures a beneficial outcome, it is called in some articles
Absolute Benefit Increment (ABI).
Formula:
ARR = CER – EER

Interpretation:
if 100 patients were treated with experiment treatment, (x)
cases of outcome can be prevented; or using the experiment
treatment will absolutely reduce the outcome by (%).
Example:
If ARR = 15% in comparing ACEI vs placebo in decreasing IHD. This
means if 100 patients were treated with ACEI, 15 cases of IHD can
be prevented compared to placebo.

By how much aspirin will absolutely reduce the
events of IHD?

Answer:
EER = 40%
CER = 54%
ARR = CER – EER = 54 – 40 = 14%
So using aspiring will absolutely reduce the IHD by 14%.
That is, if 100 patients took aspirin, 14 cases of IHD can be
prevented.
Have you noticed
that how ARR is
smaller than RRR?

Same!!
Be aware from
RRR

‫التجارة‬ ‫في‬ ‫غش‬ ‫يصير‬ ‫كيف‬ ‫هل‬ ‫يعني‬!

▪ ARR some times becomes not easy to explain for some
readers, especially if we use both ARR and ABI.
▪ Here where they came up with easy way to interpret
the results in weighting the risk versus benefit; that is
Number Needed to Treat (NNT) and Number Needed to
Harm (NNH).

Widely used to weight risk versus benefit from used
treatment.
Number Needed to Treat / Harm
Benefits
Risks

▪ Preventing a harmful outcome:
➢ This is Number Needed to Treat (NNT). It is used to see how
many individuals needed to take the treatment in order to
prevent one bad event.
▪ Causing a harmful outcome:
➢ This is Number needed to harm (NNH). It is used to see how
many individuals needed to take the treatment in order to
develop one bad event.
Formula:
NNT or NNH = 1 / ARR

How many patients we need to treat with aspirin in
order to prevent one case of IHD?

Answer:
EER = 40%
CER = 54%
ARR = CER – EER = (54 – 40) = 14%
So ARR = 14% = 0.14
NNT = 1 / ARR
NNT = 1 / 0.14
NNT = 7.14
So we need to treat 7 patients with aspirin in order to prevent
one IHD event.
Calculation tip:
If you use ARR in (%), then
calculate NNT as
following:
NNT = 100 / ARR (%)

In the same article, the rate of developing hemorrhagic stroke in the group
using aspirin was 35%, while it was 19% in the control group.
How many patients we need to treat with aspirin in
order to cause one case of hemorrhagic stroke?

Answer:
EER = 35%
CER = 19%
ARR = EER – CER = (35 – 19) = 16%
So ARR = 16% = 0.16
NNT = 1 / ARR
NNT = 1 / 0.16
NNH = 6.25
So we need to treat 6 patients with aspirin in order to cause one
hemorrhagic stroke event.
Calculation tip:
If you use ARR in (%), then
calculate NNH as
following:
NNH = 100 / ARR (%)

Is it NNT or NNH that we need to use in this question?

If the bad events rate in control group are more than that in experiment
group (i.e. CER > EER), then it is NNT and vice versa for good events.
Example:
Bad
events
EER = 10.5%
CER = 12.1%
The experiment treatment causes less harmDr. Majdi Al-Jasim; SBFM, ABFM

If the bad events rate in experiment group are more than that in control
group (i.e. EER > CER), then it is NNH and vice versa for good events.
Example:
Bad
events
EER = 38.3%
CER = 37.1%
The experiment treatment causes more harm

Summary
Check the significance and how the results happen by
chance
Compare the risk in experiment versus control group
How much the risk is relatively reduced in treated
patient
Absolute reduced risk in 100 treated patients

The number needed to prevent one harm event
The number needed to cause one harm event
We are certain that the repeated results will be within
this range
Summary

For more EBM interactive lectures, I highly recommend
you visiting Prof Terry Shaneyfelt YouTube Channel:
https://www.youtube.com/user/UABEBMcourse/videos
Yours;
Dr. Majdi Al-Jasim

RCT Critical Appraisal - Results

More Related Content

What's hot

Similar to RCT Critical Appraisal - Results

More from Dr. Majdi Al Jasim

Recently uploaded

RCT Critical Appraisal - Results