RCT Critical Appraisal - Validity

Critique of Therapy
Article - Validity
Dr. Majdi N. Al-Jasim
SBFM, ABFM
Consultant Family Physician
PCFCM - AlAhsa

OBJECTIVES
•To explain how to choose article to read.
•To assess the quality of the therapy article to
determine if the conclusions are valid.
•To discuss the relevance of the read articles to
our practice and if we can apply their results.
Dr. Majdi Al-Jasim; SBFM, ABFM

Is it possible to be always updated?!
A lot of Articles

Before Reading
Before reading any article, try to answer these questions:
1. Is it relevant to my practice?
2. Is the sample similar to my patients?
3. Are the results DOE or POE?
4. Will it change my practice?

Will you read this article?
I doubt
that

What about this article?

Evidence Based Medicine
“Evidence-based medicine is the integration of best research evidence with
clinical expertise and patient values”
- Dave Sackett

Steps of EBM
1: ASK
Formulate clinical
question (PICO)
2: ACQUIRE
Searching the
evidence
3: APPRAISE
the evidence
4: APPLY
evidence into
decision-making
5: ASSESS
Evaluate the
process PATIENT-
CENTERED

What is Critical Appraisal?
It is a process of systematic examination of research
article to assess its validity, results and applicability.
The three key steps in critically appraising
an article are:
1. Was the study valid?
2. What are the results?
3. Are the results applicable to our patients?

How to evaluate an article
By using critical appraisal toolkit (CAT) for each
specific article type:
▪ Therapy (RCT)
▪ Systematic Review and Meta-analysis
▪ Prognosis (Cohort or case-control study)
▪ Diagnosis

Critical Appraisal Toolkit (CAT)

Appropriate Study Designs
Diagnosis
Prospective cohort study with good quality
validation against “Gold Standard”
Therapy Randomized controlled clinical trial (RCT)
Etiology
Case-control study or retrospective cohort
study
Prognosis Prospective cohort study

Validity
Results
Apply it
Is the study done in a correct way?
(Methodology Section)
Are results significant?
(Results Section)
What are benefits and risks?
Is this study generalizable?
Critique of Therapy Article (RCT)

General Scheme of RCT
Eligible sample
Excluded

Types of Endpoint (Outcomes)
Primary Outcome:
Is the outcome of interest that has been set by the researcher at
the beginning of study.
Secondary Outcome:
Is the additional outcome that occurs during study.
Surrogate Outcome:
Is an indirect lab result or physical sign used as a substitute to
measure primary or secondary outcome. E.g. in some statin
trials, they measure LDL level to indicate the rate of IHD.

Types of Endpoint (Outcomes)
Composite Outcome:
Multiple outcomes combined together.
Example:
IHD deaths, arrhythmia deaths, valvular heart disease deaths 
cardiac causes of death as composite.
Advantages:
▪ Easier.
▪ Increase number of events by magnifying the result.
Disadvantage:
When combined together, the event rate increases, so the
event of interest may reach significant or non-significant level
which may not reach if it was analyzed alone.

The new
intensive Rx
has more
serious SE

RCT Validity

Closeness to the truthValidity
Internal
Look for the presence of Biases (the deviation of the
results of a study from the truth because of the way it
has been conducted, analyzed or reported).

Closeness to the truthValidity
External
Look for the presence of Random Error (when the study
sample is not representative to general population; i.e.
not similar to the community).
Applicability

RABI
Validity Mnemonic

ValidityMnemonic

Was the assignment of patient
“randomized”?

▪ It is a process where each patient has
equal chance to be in control group or
experiment group.
▪ Every one in the sample has 50% chance
to be in either group, experiment group
or control group.
Randomization RABI

Why randomization?
1. To ensure equal base-line characteristics in both
groups.
2. Both groups are equal in known or unknown
prognostic factors and other confounders.
Randomization RABI

Concealed Allocation
▪ Assigning of patient to control or experiment group before the
beginning of study in a concealed manner.
▪ Concealment is done by opaque envelope, computerized
protected folder, and remote central call system.
Hide the patient
RABI

Selection Bias
Selection bias:
When you select intentionally
more healthy subject to be in the
experiment group and more ill
subject to be in the control group
in order to have excellent results
in experiment group.
RABI

Practice
You have come across this article in New England
Journal of Medicine, and you want to check its validity.

RCT CAT:
Was the assignment of
patients to treatment
groups randomized?

Types of RCT Randomization Design
The RCT randomization design is how the experimental and
controlled groups are distributed randomly after sampling
takes place.
The most commonly used randomization designs are:
1. Parallel
2. Cross-over
3. Stratified (clustered)
4. Factorial

Parallel:
Each participant is randomly assigned to a group, and all the
participants in the group receive (or do not receive) an
intervention.

Cross-over:
Each participant receives first treatment and then after period
received the second one in a random sequence. Its advantage is
the elimination of variations between groups.

Stratified (clustered):
First you divide the sample into different strata (clusters) and
then do randomization from each strata. Its advantage is the
elimination of variations between groups.
Experiment
Females
Control
Sample
Experiment
Control
Males

Factorial:
Each participant is randomly assigned to a group that receives a
particular intervention (alone or combination or neither one).

Were all patient who entered the trial
properly “attributed” at its
conclusion?
RABI

Attrition (loss of F/U)
▪ This includes dropouts, and withdrawals.
▪ Accepted when ≥ 80% of patients completed the study to
maintain power of the study.
▪ Duration of study should be sufficient for primary outcome to
occur.
RABI

How to deal with cases of attrition if >20%?
1. Follow the events in dropped out participants:
▪ Just look for the occurrence of events in lost participants and
add them to their corresponding group events. This can be
achieved by reviewing their files in the same or other
hospitals.
▪ If this is impossible, you either considered them to have the
same event rates of those who continued the study, or do
sensitivity analysis (mostly used).
Attrition (loss of F/U) RABI

How to deal with cases of attrition if >20%?
2. Sensitivity analysis:
▪ Re-analyzed the results based on best and worst case
scenario.
▪ If there is significant difference (p < 0.05) between actual
results and sensitivity analysis, then the study is affected by
loss of follow up.
Attrition (loss of F/U) RABI

Attrition Bias
Attrition bias:
When you loss the follow up of your
participants in the study. It is more
common in Cohort and Randomized
Clinical Trials.
The accepted rate of loss of follow up
must be ≤ 20%.
RABI

RCT CAT:
Were all patient who
entered the trial
properly “attributed”
at its conclusion?
Nothing mentioned
about dropped out.

Were patients, physicians and those
during assessment "blind" to
treatment?
RABI

Blindness
RABI
Blindness could be:
1. Single: either patient or assessor blinds to treatment; but
usually it is the patient.
2. Double: patient and assessor blind to treatment.
3. Triple: patient, assessor and data analyst blind to treatment.
How is it done?
1. Same shape, color, taste for both control and experiment
treatment.
2. Dummy tablets.
Hide the treatment

Detection Bias
Detection bias:
When the assessor or the patient able to
predict the treatment that is used during
RCT.
This bias can happened in single-blind
trial or in open-label trial.
RABI

RCT CAT:
Were patients and
physicians and those
during assessment
blind to treatment?

Were all patient who entered the trial
properly “accounted for” at its
conclusion?
RABI

Intention To Treat (ITT) analysis
▪ It is analysis of the primary outcome according to each group
original primary allocation.
▪ Once patients are randomized, they should be analyzed in the
same group even if there was drop out and because of that
the number of analyzed patients in primary outcome is the
same number of patients the trial started with.
▪ It is different from “Per Protocol Analysis” in which only
analysis occurs to only those who completed the study.
RABI

Advantage of ITT analysis:
Maintains randomization.
Disadvantage of ITT analysis:
Gives false effect estimate of experiment or control treatment
because of:
1) Including dropped out patients.
2) Including non-compliant patients.
3) Including patients with contamination and or cointervention
RABI
Intention To Treat (ITT) analysis

RCT CAT:
Were all patients who
entered the trial
properly “accounted
for” at its conclusion?

RCT CAT:
Were all patients who
entered the trial
properly “accounted
for” at its conclusion?
ITT analysis

Post Hoc Analysis
▪ When the researcher re-analyze data again and again in
different ways in order to find something extra (fishing data).
▪ This analysis type usually occurs after having non-significant
primary or secondary outcome.
▪ Most of authors used sub-group analysis as post hoc analysis in
which the analysis occurs in fragmented groups of the sample.
RABI
Fishing the
significant data.!!

1 1.5 20.50
Overall
Males
Females
95% CI (0.6 – 1.3); p= 0.12
95% CI (0.3 – 0.8); p= 0.03
95% CI (0.8 – 1.4); p= 0.20
Favor treatment Favor placebo
Sex
Age
< 65
≥ 65
95% CI (0.5 – 1.3); p= 0.35
95% CI (0.6 – 1.4); p= 0.51
The overall
outcome is not
significant but
sub-group
analysis showed
that males
actually might
benefit from
treatment.
Post Hoc Analysis

RABI
Validity Mnemonic RABI

Were all groups
SIMILAR at the start
of trial?
RABI

▪ Similarity in baseline prognostic factors, demographics,
comorbidity, disease severity, other known confounders.
▪ It is achieved by proper randomization in which the
resulted difference between study groups should be not
significant (p ≥ 0.05) to ensure similarity.
▪ Usually this information is presented in table 1 in article.
Identical
RABI

Identical
RCT CAT:
Were all groups similar
at the start of trial?

3C
Validity Mnemonic

Aside from the intervention,
were the groups TREATED IN
THE SAME WAY ?

• Co-intervention
Any extra intervention other than study treatments to either group;
like outside drug.
• Contamination
Any member from one group received treatment from other group that
is included in the study.
• Compliance
✓ Was it mentioned?
✓ How is it looked for?
EQUAL TREATEMENT

Performance Bias
Performance bias:
When there is:
▪ Contamination.
▪ Co-intervention.
▪ No blindness.
▪ Placebo effect*.
*Placebo effect means the patient has a belief in
placebo treatment that it can cure him/her.
If the whole control group has the same belief, it may
affect the study results.

RCT CAT:
Aside from intervention,
were groups treated in
the same way?
Nothing mentioned
about contamination
or co-intervention.
Compliance

Applicability

Applicability
Can I apply the study results on my patients?
Check if the study population are similar to your patients.Similarity

Applicability
Do benefits overweight risks
in applying the results?
Check for the important side
effects of this new treatment.
You need to calculate NNT and
NNH to compare.
Benefits
Risks

Applicability
Is its cost feasible?
How much will it cost?
Is it affordable to my
patients?

Applicability
Are the results DOE or POE?
Were quality of life used?
Were most important outcomes
(bad or good) mentioned?

SUMMARY

Thanks

RCT Critical Appraisal - Validity

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to RCT Critical Appraisal - Validity

Similar to RCT Critical Appraisal - Validity (20)

More from Dr. Majdi Al Jasim

More from Dr. Majdi Al Jasim (11)

Recently uploaded

Recently uploaded (20)

RCT Critical Appraisal - Validity