This document discusses key criteria for evaluating the quality of evidence from randomized trials: 1) Groups must be comparable, which randomization aims to ensure by distributing participants evenly across groups through chance alone. Randomization can be computer-generated or through processes like coin tossing. 2) Allocation must be concealed so researchers are unaware which group a participant will be in to prevent bias. This is usually done through sealed envelopes or an off-site registry. 3) Follow-up must be complete or near-complete to prevent bias from non-random dropout patterns between groups. Less than 15% total dropout is ideal, with concerns rising above 20% where groups differ substantially.

1. Can I trust this evidence?
Dr. Maria Idrees; PT
MS (RIU)
DPT(TUF)

2. Critical appraisal of evidence
about the effects of intervention

3. Randomized trials
• Were intervention and control groups
comparable?
• It is essential that the groups are comparable,
and comparability can be expected only when
participants are randomly assigned to groups.
• Randomization is best achieved by using a
computer to generate a random allocation
schedule.

4. • Alternatively, random allocation schedules can
be generated by effectively random processes
such as coin-tossing or the drawing of lots.
• Quasi-random allocation procedures are used
• That is not to say that coin-tossing and
drawing of lots is optimal, but it may be
adequate.
• Some studies match participants and
randomly allocate participants to groups. The
technical term for this is stratified random
allocation. (Strata=Groups)

5. • It ensures that there is an even greater
comparability of groups than could be
achieved by simple random allocation alone.
• For example, a randomized trial that
compared home-made and commercially
available spacers in metered-dose inhalers for
children with asthma allocated participants to
one of four groups after stratifying for severity
of airways obstruction (mild or
moderate/severe).

6. • For readers of clinical trials the important
point is that it is the randomization, not the
stratification, that ensures comparability of
groups.
• But randomization on its own is adequate.
• Reports of randomized trials will usually
explain that participants were ‘randomly
allocated to groups’.
• This might appear in the title of the paper, or
in the abstract, or in the Methods section.

7. • One concern is that particularly naive authors
may refer to ‘random allocation’ when
describing haphazard allocation to groups.
• These authors might believe that if they made
no particular effort to ensure that participants
were in one group or the other (for example, if
participants or their therapists, but not the
researchers, determined whether the
treatment or control condition was received)
then they could call the allocation process
‘random’.

8. • True randomization can be ensured only when
randomization is concealed.
• This means that the researcher is not aware,
at the time a decision is made about eligibility
of a person to participate in the trial, whether
that person will subsequently be randomized
to the intervention or the control group.

9. • Concealment is important because, even
though most trials specify inclusion and
exclusion criteria that determine who is and
who is not eligible to participate in the trial,
there is sometimes uncertainty about whether
a particular patient satisfies those criteria, and
often the researcher responsible for entering
new patients into the trial has some latitude
in such decisions.

10. • It could seriously bias the trial’s findings if the
researcher’s decisions about who was and was
not entered into the trial were influenced by
knowledge of which group patients would
subsequently be assigned to.
• For example, a researcher who favoured the
hypothesis that intervention was effective might
be reluctant to admit patients with a particularly
severe case if he or she knew that the next
patient entered into the trial was to be allocated
to the control group. (This might occur if the
researcher did not claim equipoise, and was
concerned that this patient received the best
possible treatment.)

11. • In that case, allocation would no longer be
random even if the allocation sequence itself
was truly random, because participants with
the most severe cases could be allocated only
to the treatment group.
• Consequently the groups would not differ
only by chance, and they would no longer be
‘comparable’.

12. • How can the allocation be concealed?
• Each participant’s allocation is placed in a
sealed envelope. The allocation schedule is
concealed from the researcher who enters
participants into the trial, and from potential
participants, so that neither the researcher
nor potential participant knows, at the time a
decision is made about participation in the
trial, which group the participant would
subsequently be allocated to.

13. • Then, when the researcher is satisfied that the
participant has met the criteria for
participation in the trial and the participant
has given informed consent to participate, the
envelope corresponding to that participant’s
number is opened and the allocation revealed.
• Once the envelope is opened, the participant
is considered to have entered the trial.
• This simple procedure ensures that allocation
is concealed.

14. • An alternative procedure involves holding the
allocation schedule off-site.
• Then, when the researcher is satisfied a patient is
eligible to participate in the trial and the patient
has given informed consent, the researcher
contacts the off-site holder of the allocation
schedule and asks for the allocation.
• Again, once the researcher is informed of the
allocation, the patient is considered to have
entered the trial. This procedure also ensures
concealment of allocation.

15. • There are other, less satisfactory, ways to conceal
• random allocation.
• Allocation could be concealed if, once the
researcher was satisfied that a patient was
eligible to enter a trial and had given informed
consent, allocation was determined by the toss of
a coin (‘heads’ = treatment group, ‘tails’ = control
group) or by the drawing of lots.
• The problem with coin-tossing and the drawing of
lots is that the process is easily corrupted.

16. • For example, if either the patient or the
researcher was unhappy with the coin toss or
the lot that was drawn, it might be tempting
to repeat the toss or draw lots again until the
preferred allocation was achieved.
• The benefit of using sealed envelopes or
contacting a central allocation registry is that
the randomization process can be audited,
and corruption of the allocation schedule is
more difficult.

17. • More often, trial reports do not explicitly state
that allocation was concealed, but they
describe methods such as the use of sealed
envelopes or contacting a central registry that
probably ensured concealment.

18. Was there complete or near-complete
follow-up?
• One of the difficulties is ensuring that the trial
protocol is adhered to.
• And one of the hardest parts of the trial protocol
to adhere to is the planned measurement of
outcomes (‘follow-up’)
• Most clinical trials involve interventions that are
implemented over days or weeks or months.
• Outcomes are usually assessed at the end of the
intervention, and they are often also assessed
one or several times after the intervention has
ceased.

19. • A problem that arises in most trials is that it is not
always possible to obtain outcome measures as
planned.
• Occasionally participants die.
• Others become too sick to measure, or they
move out of town, or go on long holidays
• Some may lose interest in participating in the
study or simply be too busy to attend for follow-
up appointments
• It may be impossible for the researchers to obtain
outcome measures from all participants as
planned

20. • This phenomenon of real-life clinical trials is
termed ‘loss to follow-up’.
• Subjects lost to follow-up are sometimes
called ‘dropouts’.
• Loss to follow-up would be of little concern if
it occurred at random.
• But in practice loss to follow up may be non-
random, and this can produce bias.
• Bias occurs when dropouts from one group
differ systematically from dropouts in the
other group.

21. • Randomization is undone.
• Estimates of the effect of treatment
potentially become contaminated by
differences between groups due to loss to
follow-up.

22. Example
• Imagine a hypothetical trial of treatment for cervical
headache.
• The trial compares the effect of six sessions of manual
therapy with a no-intervention control condition, and
outcomes in both groups are assessed 2 weeks after
randomization. Some participants in the control group may
experience little resolution of their symptoms.
• Understandably, these participants may become dissatisfied
with participation in the trial and may be reluctant to return
for outcome assessment after not having received any
intervention.
• The consequence is that there may be a tendency for those
participants in the control group with the worst outcomes to
be lost to follow-up, more so than in the intervention group

23. • In that case, estimates of the effects of
intervention (the difference between the
outcomes of intervention and control groups)
are likely to be biased and the treatment will
appear less effective than it really is.

24. • The potential for bias is low if few participants
drop out.
• When only a small proportion of participants
are lost to follow-up, the findings of the trial
can depend relatively little on the pattern of
loss to follow-up in such participants.
• The more participants lost to follow up, the
greater the potential for bias.

25. • How much loss to follow-up is required seriously
to threaten the validity of a study’s findings?
• Many statisticians would not be seriously
concerned with dropouts of as much as 10% of
the sample.
• On the other hand, if more than 20% of the
sample were lost to follow-up there would be
grounds for concern about the possibility of
serious bias.
• A rough rule of thumb might be that, if greater
than 15% of the sample is lost to follow-up then
the findings of the trial could be considered to be
in doubt.

26. • where loss to follow-up is much greater in one
group than in the other (clear evidence that
loss to follow-up is due to intervention), or
where loss to follow-up is clearly dependent
on the intervention, we may be suspicious of
the findings of trials that have loss to follow-
up of less than 15%.

27. • Some clinical trial reports clearly describe loss
to follow-up. It is particularly helpful when the
trial report provides a flow diagram in the
CONSORT (Consolidated Standards of
Reporting Trials) statement.

29. • More often, trial reports do not explicitly
supply data on loss to follow-up. In that case
the reader must calculate loss to follow-up
from the data that are supplied.
• Two pieces of information are required??
• it may be possible to find these data in tables
of results.
• The percentage lost to follow-up = 100 *
number lost to follow-up/number
randomized.

30. • A problem that is closely related to loss to
follow-up is the problem of protocol violation.
• Protocol violations occur when the trial is not
carried out as planned.
• In trials of physiotherapy interventions, the
most common protocol violation is the failure
of participants to receive the intended
intervention.

31. • For example, participants in a trial of exercise
may be allocated to an exercise group but may
fail to do their exercises, or fail to exercise
according to the protocol (this is sometimes
called ‘non-compliance’ or ‘nonadherence’),
or participants allocated to the control
condition may take up exercise.

32. • Other sorts of protocol violation occur when
participants who do not satisfy criteria for
inclusion in the trial are mistakenly admitted
to the trial and randomized to groups, or
when outcome measures cannot be taken at
the time that it was intended they be taken.

33. • How would we prefer data from clinical trials with
protocol violations to be analysed?
1. One alternative would be to discard data from
participants for whom there were protocol
violations.
2. Another unsatisfactory ‘solution’ is sometimes
applied when there has been non-compliance
with intervention.
• This is sometimes called a ‘per protocol’ analysis.
• Greater bias

34. • 3. The most satisfactory solution is the least
obvious one.
• It involves ignoring the protocol violations and
analysing the data of all participants in the
groups to which they were allocated.
• This is called ‘analysis by intention to treat’

35. Was there blinding to allocation
of patients and assessors?
• The placebo effect is demonstrated when
patients benefit from interventions that could
have no direct physiological effects, such as
detuned ultrasound
• The effects, it is thought, can be very large –
placebo can be more effective than many
established interventions.

36. • Blinding means that participants in
intervention and control groups do not know
which group they were allocated to.
• Blinded participants can only guess whether
they received the intervention or control
condition

37. • How is it possible to blind patients to
allocation?
• The general approach involves giving a ‘sham’
intervention to the control group. Sham
interventions are those that look, feel, sound,
smell and taste like the intervention but could
not affect the presumed mechanism of the
intervention.
• The clearest examples in physiotherapy come
from studies of electrotherapies

38. • Near-perfect shams used in clinical trials of
physiotherapy interventions include the use of
coloured light as sham low-level laser therapy.
• Quasi-sham intervention?

39. Assignment
• Differences between pragmatic and
explanatory trials
• Due Date: 7-11-2022 (Monday)

40. • Although the need for blinding of participants
is, therefore, arguable, there are compelling
reasons to want to see blinding of assessors in
randomized trials.
• Wherever possible, assessors (the people who
measure outcomes in clinical trials) should be
unaware, at the time they take each
measurement of outcome, whether the
measurement is being made on someone who
received the intervention or control condition.

41. • This is because blinding of assessors protects
against measurement bias.
• Fortunately, measurement bias is often easily
prevented by asking a blinded assessor to
measure outcomes.
• There is one circumstance that often prevents
the use of blind assessors: in many trials
outcomes are self-reported. In that case the
assessor is the participant, and assessors are
blinded only if participants are blinded.

42. • There are other participants in clinical trials
whom we would also like to be blind to
allocation.
• Physiotherapists
• Statistician who analyses the results
• Unfortunately, it is even harder to blind care
providers than it is to blind patients.

43. Summary