Research synthesis

040910 EBM Course, Oxford1
Research synthesis
Iain Chalmers
Editor, James Lind Library
(www.jameslindlibrary.org)

Research synthesis
is the application, in practice,
of the principle that
science is cumulative.

How has the science of
research synthesis evolved?

The evolution of
measures to reduce biases
in research synthesis

“As it is no easy matter to root out
prejudices, …. it became requisite to
exhibit a full and impartial view of what
had hitherto been published on the
scurvy, and that in a chronological
order, by which the sources of these
mistakes may be detected. Indeed,
before the subject could be set in a
clear and proper light, it was necessary
to remove a great deal of rubbish.”
James Lind, 1753

Joseph Goldberger, 1907
• identified 44 studies and provided comprehensive
references in a bibliography
• used a newly developed serum agglutination test to
separate reliable studies from those he considered
unreliable
• tabulated the raw data from 26 selected studies
• calculated the mean rate of bacteriuria from the
pooled data.
Winkelstein, 1998

More recently,
physicists and social scientists
led the way

Light RJ, ed. (1983).
Evaluation Studies Review Annual.
Beverly Hills, CA: SAGE Publications.
Light RJ, Pillemer DB (1984).
Summing up: the science of reviewing
research.
Cambridge, Mass.: Harvard University
Press.

Mulrow C (1987).
The medical review article: state
of the science. Ann Int Med
106:485-488.
"Current medical reviews do not
routinely use scientific methods
to identify, assess, and
synthesize information."

The evolution of
statistical methods for
reducing the play of chance
in research synthesis -
meta-analysis

Ronald Fisher (1932)
“... although few or [no statistical
tests] can be claimed individually as
significant, yet the aggregate gives
an impression that the probabilities
are lower than would have been
obtained by chance.”

In 1976, Gene Glass, an
American social scientist coined
the term ‘meta-analysis’ to
refer to:
“the statistical analysis of a
large collection of analysis
results from individual studies
for purposes of integrating the
findings”

Rosenthal and Rubin (1978); Smith and Glass (1977);
Glass and Smith (1979); Hunter et al. (1979).
Rosenthal R (1984). Meta-Analytic Procedures for
Social Research. Newbury Park, CA: Sage
Publications.
Hedges L, Olkin I (1985). Statistical Procedures for
Meta-Analysis. Orlando: Academic Press.
Yusuf S, Peto R, Lewis J, Collins R, Sleight P (1985).
Beta blockade during and after myocardial infarction:
An overview of the randomised trials. Prog Cardiovasc
Dis XXVII (5), 336-371.

By the end of the 20th
century, research synthesis
was beginning to be taken
more seriously

“There are research syntheses in such
diverse topics as advertising,
agriculture, archaeology, astronomy,
biology, chemistry, criminology,
ecology, education, entomology, law,
manufacturing, parapsychology,
psychology, public policy, zoology and
even of eyewitness accounts of the
Indian rope trick.”
Mark Petticrew, BMJ 2001.

As science is cumulative,
scientists should cumulate
scientifically!

Luc de Clapiers Vauvenarques, 1715-
47 (Réflexions et Maximes).
"Il est plus aisé de dire des choses
nouvelles que de concilier celles qui
ont étés dites“.
‘It is easier to say something new than to
reconcile things that have already been said.’
or
‘Synthesis of ideas is harder than merely
proposing a new idea.’

Lord Rayleigh, 1884
“If, as is sometimes supposed, science
consisted in nothing but the laborious
accumulation of facts, it would soon come to
a standstill, crushed, as it were, under its
own weight…The work which deserves, but I
am afraid does not always receive, the most
credit is that in which discovery and
explanation go hand in hand, in which not
only are new facts presented, but their
relation to old ones is pointed out.”

Hill AB. Reasons for writing.
BMJ 1965;4:870.
People reading research reports want
investigators to answer four questions:
Why did you start?
What did you do?
What answer did you get?
What does it mean anyway?”

Premise
The results of a particular
research study cannot be
interpreted with any confidence
unless they have been
considered, systematically,
together with the results of
other studies addressing the
same or similar questions.

How well is this premise reflected
in papers published in major
general medical journals?

Classification of Discussion sections in
RCT reports published in Ann Int Med,
BMJ, JAMA, Lancet, and N Eng J Med
19No apparent systematic attempt to set
new results in context of other trials
4Discussed a previous review but did not
attempt to integrate new results
2Contained an updated systematic
review integrating the new results
1First trial addressing the question
May ’01
n=33
May ’97
n=26

Classification of Discussion sections in
RCT reports published in Ann Int Med,
BMJ, JAMA, Lancet, and N Eng J Med
2719No apparent systematic attempt to set
new results in context of other trials
34Discussed a previous review but did not
attempt to integrate new results
02Contained an updated systematic
review integrating the new results
31First trial addressing the question
May ’01
n=33
May ’97
n=26

A contemporary example of the
consequences of not taking
research synthesis seriously
Drugs to prevent heart rhythm
abnormalities (arrhythmias)
after heart attacks

The theory:
Patients with arrhythmias are at
increased risk of early death after
heart attack
Anti-arrhythmic drugs reduce
arrhythmias after heart attack
These drugs should reduce early
death after heart attack

The evidence:
A 1983 systematic review of 14
randomized controlled trials of anti-
arrhythmic drugs in heart attack
“The theoretical potential for a preventive
or prophylactic effect of antiarrhythmic
drugs …..in the treatment of coronary
patients with ventricular arrhythmias has
not been realized.”
Furberg, 1983

A 1993 systematic review of 51
randomized trials of anti-arrhythmic
drugs in heart attack
660 deaths among 11,712 patients
allocated drugs
571 deaths among 11,517 patients
allocated to control
Teo et al. JAMA 1993.

The consequence:
At the peak of their use in the late
1980s, it has been estimated that
anti-arrhythmic drugs were causing
every year comparable numbers of
deaths to the total number of
Americans who died in the Vietnam
war (Moore 1995).

The discovery that these drugs
are lethal could have been made
a decade earlier if the Discussion
sections in each report of a new trial
had set the new results in the context
of a systematic review of the results
of all previous trials – in other words,
if scientists had cumulated
evidence scientifically.

The human costs of failing to
recognise the importance of
cumulating evidence scientifically
“Advice on some life-saving therapies
has been delayed for more than a
decade, while other treatments have
been recommended long after
controlled research has shown them
to be harmful.”
Antman et al. 1992
Antman et al. JAMA 1992;268:240-8.

Setting the results of new results
in the context of a systematic
review of the results of all other
relevant studies would become
straightforward if systematic
reviews were always done before
embarking on new research.

Would any of you have agreed
to participate in a placebo
controlled trial of prophylactic
antibiotics for colorectal
surgery after 1975?

Reduction of perioperative deaths by
antibiotic prophylaxis for colorectal surgery

Would you ever have put
babies to sleep on their
tummies?

Would a healthy young woman
volunteer at Johns Hopkins have
died after inhaling hexamethonium
if the investigators had done a
systematic review of relevant
evidence?
Clark O, Clark L, Djulbegovic B. Is clinical
research still too haphazard?
Lancet 2001;358:1648.

Would 7000 stroke patients have been
persuaded to participate in controlled
trials of nimodipine if researchers had
done systematic reviews of the relevant
animal experiments?
Johanna Horn et al (2001).
Nimodipine in animal model experiments of
focal cerebral ischaemia.
Stroke 32:2433-2438.

Pound P, Ebrahim S, Sandercock P,
Bracken MB, Roberts I. The Reviewing
Animal Trials Systematically (RATS)
Group.
Where is the evidence that animal
research benefits humans?
BMJ 2004;328:514-17 (28 February)

Principles of good practice
All new research studies should be:
• designed in the light of
scientifically-defensible syntheses of
relevant existing research evidence
• reported using the new evidence to
update these research syntheses
(thus making clear what contribution
the new study has made to the total
evidence)

An important innovation by the BMJ

The main obstacles to applying
these principles in practice are
psychological and social, not
practical.
The practical challenge is being
met by very rapid evolution and
application of the science of
research synthesis and
electronic publishing.

Can the public rely on scientists
to use research synthesis
to ensure that limited resources
for research are used more
efficiently and ethically?

Needed:
academic recognition that
research synthesis is
research

“….. The work which deserves, but I
am afraid does not always receive,
the most credit is that in which
discovery and explanation go hand
in hand, in which not only are new
facts presented, but their
relation to old ones is pointed
out.”
Lord Rayleigh 1884

If academia cannot be relied
on to ensure that new
research projects begin and
end with syntheses of the
results of other relevant
studies, who will protect the
public from the adverse
consequences of current
scientific indiscipline?

Danish Research Ethics Committee
System, 1997
“For a research ethical evaluation of
scientific experiments involving man it is
crucial that all relevant literature has been
reviewed by the research group before
submission……
....This will be a precondition when the
evaluating committee is judging the
originality of the project and, for example,
the permissibility of using placebo and not
an already known treatment in a control
group.”

UK Medical Research Council
requirements of people applying for
support for new controlled trials, 1999.
“Give references to any relevant systematic
review(s)* and discuss the need for your trial
in the light of the(se) review(s).”
“If you believe that no relevant previous trials
have been done, give details of your search
strategy for existing trials.”
“* For definition of a systematic review, see
Oxman, AD. Checklists for review articles, BMJ, 1994;
309;648-51.”

“The European Science Foundation (2001)
endorses the view that embarking on new research
without first preparing systematic reviews of relevant
existing evidence is indefensible on scientific and
ethical grounds, and recommends that its Member
Organisations should:
• Require applicants for support of new trials to refer
to scientifically defensible reviews of relevant existing
trials, or demonstrate that no other relevant trials
exist.
• Require data monitoring committees to take account
of new evidence accumulating in updated systematic
reviews.”

In conclusion
Science is cumulative, and
scientists must cumulate
scientifically!
As the main funder of science, the public has a
right to expect that this will be reflected more
effectively in the way that science is conducted
and reported.
It remains unclear how effectively academia
will respond to this fundamental challenge to
its traditional ways of working.

This talk has been based in part on
material published in:
Chalmers I, Hedges L, Cooper H.
A brief history of research synthesis
Evaluation and the Health Professions
2002;25:12-37.

Francis Galton, 1901
"I have begun to think that no one
ought to publish biometric results,
without lodging a well-arranged and
well-bound copy of his data in some
place where it should be accessible,
under reasonable restrictions, to
those who desire to verify his work.”

Research synthesis

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

Similar to Research synthesis

Similar to Research synthesis (20)

Recently uploaded

Recently uploaded (20)

Research synthesis