MR FIT (Multiple Risk Factor Intervention Trial) (1972)
CRASH Trial (1992)
EC/IC (Extra-cranial/Intra-cranial bypass)
Macular Photocoagulation Study (1991)
Parkinson’s stem cell trial w/sham surgery.
NETT (National Emphysema Treatment Trial) (2004)
Poliomyelitis is a highly seasonal viral disease. 95% infections are asymptomatic. Small percentage cause permanent paralysis.
Polio peaked in 1952 with 58,000 cases (~37 per 100,000). Hardest struck were middle and upper class children.
The National Foundation for Infantile Paralysis (NFIP), or March of Dimes, mounted public donation campaign to raise funds to develop an effective vaccine.
Small field trial of the formalin-inactivated Salk vaccine (N=700) showed it to be safe and potent.
Salk vaccine trial timeline
January, 1953 : NFIP Immunization committee considers Salk preliminary data and whether trial is needed. Salk wants to do more development and safety studies
May 1953 : O’Connor pushes trials and convenes committee w/o virologists to decide on design. Intense discussion of need for randomization, and nature of control. O’Connor and Salk opposed randomization, key scientists (Joseph Bell) supportive. Observed control preferred.
June 1953 : Tommy Francis, PhD, U Mich, chosen to direct trial. Insists on randomization with saline injected placebo.
November, 1953 : States contacted re participation. Several object to randomization, others object to observed controls. Negotiation starts with health authorities in every state.
January 11, 1954 : Final decision on trial design. 11 states agree to randomization, 33 states use observed controls.
Basil O’Connor, head of NFIP
Tommy Francis, Trial director
15,000 public schools
60,000 MDs, nurses, public health officials
>200,000 lay volunteers
Average public donation
Debate over randomization
Observational design supported by Basil O’Connor, head of NFIP. Randomization pushed by some scientists, vehemently opposed by Salk:
“ . . .if we are aware of the fact that the presence of antibody is effective in preventing the experimental disease in animals and in man, then what moral justification can there be for intentionally injecting children with salt solution or some other placebo for the purpose of determining whether or not a procedure that produces antibody formation is effective. . .I would feel that every child who is injected with a placebo and becomes paralyzed will do so at my hands. . .The use of a placebo control, I am afraid, is a fetish of orthodoxy and would serve to create a ‘beautiful epidemiologic’ experiment but would make the humanitarian shudder and would make Hippocrates turn over in his grave”.
Basil O’Connor, Fall 1953, during trial design debate:
“ I have just figured out that during this coming summer thirty or forty thousand children will get polio. About fifteen thousand of them will be paralyzed and more than a thousand will die. If we have the capacity to prevent this, we have a social responsibility here that none of you have been talking about. Let me remind you that we are supported by the people and it is our duty to save lives, no matter how many difficulties may be involved.”
Tom Rivers, MD, Chair of NFIP Immunization committee:
“ We all believed that the Foundation had to put on a field trial in the spring of 1954; if it didn’t, the lid would be off and the following year everybody and his aunt would be trying out their own vaccines unless the Public Health Service could put a muzzle on them.”
Thousands of informational meetings held with health officials, parents and children across the country.
NYC health commissioner, Leona Baumgartner, recalling talking to children at P.S. 61 in Manhattan’s Lower East Side about the placebo controls:
“ Those youngsters [gave] as good a description of a controlled experiment as I’ve ever heard. I was very excited, because it seemed to me that if you could teach a generation of kids about what a controlled experiment was, and about what science really was, this was a plus value regardless of whether the vaccine was any good or not.”
Salk vaccine trial timeline
March, 1954 : Viral particles found in 4/6 vaccine lots. Winchell broadcasts warning against participation caused withdrawal of Wisconsin and 150K participants.
April 25, 1954 : Meeting to decide on whether to go forward on trial, with several committee members unable to sleep.
April 26, 1954 : Innoculations start, ending June 15.
Two different studies of school children, aged 6-9 years:
Randomized, double-masked, placebo-controlled trial: (455,474 1 st -3 rd graders from 11 states)
Observational Study (567,210 participants from 33 states)
2 nd graders received vaccine
1 st and 3 rd graders served as controls
On the parental consent form, the phrase ‘I give my permission’ was changed to ‘I hereby request,’ implying that not every child would be fortunate enough to be picked.
Due to the difficulty of diagnosing nonparalytic cases of polio, clinical evaluations initially performed by local physicians and physical therapists had to be reassessed by an advisory committee in order to standardize clinical definitions .
Approximately 600K children did not show up, and 200K did not have signed consents.
The evaluation center was under intense pressure to release the findings of the trial by the next polio season, shortening the timeline available for data analysis. Although the results were released to the public in the spring of 1955, the full study report did not appear for another 2 years.
April 12, 1955 : Results announced. Vaccine licensed by HHS 2 hrs later, shipments started 4 hrs hence.
April 13, 1955 : Nationwide immunization commences with stockpiled vaccine.
Total cases reported differed significantly between vaccinated and controls (respective attack rates were 41 vs 81, P<0.001).
Difference in cases of paralytic polio was marked (33 among vaccinated vs 115 among controls (attack rates of 16 vs 57; P<0.001)
Also measured significant difference in incidence of paralytic polio with 38 cases among vaccinated vs 330 cases among controls (attack rates were 17 vs 46; P<0.001).
P olio incidence in the randomized study was similar to that in the observational study and neither study showed any appreciable difference between the two treatment arms in incidence of nonparalytic polio.
Vastly accelerated the vaccine development process and undoubtedly saved hundreds of lives and thousands of paralytic cases.
Enhanced public support for medical research at the time and possibly the future (through the involved children.)
Improved adherence to vaccine manufacturing protocols and safety testing guidelines.
New understanding of poliovirus epidemiology and serology helped shape future vaccine studies.
The role in the Cutter incident of the recently formed EIS at the CDC established its reputation.
The process by which each community decided the terms of its participation gives new meaning to the term “consent”.
Lessons from the Salk trial (T. Burke, 2004)
VERY large trials are possible.
Committed, effective leadership is essential
Precise case definitions are important
Clinical endpoints are key; correlates of immunity don’t suffice
Serious potential safety problems may not be detected in even a VERY large trial
A nongovernmental organization can do the job,
A successful phase III trial is just a beginning.
Sometimes observational trials must do; you cannot foist randomization on unwilling participants.
Meaningful public (aka consumer) involvement led to rapid public acceptance of both risks and benefits.
The ECMO controversy When tough cases don’t make good trials
The ECMO Circuit
ECMO had not improved survival in adults with acute respiratory failure, but it was hypothesized that it might be more effective in treating neonatal respiratory failure.
Robert Bartlett began treating critically ill infants with ECMO. Encouraging results (6/16 survived)
By the 1980s, Bartlett had achieved a 75% survival in patients judged to have a 5% survival when managed by conventional therapy.
Bartlett: Play-the-Winner Design ECMO Survived CMT Died 10 ECMO: survived 1 CMT: died
Increasing demand for more rigorous evaluation of ECMO in neonates:
Bartlett conducted a trial using a “play-the-winner” design
1 patient received conventional therapy and died (smallest, most premature, lightest)
11 patients received ECMO and survived
Article published in same issue (Wung et al. Pediatrics, 1985) reported 100% survival in a cohort of similar neonates treated with new low intensity ventilation.
The ECMO Controversy “ Children’s Hospital made a serious error .” - Office for Protection from Research Risks, NIH
The Harvard Neonatal ECMO Trial Randomized newborns with PPHN to conventional therapy versus ECMO Conventional Therapy NICU: 7th Floor Neonatologists No patients had ever been offered ECMO Anti-ECMO ECMO PICU: 5th Floor Anesthesiologists & Surgeons Already had experience with ECMO for newborns with CDH Pro-ECMO
The Harvard Neonatal ECMO Trial: Randomized Consent
RANDOMIZE Do not seek consent Seek consent for ECMO CMT CMT ECMO Yes No
Phase I: 19 children enrolled and randomized
Conventional Medical Therapy Group: 6/10 survived
ECMO Group: 9/9 survived
Phase II: 20 patients assigned to the ECMO group
A Harvard study on newborns draws fire;
Doctors faulted for limiting life-saving treatment
August 7, 1989
Richard A. Knox, Boston Globe
A Harvard University study involving mortally ill newborns is being challenged as unethical in a debate that raises important questions about how to do research on promising new therapies.
The Response to the ECMO Trial The hospital IRB “made decisions that rightfully belonged to the parents. They really blew it.” Charles McCarthy, Director of OPRR The NIH Office for Protection from Research Risks (OPRR) reprimanded the hospital: The doctors “were doing exactly what physicians did before we had a doctrine of informed consent - making decisions for parents.” George Annas, Boston University
Donald A. Berry. : Ethics and ECMO
Robert E. Kass, Joel B. Greenhouse. A Bayesian Perspective.
Colin B. Begg.
Peter Armitage, D. Stephen Coad.
D. Y. Lin, L. J. Wei.
Richard G. Cornell.
Janis Hardwick. Recent Progress in Clinical Trial Designs that Adapt for Ethical Purposes.
James H. Ware. Rejoinder
The Problem of “Randomized Consent” "It's clear to me they did not ask consent because it would be hard to get a control group otherwise. Properly informed parents would say 'No thank you.’ ” - Richard M. Royall, Johns Hopkins School of Public Health “ [Seeking consent from those randomized to the conventional medical treatment] would impose unacceptable psychological burdens on the parents of children not included in the ECMO arm of the study.” - Dr. Peter H. Wolff , Chair of Children’s Hospital IRB
The Problem of Adaptive Randomization "The clear expectation was that more patients would die on conventional therapy. So the question is whether having an excess of deaths balances the worth of information gained . Since I believe such information is available without randomizing, my answer is a resounding no.” - Donald A. Berry, University of Minnesota "The Harvard study was informed by the slightly hysterical view that we must immediately stop a study as soon as we have an idea which treatment might be better. If we're not careful, we will soon have a system in which we can establish nothing. ” - Paul Meier, University of Chicago
Are RCTs the only way to learn?
“ The brilliant success of the RCT has now become a form of intellectual tyranny” Freireich
“ We should not proceed on the fallacious assumption that where there is no randomization, there is no truth.” Royall
Meinert on ECMO (Pediatrics, 1990)
I have reservations about any trial that is designed to detect a treatment difference of “miracle” proportions, such as the one postulated in this trial. Generally, use of such a difference for sample size rationale means either that the investigators “shopped” for a set of specifications to yield a desired sample size or that the trial is being done to demonstrate or “prove” the value of a treatment.
… The desire to shield patients from upsetting discussions by using nontraditional approaches to consent is a natural one, but its pursuit should be reserved for settings in which there is a legitimate state of equipoise.
Chalmers: “A Belated Randomized Trial”
“ A plague on all houses except those who reported and conducted [this] study…”
All new treatments should be administered under a randomized protocol.
“ IRBs should recognize the ethical and scientific deficiencies of uncontrolled pilot trials and innovative practice. They should demand… that physicians provide evidence for all that they do.”
“ Professional ethicists should get off the back of people performing good clinical research…In the absence of evidence of efficacy, both the withholding and delivery of a new treatment are equally less ethical than [RCTs].”
Payers should only pay for innovate therapies under protocol.
Trial registration would have to start. (KD cited)
Data monitoring and peer review boards would have to decide when to stop.
Electronic communication would have to be devised to keep track of rapid changes.
A portion of the money spent for care would be spent on experimentation.
Lantos and Fader, NEJM, 1991
NIH Consensus Conference
Report of the Workshop on
Diffusion of ECMO Technology
Extracorporeal Membrane Oxygenation
May 31,1990—June 1,1990
National Institute of Child Health and Human Development, NIH
Office of Medical Applications of Research, NIH
National Institute of Neurological Disorders and Stroke, NIH
National Heart, Lung, and Blood Institute, NIH
Agency for Health Care Policy and Research
Federal Drug Administration
NIH Consensus Recommendations
Prevention and Therapeutic Alternatives
Alternative therapies and prevention have not been adequately explored. There is a serious lack of knowledge regarding epidemiologic factors that influence the requirement for ECMO therapy.
The actual incidence of conditions treated with ECMO.. is not well documented. …and the relative effectiveness of alternatives versus ECMO is unclear. …
Education and prevention strategies should be developed …. It is not yet known whether the use of ECMO can be reduced through education …..
The increasing use of ECMO especially in new patient populations creates an urgent need for further research. Clinical studies are needed to determine when ECMO is the most appropriate treatment alternative and what technical improvements are safe and feasible. The short-term and long-term effects of ECMO on the nervous system, pulmonary system, cardiac system, and the blood in all age groups must be further defined. Quality of life as well as specific biologic parameters should be studied.
The UK Neonatal ECMO Trial
1993-1995: 185 neonates randomized to ECMO vs CMT
Trial stopped early by DSMB,
ECMO survival 60/93 = 65%
CMT survival 38/92 = 41%, p<0.0005
The Problem of Adaptive Randomization
Infants assigned to ECMO or conventional medical therapy with 50:50 randomization
Randomized assignment continues until 4 deaths occur in either treatment group
Infants assigned to the “winning” treatment group until 4 deaths or 28 successes occur
Null hypothesis of 80% mortality in both groups
Alternative hypothesis of 80% mortality in conventional treatment group and reduction to 20% mortality in the ECMO group
alpha = 0.05, Power= 0.74
Extracorporeal Life Support:
History and New Directions
Robert H. Bartlett, MD
“ The use of ECLS peaked in 1992 and has been steadily decreasing since that time. The primary reasons are identification and avoidance of ventilator-induced lung injury and the use of inhaled nitric oxide as a pulmonary vaso dilator. In our experience, 40% of the patients who would have required support with ECMO in 1992 are now successfully treated with pressure-controlled (low stretch) ventilation and inhaled nitric oxide using standard or oscillation ventilator techniques.”
How to evaluate promising new technology / interventions whose development produces results that disturb equipoise.
How to create a system – both professional and economic- that rewards evaluation as much as innovation.
Whether RCTs are necessary for breakthrough technologies.
How to evaluate interventions with possible large short term benefit and similar long term harm.
Damage can be done to the RCT enterprise if it is imposed on subjects, or a society, who are unwilling to accept it, or don’t understand it.
Women’s Health Initiative 1993 - 2002
Previous systematic review showed increased risk of cardiovascular events with HRT
Several top women epidemiologists complained about the size and complexity of the study
Huge budget ($625 million) was criticized as being unnecessary, that we were certain enough of the answer.
HR and 95% CI
CVD : 1.29 (1.02-1.63)
Stroke: 1.41 (1.07-1.85)
VTE: 2.11 (1.58-2.82)
Invasive breast cancer: 1.26 (1.00-1.59)
Colorectal cancer: 0.63 (0.43-0.92)
Hip fractures: 0.66 (0.45-0.98)
Global index: 1.15 (1.03-1.28)
Results were shocking and extremely surprising
Significant reduction in initiation as well as a marked discontinuation of HRT
Reinforced the importance of conducting large scale RCTs
Endpoints in WHI trial 06/06/09
WHI stopping rule
Test CHD against an O-F upper 2.5%ile
If benefit for CHD, test Global index (any endpoint) against 5% upper boundary.
Lower 5% O-F Boundary for any major AE.
AND >1 SE in harm direction for Global index.
WHI DSMB recommendation 06/06/09
NIH Letter #1 06/06/09
Issues in stopping the ERT trial
No stopping guidelines were crossed
No adverse CHD effect, beneficial (??) breast ca effect.
Global index showed equal risk, benefit.
Strong evidence for high stroke risk.
WHIMS trial showed slight (unexpected) increase in dementia in both trials.
No CHD benefit.
DSMB vote: May, 2003
“ Each one of us realized our vote was on a knife edge.”
5-4 vote to continue.
Degrees of confidence (0-100) were 45-55.
NIH convenes ad-hoc panel. 2 members of original DSMB observe as non-voting members.
July, 2003 : New panel votes to continue.
November, 2003 : DSMB meets again, augmented w/2 members of ad hoc panel dementia experts. Decide to continue trial w/letter to subjects explaining stroke risk and breast cancer benefit.
Unable to craft letter.
Re-vote: 5-4 to continue – different members in majority.
Recommendation to continue sent to NHLBI.
NHLBI stops trial on 2/04.
Importance of board independence.
NIH and CC both had structural conflicts.
NIH replaced DSMB members and required permission for additional analyses, both inimical to optimal board function.
NIH reps sat in DSMB closed sessions.
Monitor for both benefit and harm regardless of prior.
Need flexibility for weighting harms when monitoring multiple endpoints in long-term prevention trials. Is obligation to participants different in prevention than treatment trials?
“ JW adds special thanks to William Harlan who prevailed on her to join the DSMB when she knew that the trial was going to show that hormone replacement therapy reduced coronary disease and therefore that serving on this committee would be boring.”
Comparison of Total Mastectomy and Lumpectomy With or Without Radiation (NSABP B-06) 1976-1984 National Surgical Adjuvant Breast and Bowel Project Bernard Fisher, MD Chairman and Scientific Director, NSABP Carol K. Redmond, ScD Director, NSABP Biostatistics Center
Radical mastectomy—removal of the affected breast, underarm lymph nodes and both underlying chest muscles (en bloc dissection)—was the standard of care for breast cancer.
Two divergent hypotheses existed regarding tumor biology. Most believed that cancer was a local/regional disease spread by the lymph system to distant sites. Others proposed that breast cancer was a systemic disease better treated by systemic therapy.
During the 1970s, social pressure/advocacy groups were starting to raise awareness of breast cancer, advocating research and promoting less disfiguring surgery.
The NSABP—a clinical trials cooperative group supported by the NCI—had previously found that patients treated by total mastectomy without axillary-node dissection and pectoral muscle removal were at no higher risk of distant disease or death than those undergoing a Halstead operation (radical mastectomy).
This prospective, randomized, clinical trial wished to determine:
The effectiveness of lumpectomy for breast preservation
The effect of post surgery radiation therapy on ipsilateral breast tumor incidence reduction
The clinical importance of tumor multicentricity
In 1976, 2163 women with primary operable breast cancer and with tumors ≤ 4cm were randomized into:
Lumpectomy with radiation
All patients underwent an axillary lymph node dissection
Standard randomized treatment assignments (stratified by participating institution) were changed to a Zelen“prerandomization” design during the course of the trial.
Eight percent of women (174/2163) did not accept randomly assigned treatment but were followed.
Due to low enrollment rates, the standard randomization protocol (stratified by participating institution) was changed to a Zelen design (prerandomization) during the course of the trial. This change was accompanied by a six-fold increase in enrollment rates. This practice provoked considerable ethical debate.
At five years, 10% of patients initially treated with lumpectomy were found to have a tumor at the margins of resected specimens and therefore subsequently underwent total mastectomy. The incidence of positive margins was similar in both lumpectomy groups. Recurrences at the margin were positively related to the presence, number of positive nodes, and centrally located tumors.
Treatment by lumpectomy with or without breast irradiation resulted in disease-free, distant-disease-free and overall survival at five years that was no different than breast removal.
Distant-disease-free (p not significant) and disease-free survival (p=0.04) were higher in patients undergoing lumpectomy plus radiation as compared to the total mastectomy group.
Among women treated by lumpectomy, the addition of radiation therapy significantly decreased tumor recurrence (P<0.001) in the ipsilateral breast . The advantage of irradiation therapy was observed in both subjects with negative and positive nodes.
Coupling breast irradiation with lumpectomy also improved disease-free survival (p=0.02) when compared to lumpectomy alone. However, no significant differences between the two groups were found with respect to distant-disease-free or overall survival.
The fewest treatment failures were observed in the group receiving lumpectomy + radiation therapy. More than half of the first-reported treatment failures in all three treatment groups were due to distant recurrences. Second cancers and deaths from other causes were distributed equally across treatment groups.
Findings along with B-04 trial repudiated the Halstedian principles of breast cancer treatment and provided support for the theory that breast cancer is a systemic disease.
Lumpectomy was shown to be an acceptable alternative to mastectomy for women with Stage I and II breast cancer. Radiation therapy was shown to be beneficial in reducing local recurrence of tumor in lumpectomy patients..
At St. Luc Hospital, a Dr. Poisson admitted fraud after a routine audit revealed that records had been altered. He stated that he altered records to ease the entry requirements because he felt the treatment provided by the NSABP protocol was beneficial to potential participants in the study.
Consequences of “fraud”
Fisher: “All Hell Broke Loose”
Fisher and Carol Redmond dismissed from NSABP
2 congressional hearings, one under John Dingell
New Clinical Trials Monitoring Branch at NCI
1993 Federal Register notice: Roger Poisson guilty of research misconduct
Poisson falsified or fabricated 111 separate instances of data in 14 NSABP clinical trials (<0.5% of total).
Chicago Tribune Front Page Story March 13, 1994: “Fraud in Breast Cancer Study: Doctor Lied on Data for Decade”
150 articles tagged “scientific misconduct” by NLM
In 1997, Bernard Fisher received an apology from the University of Pittsburgh, his rank as Distinguished University Service Professor reinstated, NCI and U. Pitt. restrictions lifted, and $2.75 million for wrongful punishment.
NCI said: "Through his role as the scientific leader of the National Surgical Adjuvant Breast and Bowel Project (NSABP), [Fisher] has not only changed the way breast cancer is treated, but enlightened medical science to view breast cancer as not just a tumor confined to the breast, but as a systemic disease requiring more than surgical intervention,"
This trial raised ethical issues regarding both informed consent, fraud, and the appropriateness of actions against the investigators after the fraud had been detected. Thus, its impact and legacy was broad, affecting the understanding of biology, clinical practice, the ethics of randomization, the impact and handling of research misconduct, and the role of regulatory agencies in clinical trial oversight.
John Bailar III
Sir Iain Chalmers
Sir Richard Doll
P. J. Devereaux
Barbara S. Hawkins
Curtis L. Meinert
David L. Sackett
Jonathan M. Samet
Jan P. Vandenbroucke
ImpACT Staff Emily Evans Gila Neta* Alison Brown Sara Lowther* Olaide Odelola
Nyasha Bakare (2006)
Juliana Cuervo (2007)
Lara Devgan (2006-2007)
Ishveena Duggal (2004)
Julia Gage (2005-2006)
Emily Henkle (2006)
Jay Herson (2005)
David Majure (2006-2007)
Tram Lam (2005-2006)
Sara Lowther (2005-2007)
Nancy Maldeis (2004-2005)
Todd Meyers (2005)
Kamal Patel (2006)
Sarah Post (2003)
Samara Rifkin (2006)
Jordana Rothschild (2004)
Lauren Singer (2004)
Ellen Wasserman (2003-2005)
Lashawn Worsley (2005)
Project leaders Harry Marks Karen Robinson
Final thoughts on the need for RCTs
The fatal tendency of mankind to leave off thinking about a thing when it is no longer doubtful is the cause of half their errors.
John Stuart Mill ( On Liberty ,1859)
In theory there is no difference between theory and practice. In practice there is.