Institutions of Objectivity in Clinical CancerTrials: A History and Sociology of Data Monitoring Committees in Clinical Cancer Trials
Peter Keating & Alberto Cambrosio
Paper presented 03-12-08
A word from our sponsors: CIHR, FQRSC and SSHRC funded research for this presentation.
I. The emergence and development of DMCS
what are DMCs?
DMCs consist of group of experts who periodically review accumulating clinical trial data for the purpose of monitoring safety and treatment efficacy
After review, DMCs issue a stop or go judgment
DMCs have raised epistemic and governance questions
‘ There is irony in that, in the case of trials where we experiment on human beings, those who do so cannot be ‘trusted’ to see the data they generate while they so experiment. Surely clinical trials must be one of the few fields of scientific inquiry in an open society where researchers are so regarded’. [Meinert 1998a: 542] ‘ Imagine a scientific methodology that bases its experimental technique on a technology that alters sample size, modifies the study design, and provides guidance to the interpretation of data, but which the methods sections of papers rarely mention and to which papers rarely allude at all. … In fact, participation on a data monitoring committee requires a vow of silence , a silence that often remains unbroken for years’. [Wittes 1993: 419]
DMCs in the United States: an overview
less than 50% of clinical trials have a DMC
100% of NIH-sponsored multi-institutional trials have a DMC
42% of industry-sponsored trials have DMCs
90% of trials with DMCs are multi-centered
most DMCs involve Phase II and Phase III trials
trials with DMCs are stopped a significantly higher number of times than trials without DMCs
DMCs are deployed extensively in clinical cancer trials
participant history 1: the administrative origin
DMCs emerged in the late 1960s because of an administrative innovation at the National Heart Institute
the 1967 Greenberg Report created the NIH model for clinical trials (first published in 1988: Controlled Clincial Trials )
used in a number of clinical trials in the 1970s (but not in cancer)
the NIH model is the source of present-day DMCs
the original Greenberg diagram a recent representation of the NIH model The Executive Committee has “the responsibility … to review the data analysis coming from the Coordinating Center and to prepare frequent reports to the participants, as well as annual reports to the National Heart Institute”
participant history 2: the statistician’s view
DMCs emerged as the result of innovations in the field of statistics
Armitage and McPherson ( NEJM 1973)
in 1969 Armitage and McPherson observed that repeated significance testing would lead to the observation of false positive results; they thus invented the first ‘stopping rules’ ( Armitage, McPherson and Rowe, J.Roy. Stat. Soc. , 1969 )
nonetheless, informal rules of thumb already functioned as ‘stopping rules’ in the 1970s
‘ In the early days of ECOG ... we would look at the data formally every six months because we prepared … the agendas for a group. … So we were looking every six months, but we knew even as the theory was just developing, every statistician, every trained statistician knows that if you look too often you might stop a trial for which there is no difference; you stop it by chance alone. … And those were rules of thumb, quite useful rules and there was a very early paper by Haybittle  that made that a bit more formal, that said you know, any decent rule of thumb in sequential design says: “look as often as you want”, well, not exactly but: “look fairly often, and only stop the trial or think about stopping the trial if you see a difference that’s more than 3 standard deviations above the null, or below the null”’. [Interview with David Harrington, Boston, 22 April 2005]
in the mid 1970s Stuart Pocock, an ECOG statistician, generated the first group sequential designs for clinical trials
further innovations in the early 1980s in group sequential methods made stopping rules useful for large trials
Sequential (2X2) medical trial: Hodgkin’s Disease, Stanford Medical Center, 1967 The group methods replaced the 2X2 sequential methods first developped and used in the 1960s
a simple calculation: if during interim analysis the trial crosses a preset boundary then it must be stopped
statistics and the ‘mechanical’ mandate of DMCs
BUT: ‘the process process of making such judgments is rarely straightforward; the observation of a low p-value or a sequential boundary crossed represents only the beginning of a process, not the end’. [Ellenberger et al. 2002]
the statistical procedures developed to monitor trials must be interpreted to produce decisions
results of a trial must be ‘statistically’ convincing in order to stop a trial for reasons of ‘early success’ ... but convincing to whom: researchers or practitioners?
‘ I was involved in a trial in which there were no boundaries set in advance and the co-ordinating centre said that if we had used Pocock stopping rules we would have just barely crossed, if we used O’Brien-Fleming stopping rules we wouldn’t have, if we had used stochastic curtailing... and it became completely meaningless’. [Michael Proschan, 1994]
In the mid 1990s, the proliferation of rules sometimes raised issues of choice: which rules should apply?
what do the participant histories omit?
between the late 1960s and the advent of DMCs (in cancer) in the mid 1980s, clinical trials changed in scale and scope to become a new ‘style of practice’
this transformation entailed the emergence of full-time clinical cancer trialists interested in the success and objectivity of trials
cancer clinical trials: a new style of practice
Style of practice = distinctive configuration of institutions, epistemic concerns and materials that generates specific ways of identifying and investigating research questions, of producing and assessing results, and of regulating these activities [Hacking]
the elements of style in clinical cancer reseach
a continuum of activities: from screening to treatment through research, testing and training
performed by new collective entities, the cooperative oncology groups
a distributed, multi-center network
held together AND performed by and through protocols
entrenched in new institutions: data centers, data managers, DMCs, etc.
The Groups are often international in scope
SWOG-S0106 includes 17 sites in Canada and 8 sites in Sweden
the data center
Establishment of centralized data centers in 1970 in the US (ECOG), and in 1974 in the EU (EORTC)
EORTC (1974): 1 coordinator, 2 statisticians, 2 computer analysts, 4 “data managers” (a new term), 2 secretaries, 2 US consultants
Involvement in the cooperative groups’ activities from the planning stages to the final report of a study’s results :
Protocol design, estimate of number of patients required and expected study duration
Design of study forms: standardization of data forms, structure of data files and routine procedures for data handling
‘ Designing forms for data collection in clinical trials: it was a big advance. It is hard to imagine that a pedestrian thing like that would be a big advance’ (Zelen)
Central registration and randomization of patients
Review of received patient forms for ambiguities, completeness or protocol violation
Presentation of interim administrative reports (including summaries of any toxicities observed) and statistical analyses at regular meetings with the cooperative group
Detailed statistical analysis at the end of the trial
‘ We applied all kinds of sophisticated statistical techniques for data analysis. Before, the statistical techniques were very pedestrian. In fact, we developed a lot of statistical software that no one else had’ (Zelen)
The EORTC Data Center: An instance of ‘Central Command’
CALGB 2006 The protocol lies at the core of the new style of practice
DMCs as a solution to the problem of too much information and too few patients
the cooperative groups discovered that the circulation of information about positive or negative results hindered participation:
North Central Cancer Treatment Group inaugurated the first DMC in 1977
SWOG followed suit in 1984, ECOG in 1990
in 1992 the NCI made DMCs mandatory for all cooperative groups
DMCs were thus established in order to:
restrict data circulation during long-term clinical trials: the original impetus was not to protect patients
reduce apparent and real conflicts of interests
assure good governance
‘ Because clinical trials are under increasing public scrutiny, we must use procedures that protect our research against the appearance of impropriety’ [R. Simon and R. Ungerleider, NCI, 1997]
the road since 1992
1994: NCI instituted policy requiring DMCs to be independent from cooperative group investigators
Statisticians from seven major cooperative groups initially strongly opposed this policy
‘ Monitoring a clinical trial is a complex task, best conducted by a small group integrating all the issues that must be addressed. This group should freely and confidentially discuss patient safety, the accuracy of accrual and outcome predictions made at the design stage, protocol compliance, unexpected side effects, recent results from other, similar trials, the place of the trial in the scientific program of the organization conducting the study, and, when appropriate, outcome data. These issues are not separable and should not be discussed in isolation’. [Statisticians for seven major cooperative groups]
Opposition subsequently subsided
‘ I watched [the independent DMC] grow from a group of ten or twelve people who really didn’t know about the trials they were monitoring to a group that worked very hard to understand the context of those trials in cancer research. … I didn’t see the conflict of interest issue being as large as the NCI people did. I still don’t but I have seen the external data monitoring committees work reasonably well and have even been on a few of them, so in that sense I think it’s turned out to be a good thing’. [Interview with David Harrington, Boston, 25 April 2005]
II. What kind of institution is a DMC? DMCs are reflexive institutions.
reflexivity as a form of endogenous critical inquiry
‘ endogenous critical inquiry’ (Lynch 1982): ‘practically solves the issue of what counts as the relevant contingent relationship between “objective” and “socio-historical’ states of affairs’
the search for an objective assessment of interim data, far from resting solely on methodological principles, resorts to reflexive practices focusing on:
the qualifications of DMC members
their attributes and prerogatives
the extent to which DMCs should be able to redefine the parameters of a trial and restrict the circulation of data
In sum: DMCs institutionalize endogenous critical inquiry
A typical DMC charter Informal reflexion and formal coding
The Guidance process i) 2001 Draft Guidance document issued ii) 2001 Public Meeting iii) 2006 Guidance document issued Forums for Reflexion
(1) role and purposes of DMCs
should a DMC have the power to change a trial protocol and under what conditions?
‘ In at least three DMCs that I’ve been a part of, rather acrimonious fights erupted at the beginning about my raising these kinds of issues, charges being made that this is a data monitoring committee; those are IRB or steering committee functions; it’s not for the DMC to do’. [Norman Fost, FDA meeting]
should a DMC be allowed to change sample size? ‘ A clinical trial ... is not a fixed quantity. It’s almost like a living thing. It evolves; it changes; it can change. One of the obvious ways in which a clinical trial might need to be modified has to do with the sample size. When the sample size is calculated, different things are assumed about the rate in the control arm, the rate in the treatment arm. Those assumptions may or may not be valid. And it may turn out that the trial is underpowered and the sample size needs to be adjusted. A data monitoring committee, although it’s not easy, can grapple with this’. [Gregory Campbell, FDA meeting]
should a DMC be allowed to change sample size? ‘ [a DMC] is seeing data on efficacy and for it to have the ability and the right to change end points and to change crucial aspects of design I think can sacrifice the integrity of the design’. [Janet Wittes, FDA meeting] ‘ the unblinded data monitoring committee really can’t credibly change end point, sample size, subset plans or anything, any more than an unblinded sponsor could without at a minimum affecting alpha or introducing bias that we don’t know how to correct’. [Robert Temple, FDA meeting]
should a DMC be allowed to change sample size? ‘ Not all studies are confirmatory but those studies that are confirmatory, I’d like to be able to interpret them in that manner. It means ... I’d like to have a pre-specified hypothesis that I then confirm. At the same time, there can well be during the course of a long trial external information that could enlighten us as to what the hypothesis really ought to be. I actually don’t have a problem if I’m certain that it’s external data that leads to refinement and this is the essence of where this independence and separation enables or empowers the sponsor to have that flexibility’. [Thomas Fleming, FDA meeting]
at what point does monitoring become intervention?
DMCs can be asked to validate initial designs and to participate in final analysis
should DMC members sign papers produced by these trials?
‘ adaptive designs’ allow DMCs to redesign the trial on the basis of interim results.
if DMCs participate in the design and/or redesign:
do they have interests in the trial?
are they still objective?
(2) structure and membership of DMCs
i n Britain, DMCs organized by the MRC have 3 members: two clinicians and a statistician; in the United States, DMCs may have five times as many members
what kind of expertise is needed for a DMC to function properly (‘objectively’)?
how to address potential conflicts of interest?
what kind of decision do DMCs seek: i) expert diagnosis, ii) in-house consensus or iii) majority rule by constituency ‘representatives’?
‘ I’ve often had it said that we have to have an odd number of people on the DMC so that when we vote it won’t come out tied. I object strongly to votes on DMCs. I believe that the DMC’s responsibility should include discussing issues at a length and in a depth to arrive at consensus about what ought to be done’. [Thomas Fleming, FDA meeting]
the issue of interests
DMC members may have financial and research interests
clinicians as consultants to pharmaceutical and biotech companies
program interests of government sponsors
‘ disease interests’ of patient representatives
and yet, members without interests are not interesting
endogenous critical inquiry focuses on the search for ‘acceptable levels of conflict of interest’
how many statisticians?
DMCs should be provided with three statisticians:
a statistician member of the trials’ protocol team or steering committee: BLINDED to the data
a statistician reporting to the DMC but independent of the trial sponsor: UNBLINDED access to the data
a statutory statistician member of the DMC
‘ Some wag said that this document is a full employment act for statisticians’. [DeMets, FDA Meeting] The problem: ‘Suppose that a trial steering committee were considering changing the primary endpoint of the trial. The statistician, knowing the interim results to that point, would know whether the drug being tested was in fact showing a stronger effect on the ‘new’ endpoint than the original endpoint, and would therefore be in an awkward position during this discussion. If the change were to be made, and if final analysis showed a significant effect on the new endpoint, there could be the perception that the interim data influenced the change – that if the trend for treatment effect had been different, the statistician would somehow have found a way to discourage the change in endpoint’.
Does the independent staistician have the expertise? ‘ In the multicenter, cancer cooperative group setting … the trial data manager prepares the data for the interim analysis which is carried out by the trial statistician. Within this setting we feel that it is difficult for a completely independent third party to carry out the analysis for the following reasons: 1. They will not be as familiar with the trial or the data as the trial data manager and trial statistician and are thus more likely to make mistakes in the analysis. 2. With a large number of trials it becomes much more difficult both practically and financially to have the trial analyzed by a third party’. [Richard Sylvester, FDA Dockets]
Should the committee itself remain blind?
DMCs established to keep interim results confidential so as not to compromise completion of the trial
what results, if any, should be transferred to those conducting the trial?
should DMCs monitoring different but complementary trials be allowed to share information?
how can DMC members themselves maintain clinical equipoise if they know that the interim results are positive or negative?
Is there a DMC culture and know-how? ‘ [DMC] decisions and the discussions are very complicated, they’re very nuanced, and they reflect a certain sociology of a committee that varies from committee to committee ... go to every DMC … that you can go to because you can learn a lot, it’s the only way you’re going to understand it and it’s really fun’. [Wittes, FDA Meeting] ‘ the need for folks specifically experienced and mentored in the process of DMC activities is really very critical and the confidence with which folks are able to invest the responsibilities into the groups is very importantly based on the talent base that exists to accomplish those goals’. [Conner, FDA Meeting]
create an ‘apprentice’ system to pair inexperienced statisticians with experienced statisticians on DMCs?
‘ Since there are no accepted standards of what constitutes conclusive evidence, the process by which (DMC members) arrive at their decisions must be made transparent’ [NEJM 2004]
III. DMCs and objectivity
objectivity as a historical construct :
regimes of objectivity (Daston & Galison)
mechanical (a-perspectival) objectivity
a new form of objectivity for biomedicine
True to nature: selection and synthesis
Mechanical objectivity: automatic transfer
Trained judgment: pattern recognition
a new form of objectivity for biomedicine
regulatory objectivity: less an epistemic virtue than a product of institutional action
generates and institutes conventions through concerted programs of collective action based on the use of a variety of systems for the production of evidence (e.g., clinical trials), that in turn call upon standard substances and standard practices (e.g. practice guidelines)
incorporates unprecedented levels of reflexivity: in their debates and discussions, biomedical practitioners take into account the conventional dimension of their endeavors
corresponds to a new regimen of coordination of medical practices that increasingly operates as a condition of possibility for the very existence of these practices: regulatory mechanisms are endogenous to and constitutive of research practices
DMCs and regulatory objectivity
DMCs have become part of the design of large-scale cancer clinical trials within which they operate as an institutionalized form of endogenous inquiry.
The implication of multiple agencies and institutions that seek the governance of clinical trials through objectivity and the reflexive layering and interlocking of regulatory arrangements serve to generate regulatory objectivity.
The kind of objectivity produced by DMCS is not reducible to an absence of ‘subjective bias’ or ‘conflict of interests’: the ‘integrity’ of the clinical trial process, established by DMCs, is also a sought-after good.
Following the rules
DMCs can be seen as a solution to the wittgensteinian problem of rules: every rule requires an interpretation, every interpretation, a rule for interpretation.
BUT: DMCs are not a sociological solution to the problem of infinite regress: acculturation to a specific community (i.e ‘this is how we do it’).
Instead: the rules are constantly revised in light of contradictory and alternative interpretations and emergent values.