Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to Innovations in reasoning about health: the case of the Randomized Clinical Trial--ISSA 2018-07-05
Similar to Innovations in reasoning about health: the case of the Randomized Clinical Trial--ISSA 2018-07-05 (20)
More from jodischneider
More from jodischneider (20)
Recently uploaded
Recently uploaded (20)
Innovations in reasoning about health: the case of the Randomized Clinical Trial--ISSA 2018-07-05
- 1. Innovations in Reasoning About Health: The Case of the Randomized Clinical Trial Jodi Schneider, Sally Jackson University of Illinois at Urbana-Champaign United States of America ISSA, Amsterdam, 2018-07-05
- 2. Source for argument: Bada et al. (2015). Morphine versus clonidine for neonatal abstinence syndrome. Claim: Clonidine may be a favorable alternative to morphine as a single-drug therapy for NAS. Data: Infants with NAS given clonidine improved more rapidly than those given morphine and had equal outcomes on other measures.
- 3. Claim: Clonidine may be a favorable alternative to morphine as a single-drug therapy for NAS. Data: Infants with NAS given clonidine improved more rapidly than those given morphine and had equal outcomes on other measures. Source for argument: Bada et al. (2015). Morphine versus clonidine for neonatal abstinence syndrome. Warrant: Randomized Clinical Trial
- 4. “Innovations” in reasoning and arguing Our research project focuses on fields where innovation is occurring rapidly (e.g., health): • Search for novel ways of drawing and defending conclusions (new warrants) • Examination of how these novel warrants are established (and sometimes dis-established) within fields
- 5. Warrants • Warrants are inference rules governing how conclusions are drawn, not statements from which conclusions are drawn. • “The claim is not presented as following from the warrant; rather it is presented as following from the grounds in accordance with the warrant.” (Hitchcock 2003, p. 71) • Warrants do not typically appear in the argument. • “Arguments instance inference rules, rather than include them as elements…” (Freeman 2011, p. 88; Hitchcock 2003 agrees) • Warrants generate conclusions as well as justify them • In law, relevant statute allows for “findings” of guilt or innocence (Toulmin, 1958, p. 99) • In science, “standard equations” generate predictions (Toulmin, 1958, p. 121) • They can be technical in multiple senses: • Expert communities devise them as tools for conclusion-drawing • They depend on “built” things (Jackson & Schneider 2018)
- 6. Warrants • Warrants are inference rules governing how conclusions are drawn, not statements from which conclusions are drawn. • “The claim is not presented as following from the warrant; rather it is presented as following from the grounds in accordance with the warrant.” (Hitchcock 2003, p. 71) • Warrants do not typically appear in the argument. • “Arguments instance inference rules, rather than include them as elements…” (Freeman 2011, p. 88; Hitchcock 2003 agrees) • Warrants generate conclusions as well as justify them • In law, relevant statute allows for “findings” of guilt or innocence (Toulmin, 1958, p. 99) • In science, ”standard equations” generate predictions (Toulmin, 1958, p. 121) • They can be technical in multiple senses: • Expert communities devise them as tools for conclusion-drawing • They depend on “built” things (Jackson & Schneider 2018)
- 7. Data Claim Warranting rule Dependable because backed by: Material assurances Procedural assurances Institutional assurances Source of figure: Jackson & Schneider 2018. Cochrane Review as a "Warranting Device" for Reasoning About Health. Warranting Devices
- 8. Warranting Devices - Definition (1) an inference license in Toulmin’s sense (2) invented for a specialized argumentative purpose and (3) backed by institutional, procedural, and material components that provide assurances of the dependability of conclusions generated by the device Source: Jackson & Schneider 2018. Cochrane Review as a "Warranting Device" for Reasoning About Health.
- 9. Claim: Clonidine may be a favorable alternative to morphine as a single-drug therapy for NAS. Data: Infants with NAS given clonidine improved more rapidly than those given morphine and had equal outcomes on other measures. Source for argument: Bada et al. (2015). Morphine versus clonidine for neonatal abstinence syndrome. Warrant: Randomized Clinical Trial material resources procedures institutions
- 10. patients & providers recruited Treatment B Treatment A measurements measurements random allocation blinding, other controls monitoring simple randomized clinical trial which group did better? statistical comparison
- 11. simple randomized clinical trial patients & providers recruited Treatment B Treatment A measurements measurements random allocation blinding, other controls monitoring which group did better? statistical comparison protocol approved
- 13. Published RCTs (by 1944) Institutional Review (1966 in US) Registration of Trials (2004) The RCT continues to evolve
- 14. Bradford Hill’s “The Clinical Trial”, 1952
- 15. Conclusion: Use CCTs to form beliefs about medical treatments. [P27] Goal: advancement of knowledge on which good treatment of patients depends at population level [P27] Circumstances: problems in prevailing methods for forming beliefs about treatment • variability in patient response to treatment [P6,7,9] • inability to distinguish between small effects and absent effects [P25] • inability to identify cause [P9,22] • inability to generalize [P9,10] • bias in individual clinician judgment [P13,22] and in patient judgment [P22] • a ‘literature’ composed of conflicting true findings [P6] Values: medical ethics (primacy of patient welfare) [P3], expertise [P1] Means-End: CCTs correct weaknesses in individual clinical judgments • successful in Great Britain [P21,22,25] and spreading [P14] • trial design eliminates various problems • random allocation of patients [P13] • blinding [P21] • exact specification of treatment schedule [P17,18] • standardized measurement [P19] • independent assessors [P21,23] Reconstruction of Bradford Hill’s Warrant-Establishing Argument, adapted from the practical argument scheme presented in I. Fairclough 2016: A Dialectical Profile for the Evaluation of Practical Arguments.
- 16. Circumstances: problems in prevailing methods for forming beliefs about treatment Means-End: CCTs correct weaknesses in individual clinical judgments Conclusion: Use CCTs to form beliefs about medical treatments. [P27] Imputed counterclaim: Reject CCTs [P2] Goal: advancement of knowledge on which good treatment of patients depends at population level Because the rationale for the action is faulty: • primary goal must be best care for the individual patient [P3] • don’t need CCTs to know what works [P24,25] • statisticians lack medical knowledge, count the wrong things, care only about the group [P1,3,4] • measures can’t replace clinical judgment [P5] • humans too variable to allow CCTs to generate medical wisdom [P6] Because the action itself is: • impractical (large numbers [P7-11], co- ordination [P18]) • unethical to experiment on patients [P26] • disregard of primary clinical responsibility • set treatment schedule not best [P17,18] • wrong to reduce patients to numbers • challenges authority of clinician [P1] Adapted from the deliberation scheme presented in I. Fairclough 2016: A Dialectical Profile for the Evaluation of Practical Arguments.
- 17. Black boxing • Warranting devices, once established within a field, may be employed without further defense across many situations. • Their initial establishment within a domain may be followed by an open-ended process of reconfiguration in response to newly-noticed problems. • “Black box arguments are a constantly evolving technology for coming to conclusions and making these conclusions broadly acceptable.” (Jackson, 2008)
- 18. “Beyond RCT” • Although RCT remains the gold standard for causal reasoning today, new competitors can arise at any time.
- 19. A Summary of Our Argument • Real argumentation serves practical purposes. • Innovation in reasoning and arguing is driven by these practical purposes. • New inference rules have to be argued into existence; they can be argued out just as well. • Change in the stock of usable warrants is normal. New things are invented and old things are abandoned.
- 21. patients & providers recruited Treatment B Treatment A measurements measurements random allocation measurements measurements blinding, other controls monitoring protocol approved randomized clinical trial with pretest and posttest measurements did Group A improve? did Group B improve? which group did better?
- 22. Conclusion: Apply rule R for belief formation in domain D. Goal: To form correct beliefs in domain D. Circumstances: Various ways incorrect beliefs may form in domain D. Values: Professional ethics in domain D, reason domain D is studied, etc. Means-End: Reasons for believing that rule R generates correct beliefs and/or avoids incorrect beliefs.
Editor's Notes
- This is part of our ongoing project looking at how specialized fields are extending their repertoires of inference rules to improve their ability to draw and defend expert conclusions. The very fact that we innovate in reasoning all the time is a phenomenon that needs exploration.
- Here’s the kind of argument we want to talk about. This example is from a small pilot study that was used to build the rationale for a longitudinal multi-site trial that is not yet completed. (Protocol at https://clinicaltrials.gov/ct2/show/NCT03396588). But this kind of argument has become extremely common in reasoning about health and medicine. Our study has to do with what warrants the move between the data and the claim.
- In this example, the conclusion is warranted by the fact that these data were produced through a randomized clinical trial, considered the gold standard for evidence of treatment effects. That is, the logic of a randomized clinical trial warrants the conclusion. An important fact about this warrant is that it was invented not very long ago, and people were still arguing about whether to apply it in medical science as recently as the 1950’s. ------ Bada HS, Sithisarn T, Gibson J, Garlitz K, Caldwell R, Capilouto G, Li Y, Leggas M, Breheny P. Morphine versus clonidine for neonatal abstinence syndrome. Pediatrics. 2015 Feb;135(2):e383-91. doi: 10.1542/peds.2014-2377.
- Argumentation theory needs to pay more attention to newly invented inference rules like this one. Any one of these innovations in reasoning has the same status as an already-cataloged scheme. That means we need scheme definitions for each one, we need to enumerate the critical questions for each one, and we need to show how people work with each one. What we’ve been doing is searching out cases of new inventions and examining them. We can think of these as new scheme types, but it’s convenient to think of them novel warrants in Toulmin’s sense, especially since we assume that as each one is proposed, it requires a warrant-establishing argument to promote its use.
- Toulmin’s warrants still aren’t thought of the same way by everyone, but we think that a fairly mainstream understanding of warrants includes these three characteristics.
- To the fairly mainstream understanding of warrants, we add something that we expect to have to defend: Sally and I have suggested that some warrants are technical objects that expert communities invent, apply, and evolve. And we’ve introduced the idea that a warrant may be bundled with technical components to produce what we call a warranting device.
- What appears to a practitioner as a method for drawing conclusions involves procedural components, executed by an individual practitioner, but also other components that are delegated to institutions and to other expert communities. Delegated components can allow an arguer to build on “know-how” that an expert community accumulates and maintains over time. This figure is from our recent paper in Argumentation paper, where we analyzed the Cochrane Review as a warranting device. Cochrane Reviews synthesize all available evidence about a health-related question. They allow inference from research literature to the best-justified treatment guidelines. The three categories of assurance represent the distribution of responsibility among contributors to the growth of knowledge in the area. We believe that these devices undergo significant technical improvement over time.
- Here’s how we defined warranting device in that article. Today we want to view Randomized Clinical Trial from this perspective, as an inference device of relatively recent invention. Let’s look again at an example of how Randomized Clinical Trial can be used as a warrant.
- What we’re saying is that the Randomized Clinical Trial is another warranting device, an inference rule that comes packaged with specific assurances that its conclusions will be acceptable to other experts.
- So we've seen what the argument looks like at its top level, but what does the experiment itself look like? The simplest acceptable design for a randomized clinical trial involves randomly dividing a pool of patients into two groups, giving the two groups two different treatments, and taking measurements afterwards in order to try to determine which group did better. At the end, the measurements from the two groups are analyzed statistically, which in current practice means computing group averages and testing them for statistical significance. A moment ago we showed warranting devices as an inference rule with some special kinds of (non-propositional) backing: some institutional, some procedural, and some material. This diagram helps to clarify what we mean by each type of assurance. Procedural assurances that back the inference from an RCT include random allocation, blinding, well-defined treatment specifications, and consistent measurement. Many material resources are needed to conduct such an experiment, but the ones that make a difference to the dependability of the conclusion are things like access to appropriate patients and providers and statistical tools that are, at this point in time, implemented in standard computer packages.
- Institutional assurances also affect the credibility of the result. While the logic of the RCT is quite easy to understand, conducting an RCT has become a highly regulated affair that makes it impossible for anyone acting outside complex institutional environments. For example, no one can enroll patients in an experiment of this kind without getting a protocol approved at multiple levels. For a decade or more, researchers have had to register their trials before beginning to recruit, and gradually it has become common for protocols to be published even before any results have been obtained (so that the community can know what things are being tried). There’s a lot about this that would be counterintuitive from the clinician’s point of view: randomly allocate patients; blind the provider to the treatment; decide which group did better based on pre-defined measures. That’s why the randomized clinical trial did not spring fully formed into existence. Arguments had to be given for why the randomized clinical trial is an appropriate way of drawing conclusions about health.
- Randomized Clinical Trial incorporates numerous components that are separate inventions in their own right. These inventions were motivated by countering problems in practice. Together, these inventions make invention of the RCT possible. === Blinding – Wikipedia Random allocation – Peirce and Jastrow, 1883-1884. Ian Hacking, "Telepathy: Origins of Randomization in Experimental Design," Isis 79, no. 3 (Sep., 1988): 427-451. https://doi.org/10.1086/354775 And The controlled clinical trial turns 100 years: Fibiger’s trial of serum treatment of diphtheria Placebos – Meldrum p 751 Random allocation, using subgroups (1934) – Therapeutic Trials Committee of the Research Council on the serum treat- ment of lobar pneumonia (Lancet, 1934, 1, 290), cited in Bradford Hill’s Principles of Medical Statistics. Published RCTs in Medicine – patulin trial on the common cold per Randomised controlled clinical trials statistical significance testing – Wikipedia confidence intervals – Wikipedia Outcomes - Outcomes Research in Oncology: History, Conceptual Framework, and Trends in the Literature Registration of trials - Clinical trial registration: a statement from the International Committee of Medical Journal Editors [Editorial]., Ann Intern Med, 2004, vol. 141 (pg. 477-8) 1966: NIH Office for Protection of Research Subjects created. Policies call for IRBs https://www.ctsi.ucla.edu/education/files/training/docs/research-ethics-wenger.pdf
- The RCT has continued to evolve, with new practices and requirements bundled into its performance.
- It took work for RCT to become established. One of the people doing that work was Austin Bradford Hill; we analyzed the published report of a speech he gave at Harvard Medical School. So this is a part of our data, a source of insight into how the device became established as the "gold standard" for reasoning about medical treatments. Our method: we went paragraph by paragraph, searching for argumentatively relevant content. Most of the content fit well into a practical reasoning scheme such as the one offered by Isabella and Donald Fairclough. Although it was a single long-form argument, we found it necessary to import a dialectical framework, based on what Austin Bradford Hill treated as major counterarguments against his position.
- In the Faircloughs’ work on practical reasoning in political discourse, the conclusion is always an action claim, such as a claim about what policy should be adopted. We’ve put the conclusion in a directive form, to highlight that until people start taking this action, they still haven’t agreed to it. Bradford Hill used the term “cooperative clinical trial” or CCT, but that phrase has now dropped out of use, and now we would refer to this as an RCT.
- In addition to presenting the case for clinical trials (left), Bradford Hill also introduced and countered the case against clinical trials. Fairclough points out that in deliberative discussion a practical argument can come under attack both for the adequacy of the rationale offered for the action and for the acceptability of the action considered on its own. These aren’t equivalent attacks, but they can be completely separate bases for rejecting the proposed action. Bradford Hill alludes to a number of such attacks, possibly coming from multiple different attackers, and provides rebuttals to each one.
- What we see in Bradford Hill is what it takes to establish the warrant prior to its use. By contrast, the argument that we opened with used this warrant without explicit defense, as most conclusions justified by Randomized Clinical Trials do. Once Bradford Hill and others like him have explained how random allocation strengthens the conclusion, that need never be mentioned again. All that is needed is a mention that randomized allocation occurred in such-and-such a way. Establishing a warrant means getting to this point where it can be used to defend individual conclusions without explicit defense. All of the backing elements are “boxed up” for repeated taken-for-granted use. Latour has described this process of black boxing generally, not with specific reference to inference rules, but with attention to the marked difference between the period in which the black box is being assembled and the period in which the black box is used by practitioners who feel no need to inspect its inner workings. Once a warrant is established convincing and coherent way of drawing a conclusion, we may find ways in which it can go wrong, in which case we can revisit conclusions drawn from it in the past. So, black boxes can be re-opened, but they aren’t kept open all the time.
- We’re currently examining the rise of n-of-1 experiments, which evaluate effects at the individual level, and aggregate effects rather than aggregating outcomes. One reason for re-opening a black box is that the assumptions that are concealed inside become controversial and have to be excavated. One assumption buried inside the RCT is that our goal is to find the best treatment that can be given across the board. In the past few decades it has become clear that in many cases there is no single best treatment, bolstering the critique that the “primary goal must be best care for the individual patient” since the RCT cannot tell us the best treatment for an individual patient. Rise of N-of-1 experiments and their aggregation. Conducted on one individual person, but aggregated. Learn from the distribution Set up experimentally to create subgroups “at a distance”, facilitated by technology, often mobile technology We are starting to see warrant-establishing arguments for claims based on n-of-1 data Material components in Trialist: iPhone, Android, software Expecting individuals to reliably follow the protocol Change in circumstances: “ridiculously easy organizing” – Clay Shirky
- In real argumentation, people are far less interested in whether their inferences are valid than in whether their inferences serve practical purposes (such as providing proper health care). These practical purposes drive innovation in reasoning and argument: looking at warrant-establishing arguments through the lens of practical reasoning exposes why people invent new inference rules—what we want to do with what we can conclude from a given rule. Innovation can be iterative, with a rule ‘boxed’ as it achieves stability or ‘unboxed’ as it encounters new challenges: a new inference rule can be argued into existence and argued out just as well. How the rule develops depends on what people have noticed as vulnerabilities and objections. The rule can develop different ways in different places (e.g., toward being able to generalize vs toward being able to address the particular case). Rule emergence is highly sensitive to technical capabilities (from availability of inferential statistics in the rise of RCT to availability of online coordinating platforms and Big Data methods in the rise of N-of-1 experiments).
- the design can be elaborated in all kinds of ways, including through the addition of pretest measurements. If these pretest measures are the same as the posttest measures, this design allows the researcher to evaluate how much each group changed, if at all, as well as to assess which group changed more. While the logic of the RCT is quite easy to understand, conducting an RCT has become a highly regulated affair that makes it impossible for anyone acting outside complex institutional environments. For example, no one can enroll patients in an experiment of this kind without getting a protocol approved at multiple levels. For a decade or more, researchers have had to register their trials before beginning to recruit, and gradually it has become common for protocols to be published even before any results have been obtained (so the community can know what things are being tried).
- Generalized form of a practical argument in support of a new inference rule One thing to expect when you go looking for a warrant-establishing argument We’ve analyzed two cases in detail, Cochrane Review (June 2018, Argumentation 32: 241-272.) and Randomized Controlled Trial Sally Jackson will present a third case in the afternoon about modeling natural phenomena as computations.