Successfully reported this slideshow.

Rhetorical moves and audience considerations in the discussion sections of randomized controlled trials of health interventions--ECA--2017-06-22

1

Share

1 of 53
1 of 53

Rhetorical moves and audience considerations in the discussion sections of randomized controlled trials of health interventions--ECA--2017-06-22

1

Share

Download to read offline

European Conference on Argumentation talk

Jodi Schneider, Graciela Rosemblat, Shabnam Tafreshi and Halil Kilicoglu “Rhetorical moves and audience considerations in the discussion sections of Randomized Controlled Trials of health interventions” [Conference Panel Presentation], 2nd European Conference on Argumentation: Argumentation and Inference, Fribourg, Switzerland, June 20-23

1 of 3 talks in Jodi Schneider and Sally Jackson, organizers, “Innovations in Reasoning and Arguing about Health ”[Conference Panel], 2nd European Conference on Argumentation: Argumentation and Inference, Fribourg, Switzerland, June 20-23.

European Conference on Argumentation talk

Jodi Schneider, Graciela Rosemblat, Shabnam Tafreshi and Halil Kilicoglu “Rhetorical moves and audience considerations in the discussion sections of Randomized Controlled Trials of health interventions” [Conference Panel Presentation], 2nd European Conference on Argumentation: Argumentation and Inference, Fribourg, Switzerland, June 20-23

1 of 3 talks in Jodi Schneider and Sally Jackson, organizers, “Innovations in Reasoning and Arguing about Health ”[Conference Panel], 2nd European Conference on Argumentation: Argumentation and Inference, Fribourg, Switzerland, June 20-23.

More Related Content

More from jodischneider

Related Books

Free with a 14 day trial from Scribd

See all

Rhetorical moves and audience considerations in the discussion sections of randomized controlled trials of health interventions--ECA--2017-06-22

  1. 1. Rhetorical Moves and Audience Considerations in the Discussion Sections of Randomized Controlled Trials of Health Interventions Jodi Schneider (University of Illinois at Urbana Champaign, USA) jschneider@pobox.com Graciela Rosemblat (National Library of Medicine, USA) Shabnam Tafreshi (The George Washington University, USA) Halil Kilicoglu (National Library of Medicine, USA) European Conference on Argumentation, Fribourg, Switzerland 2017-06-22
  2. 2. Disclaimer The views and opinions expressed do not necessarily state or reflect those of the U.S. Government, and they may not be used for advertising or product endorsement purposes.
  3. 3. Why study scientific papers? • “The interpretive face of science is characterized by controversy, disagreement, dissensus.” (Prelli, “A Rhetoric of Science”, 1989, p4) • “Scientific publications are documentary representations of defeasible arguments, supported by data and repeatable methods.” (Clark, Ciccarese, Goble 2014) • Prelli, Lawrence J. "A rhetoric of science: Inventing scientific discourse." (1991). • Clark, Tim, Paolo N. Ciccarese, and Carole A. Goble. "Micropublications: a semantic model for claims, evidence, arguments and annotations in biomedical communications." Journal of biomedical semantics 5.1 (2014): 28.
  4. 4. Why study medical papers? • Large potential impact on human health • Input to evidence-based practice • Well-elaborated study of evidence Figure credit: Duke University Medical Center Library. Introduction to Evidence-based Practice. What is Evidence-Based Practice (EBP)? http://guides.mclibrary.duke.edu/c.php?g=158201&p=1036021
  5. 5. Hierachy of Evidence
  6. 6. Hierachy of Evidence
  7. 7. Randomized Controlled Trial (RCT) • Medical RCTs test the efficacy of medical treatments. • Features: – Control group (placebo or standard treatment) – Random assignment – Neither doctors nor patients know which group is which (“double blind”) Image credit: Ancker, Jessica S & Quynh Pham. "Beyond the RCT Evaluating innovations in the Learning Health System” AMIA 2016 Tutorial
  8. 8. Corpus • 37 research articles reporting on medical trials, mainly RCTs • Full-text available from PubMed Central • Chosen for an ongoing citation annotation study using Xu et al.’s corpus Xu, Jun, Yaoyun Zhang, Yonghui Wu, Jingqi Wang, Xiao Dong, and Hua Xu. "Citation sentiment analysis in clinical trial papers." In AMIA Annual Symposium Proceedings, vol. 2015, p. 1334. American Medical Informatics Association, 2015.
  9. 9. Other ongoing study on this corpus • Understand citation structure of Discussion sections of Randomized Clinical Trials • Annotate two aspects of citations: – Citation scope – Citation matter • Began summer 2016 at U.S. National Library of Medicine, led by Halil Kilicoglu, with Graciela Rosemblat leading annotation. • Ongoing study in biomedical informatics
  10. 10. Corpus characteristics • Various conditions, e.g. – Brain injury – Degenerative eye disease – Malaria – Postpartum back pain • Various settings, e.g. – Critical care – Community/Ambulatory care – Internet-based treatment – Laboratory treatment – School-based treatment
  11. 11. Corpus characteristics • Various locales, e.g. – Madagascar – Malawi – Netherlands – Uganda – United Kingdom – United States • Single site and Multisite • Industry-sponsored and Researcher-focused • Various stages of clinical trial
  12. 12. Focus: 37 discussion sections http://europepmc.org/ articles/PMC1420299
  13. 13. Rhetorical Moves in RCT Discussion Sections
  14. 14. Rhetorical Moves in RCT Discussion Sections • Novelty • Gaps/opportunities • Contextualization • Comparability to previous results • Validity of methods • Anticipating objections • Pointers to ongoing work • Acknowledging limitations • Future work • Summaries
  15. 15. Novelty • “We report the first safety data collected in the setting of a randomized, controlled study for patients sustaining TBI with polytrauma.” (2206502) • “This is the first double blind prospective study designed to evaluate the effects of fruit juice feedings during diarrheal disease in young children.” (1217327)
  16. 16. Gaps/opportunities • "At the start of this study, little was known about the specific effect of psychosocial determinants in SCs, although interventions aiming to modify these determinants, such as cognitive behavioural programs were considered to be promising for musculoskeletal pain in general [4-8]. We developed the EAP to fill a gap in UC, which focuses mainly on biomedical determinants." (2211478) • "We have already studied the stimulation of periodontal tissue regeneration by FGF-2 in animal models and believe that the protein represents a major candidate for a periodontal tissue-regenerating agent." (2432040)
  17. 17. Contextualization • “This study is part of a series to explore effective xylitol delivery vehicles that can be used in school programs in the U.S. Results from the xylitol dose study [3] showed...” (2527560) • “In general we did not deviate from the original study design, which was described independently of the study results [15].” (1420299) • "The first report on this study showed that the intra- operative use of TA significantly decreased the incidence of intra-operative complications associated with PPV, and no adverse events related to surgery were found over a 3-month observation period [13]." (2413124)
  18. 18. Comparability to previous results • "However, none of these 11 studies were performed on pediatric populations, making further inferences or comparisons problematic" (1876598) • "This is in line with previous studies of internet-driven CBT for posttraumatic stress reactions [9], complicated grief [30] and CBT interventions in face-to-face studies [10,31]." (1885249)
  19. 19. Validity of methods "Algometry is used to measure the sensitivity of pain or pressure [4]. Algometer instrumentation can include manual and electric models. The use of any experimental instrument including the PGA must be tested for validity and reliability between examiners and between performances of the same examiner. The PGA used in this study has been tested against itself, palpation, pressure plates with reliable results [17,18]. However, the above is based on the assumption that the tester is trained in the application of the PGA otherwise issues associated with the rate of pressure application [45- 47] the determination of an end point based on a verbal patient response [45,46] and the possible sensitisation of a selected landmark based on repeated measures may all alter the accuracy of the outcome." (2427032)
  20. 20. Validity of methods "The technique of using the consumption of morphine during PCA treatment of postoperative pain, as a measure of the effect of the analgesic regime under study, has been used in several other studies of this kind [5,6]." (1637100)
  21. 21. Anticipating objections "It might be argued that these observations of blood stage immunity (reduction in growth rates, presence of crisis forms etc.) may rather be related to the potential pre-erythrocytic action of PEV3A. However, in previous challenge studies, the number of parasites emerging from the liver of unvaccinated control volunteers has been shown to vary as much as five fold [7]. Despite this variation, rates of parasite growth in these volunteers were similar. Equally, crisis forms have never been observed historically in our studies of pre-erythrocytic vaccine candidates. Indeed, following their observation in this study, the same slide reader went back to examine a selection of blood films from vaccinated and control volunteers in two previous studies where some evidence of pre-erythrocytic efficacy has been observed (VAC021 [28] and VAC023 [29]). The slide reader was blinded to the group allocation of the volunteers. In total 72 slides were selected for re- examination (6 slides each from 6 volunteers from each study, 3 vaccinees plus 3 controls) and no crisis forms were observed. It seems likely therefore, that the differences observed here are indeed related to vaccine induced blood stage immunity." (2204057)
  22. 22. Pointers to ongoing work • "An ongoing phase III study is likely to provide a better evaluation of safety in this patient population." (2206502) • Further analysis of the 18 months follow-up data and the examination of other potentially relevant moderators such as posttraumatic growth [Maercker & Knaevelsrud, in preparation] will hopefully enhance our understanding of online therapeutic processes." (188529)
  23. 23. Acknowledging limitations • "Among the limitations of this study is the screening strategy for the recruitment of the patients. We deliberately handled strict exclusion criteria for participation in this study. We excluded 72% (n = 253) of the patients who wanted treatment but did not meet the inclusion criteria. This might limit the generalizability of our results.” (1885249)
  24. 24. Future work • "New studies should be carried out, with longer follow-up times and larger samples, to evaluate the effects of NPPV on the quality of life of patients on weaning ventilation support and to understand how the cause of ARF could affect the results of different weaning ventilation methods. Cost evaluation should also be included in these studies." (2447605) • "Future studies directed on possible central mechanisms of this complicated chronic pain syndrome are warranted."(1815612)
  25. 25. Summaries • "Based on present evidence BTX cannot be recommended as treatment for neck pain in chronic whiplash patients."(1815612) • "The evidence from this study is that Hall PMCs allow a reliable, low-maintenance seal to be achieved by GDPs." (2265270) • "Therefore, the mouthwash clearly demonstrated an anti-malodor effect on morning breath potential without any measurable side effects in healthy subjects." (2637235)
  26. 26. Rhetorical Moves in RCT Discussion Sections • Novelty • Gaps/opportunities • Contextualization • Comparability to previous results • Validity of methods • Anticipating objections • Pointers to ongoing work • Acknowledging limitations • Future work • Summaries
  27. 27. Audience
  28. 28. Multiple audiences “Audiences of specialists…do not comprise the only audiences who respond to scientific or other professional discourse. There may be a number of interested audiences for any particular discourse.” (Prelli,1989, p26)
  29. 29. Interested audiences include: • Specialists responsible for “authorizing the claims to knowledge” (Prelli,1989, p26) • Non-specialists who might “judge the work as interested witnesses” especially if there are “broad social implications” (Prelli,1989, p27)
  30. 30. Fulfilling audience expectations “To succeed rhetorically, any rhetor must ground a position in what the addressed audience considers reasonable”(Prelli,1989, p27-8) “…[draw] on standard or conventional patterns of thinking that will be employed by one’s specific audience”(Prelli,1989, p37) “…choose purposes and contents likely to be deemed pertinent and conventionally appropriate by the specifically situated thought groups that comprise the audience.”(Prelli,1989, p38)
  31. 31. Audience-specific sections from discussion sections of reports of Randomized Controlled Trials
  32. 32. Researchers are the expected audience for RCTs Mellis, Craig. "Evidence-based Medicine: What Has Happened in the Past 50 Years?" J Paediatr Child Health 51, no. 1 (2015): doi:10.1111/jpc.12800.
  33. 33. Audiences we find • Specialist researchers in the same area • Clinicians • Clinical practice managers • Grant funders • Health systems
  34. 34. Specialist researchers in the same area • “The most plausible explanation for the similar trends in BV reduction in both arms of this study is the inherent activity of the placebo formulation.” (1851729)
  35. 35. Clinicians • “caution should be exercised in applying these findings to the general management of dental caries without being aware of potential pit falls.”(2265270) • “none of the results of this particular clinical trial suggest any clinical problems concerning the safety of administering FGF-2 to patients with periodontitis. One more piece of evidence supporting the high safety of FGF-2 applied locally to periodontal tissue is that this therapy has already been used for more than 5 years as a remedy for intractable ulcers (Fiblast spray).” (2432040)
  36. 36. Clinical practice managers • “Our findings are particularly pertinent to smaller clinical practices that might not be aligned with larger organizations, and who will have to choose EMR systems and DSSs on their own.”(1876598)
  37. 37. Grant funders • “Additional research is needed to determine whether these improvements translate into even longer-term reductions of health-related costs. Although the sample was heterogeneous on demographic factors such as income, the group was relatively ethnically and racially homogenous, consisting primarily of white women. Future replications with diverse populations are indicated to further evaluate the promise and translation potential of the MLP.” (1783667)
  38. 38. Grant funders "Admittedly, overall complexity of the issue addressed by the present study clearly merits pursuit of further studies of a closely similar, multi-factorial intervention design, though preferably conducted over much longer time span, as well as best targeting the over 60s as the population standing by far the best chance of a successful outcome, in order to verify whether the present findings, clearly encouraging as they appear, might actually have sufficient potential to develop into a trend of indisputable clinical significance, especially in terms of possible application in a comprehensively designed, nationwide programme specifically aimed at addressing the problem of a steadily growing proportion of the frail seniors dependent for their activities of daily living." (2637855)
  39. 39. Health systems • “Given the rising costs and growing burden of diabetes on the health-care system [57], strategies that aim to prevent or delay the onset of this disease might be beneficial [58]” (1865086) • “AQ+AS (0.51 US$) [28] is currently four times more expensive than AQ+SP (0.13 US$). In Madagascar, with a reported 2,114,400 cases of suspected malaria (2003), the use of AQ+SP instead of AQ+AS would reduce the annual antimalarial treatments costs by 800,000 US$. In view of the recent report by Tagbor and colleagues [29] there is hope that AQ+SP could serve as a safe and effective alternative for malaria treatment in pregnancy and for intermittent preventive treatment until the safety of ACTs for pregnant women has been established.” (1887535)
  40. 40. Audiences we find • Specialist researchers in the same area • Clinicians • Clinical practice managers • Grant funders • Health systems
  41. 41. Rhetorical Moves in RCT Discussion Sections • Novelty • Gaps/opportunities • Contextualization • Comparability to previous results • Validity of methods • Anticipating objections • Pointers to ongoing work • Acknowledging limitations • Future work • Summaries
  42. 42. Future work • Systematically code rhetorical moves and audiences – likely to lead to revisions • Compare to previous work in English for Specific Purposes for medical and general scientific work • Identify key suggestions for writing effective RCT reports, to reach their intended audiences. • Develop Argumentation Mining algorithms to automatically detect rhetorical moves. – Could be used to automatically extract discussion topics for evidence synthesis of RCTs. – Could be used to give authors fast, automatic feedback about readability.
  43. 43. Citation examples from discussion sections of reports of Randomized Controlled Trials
  44. 44. “This” work agrees with… • “This is in accordance with earlier studies in the ambulatory surgical setting [3]” - PMC1637100
  45. 45. Definitions and background info • “Self-efficacy, which may relate to motivation, is the perceived confidence in one's ability to accomplish a specific task [19].” - PMC2194735
  46. 46. Presenting a range of evidence • “Except in one study [20], short-term administration of GH transiently worsened insulin resistance [19,53] and increased fasting glucose levels [53].” - PMC1865086
  47. 47. Clause-level changes in meaning • “Two of four randomised clinical trials …have found a difference in admission rate [12,19] and two have not [22,23].” - PMC1142326
  48. 48. A single citation can support a whole paragraph • Dutton and colleagues [8] described a series of 81 coagulopathic trauma patients treated with rFVIIa. Of these, 20 received rFVIIa for treatment of coagulopathy related to TBI. Six of these patients had additional polytrauma. The outcome of these patients was poor and 15 of 20 patients died. The authors attributed this high mortality rate to the severity of brain injury. None of the 81 trauma patients in this series had any clinical indication of TE events.”
  49. 49. Discussing treatments, outcomes, other authors’ conclusions • Dutton and colleagues [8] described a series of 81 coagulopathic trauma patients treated with rFVIIa. Of these, 20 received rFVIIa for treatment of coagulopathy related to TBI. Six of these patients had additional polytrauma. The outcome of these patients was poor and 15 of 20 patients died. The authors attributed this high mortality rate to the severity of brain injury. None of the 81 trauma patients in this series had any clinical indication of TE events.”
  50. 50. Sometimes several parallel paragraphs. • Dutton and colleagues [8] described a series of 81 …patients treated with rFVIIa” • “Zaaroor and Bar-Lavie [23] reported the first series of five patients …” • “Morenski and colleagues [24] described …three pediatric … cases”
  51. 51. Multiple citations in a paragraph • “Berger et al. [42] compared the efficacy of hypertonic saline and mannitol to reduce ICP after a combination of two different neuronal injuries. Initially, ….The authors demonstrated that …After …. It is remarkable that … An accumulation …These different effects … [42]. Furthermore, Prough et al. observed a higher regional cerebral blood flow in dogs with induced intracerebral hemorrhage after hypertonic saline without any increase of the CPP [43].” - PMC1297608
  52. 52. Avoiding a 1-sentence paragraph? • “Berger et al. [42] compared the efficacy of hypertonic saline and mannitol to reduce ICP after a combination of two different neuronal injuries. Initially, ….The authors demonstrated that …After …. It is remarkable that … An accumulation …These different effects … [42]. Furthermore, Prough et al. observed a higher regional cerebral blood flow in dogs with induced intracerebral hemorrhage after hypertonic saline without any increase of the CPP [43].” - PMC1297608

Editor's Notes

  • Rhetorical moves and audience considerations in the discussion sections of Randomized Controlled Trials of health interventions

    In order to inform clinical practice about human health, evidence- based medicine calls for various forms of evidence to be used, including controlled experiments, observations, and clinical case studies. Clinicians and medical researchers are taught to consider the strength of the evidence in medical sciences, typically using a hierarchy of evidence. In such a hierarchy, one of the highest forms of evidence is generally considered to be randomized controlled trial (RCT) experiments. In said hierarchies, RCTs are superseded only by evidence syntheses (i.e., systematic reviews and meta-analyses) over multiple randomized controlled trial experiments. The current work draws on two ongoing projects. First, in a recent theoretical paper we have considered the iterative development of RCTs and evidence syntheses over the past century through an argumentative lens; we have described how shortcomings in new methods are revealed and iteratively overcome through argumentation, until these methods become trusted within a given community of experts, at which point they may find application in new areas addressing new audiences. Second, in an empirical project, we are annotating citation-related discourse in a corpus of open access RCT articles drawn from PubMed Central, which describes itself as "a free full- text archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health's National Library of Medicine".

    In this paper we study the argumentation in discussion sections. Our data source is a corpus of 37 research articles reporting on RCTs in diverse areas. Articles in our RCT corpus relate to wide range of conditions – including malaria, degenerative eye disease, postpartum back pain, and brain injury. These articles may test the efficacy of medical treatments or may compare physiological measures (such as methods to measure cardiac output).

    We will document and classify rhetorical moves in the discussion sections of these RCT reports. Our preliminary research suggests that, in writing these articles, the RCT authors have targeted multiple audiences including other researchers, grant funders, and clinicians.

    This finding would be consistent with the importance of RCTs in the biomedical hierarchy of evidence yet it seems in stark contrast with our expectation that scientific papers are written for, and primarily address, other experts in a given area of science.

    This paper will present our first findings, grounded in quotations from our corpus of RCTs, characterizing fine-grained rhetorical moves in discussion sections. We will use these moves in order to determine which higher-level argumentative goals seem salient in RCTs discussion section across a wide range of sub-disciplines. We will also seek to associate each move and each goal with the audience or audiences that it addresses, in order to test our hypothesis that RCT authors address multiple audiences in reporting RCTs.

    We believe that our findings could be used in teaching authors to write effective RCT reports, to reach their intended audiences. In the future, this work might also support the automatic detection of rhetorical moves; the related area of argumentation mining is under active development within the natural language processing community. We envision that future computer augmentation using a human-in-the-loop design and natural language processing might assist authors in writing reports of RCTs and aid evidence synthesizers in extracting relevant data from RCTs.
  • Image credit: Mellis, Craig. "Evidence-based Medicine: What Has Happened in the Past 50 Years?" J Paediatr Child Health 51, no. 1 (2015): doi:10.1111/jpc.12800.
  • Image credit: Mellis, Craig. "Evidence-based Medicine: What Has Happened in the Past 50 Years?" J Paediatr Child Health 51, no. 1 (2015): doi:10.1111/jpc.12800.
  • Can also compare physiological measures (e.g. methods to measure cardiac output)
  • e.g. for former, “A dozen trained scientists working to resolve exigencies that the discourse addresses”
  • ;only then can understanding and appreciation be achieved,
    and only then can approval and authorization ever be hoped for.”
  • ×