Herrenhauser big data poster: Decision support on overwhelming amounts of data 2015-03-26

•Download as PPTX, PDF•

1 like•689 views

Poster for Herrenhausen Conference on Big Data: http://www.volkswagenstiftung.de/en/events/calendar-of-events/details-of-events/news/detail/artikel/herrenhausen-conference-on-big-data-1/marginal/4526.html

Internet

How to provide decision support on overwhelming amounts of data?
Jodi Schneider - INRIA • jschneider@pobox.com
Step 1: Understand the decision process &
criteria
• Ethnography
o Participant Observation
o Interviews
• Annotation
o Argumentation theory
Step 2: Build a computer support system
• Web standards
o Develop an OWL ontology
o Structure data in RDF format
o Query with SPARQL
• Human computer interaction: Design
Step 3: Test & improve the system
• Human computer interaction: User
testing & Design
2. Method
3. Decision Support Case Study:
Wikipedia Article Deletion
Step 1: Understand the decision process & criteria
Figure 3 - "CriteriaFilter" (red) improves on
the native Wikipedia interface (blue), except
in terms of perceived effort.
1. Challenges
• Managing large amounts of data
• Understanding the decision-making
process
• Designing interfaces that support
decision making
4. Discussion
• Process depends on determining key factors in the
decision.
• New application of method to medication safety:
Which drugs shouldn't be taken together?
o Provide support to evidence curators
o Ontologies: micropublication, nanopublication
Problems identified from interviews & participation:
• Large volume: 500 deletion discussions per week
• Consensus is difficult to determine.
• Newcomers don't understand process &
standards.
Results from annotation:
• Identified key criteria in discussions:
Notability, Sources, Maintenance, Bias
• Classified comments by key criteria.
• Validated classification two ways:
Interannotator agreement (.64-.82 κ)
Coverage (key criteria used in 90% of comments).
Step 3: Test and improve the "CriteriaFilter" system
• Developed the WikipediaDeletion ontology.
• Embedded the classification from the manual
annotation into web pages.
• Wrote custom SPARQL queries to retrieve all
comments by factor.
• Made “CriteriaFilter” interface by embedding
queries into JavaScript.
• 20 users perform tasks with both "CriteriaFilter" and
the native Wikipedia interface.
• Statistically significant improvements in 3 areas:
o perceived usefulness
o perceived ease of use
o information completeness
• Strong overall preference for "CriteriaFilter" (84%).
• Qualitative feedback used to improve the next
version.
Step 2: Build a computer support system "CriteriaFilter"
Figure 1 – Wikipedia deletion discussion
Figure 2 – “CriteriaFilter” interface

Viewers also liked

La tecnología de la educación (dimna)

Dimna Garcia

Programa 2

zaidajua

Open data y ciudadanía

evaristogonzal2

Dr. Sanjay Gupta | CNN | Emory University

Erica Cleveland

First Draft

Amy Peterson

Atividade 3.2 slide de apresentação unidade III

Ramaiany Marçal Tregnago

Generating Savings

Gerstein Fisher

Duke: Open Enrollment 2014

workingatduke

Antonio valencia

NinethCaicedo

Comenius Museums in Fashion - Power Point 2th meeting in Turkey 2012

rdiiorio

It's All About The Team

The Concept Store

Viewers also liked (11)

La tecnología de la educación (dimna)

Programa 2

Open data y ciudadanía

Dr. Sanjay Gupta | CNN | Emory University

First Draft

Atividade 3.2 slide de apresentação unidade III

Generating Savings

Duke: Open Enrollment 2014

Antonio valencia

Comenius Museums in Fashion - Power Point 2th meeting in Turkey 2012

It's All About The Team

More from jodischneider

Continued Citation of Bad Science and What We Can Do About It Even papers that falsify data continue to be cited. I describe network and text analysis for studying how authors continue to cite bad science: articles retracted from the literature due to serious flaws or errors. I will present an in-depth case study of a human trial cited for over 10 years after it was retracted for falsifying data. Then, I will describe how the team scaled up to study a data set of 7000 retracted papers and hundreds of thousands of citations. Finally, I will discuss an ongoing Sloan-funded stakeholder consultation that is bringing editors, publishers, librarians, researchers, and research integrity experts together to address this problem. BiographyJodi Schneider is Assistant Professor at the School of Information Sciences, University of Illinois at Urbana-Champaign where she runs the Information Quality Lab. She studies the science of science through the lens of arguments, evidence, and persuasion with a special interest in controversies in science. Her recent work has focused on topics such as systematic review automation, semantic publication, and the citation of retracted papers. Interdisciplinarity (PhD in Informatics, MS Library & Information Science, MA Mathematics; BA Great Books/liberal arts) is a fundamental principle of her work. She has held research positions across the U.S. as well as in Ireland, England, France, and Chile. She leads the Alfred P. Sloan-funded project, Reducing the Inadvertent Spread of Retracted Science: Shaping a Research and Implementation Agenda. With Aaron Cohen and Neil Smalheiser she is working on the NIH R01 "Text Mining Pipeline to Accelerate Systematic Reviews in Evidence-Based Medicine". Talk with her about scoping reviews and about citation-based methods for updating systematic reviews! Tuesday, April 20th, 2021 Noon-1PM Eastern GWU - CNHS Informatics Seminar

Continued citation of bad science and what we can do about it--2021-04-20

jodischneider

Title: Continued Citation of Bad Science and What We Can Do About It Abstract: Even papers that falsify data continue to be cited. I describe network and text analysis for studying how authors continue to cite bad science: articles retracted from the literature due to serious flaws or errors. Jodi will present an in-depth case study of a human trial cited for over 10 years after it was retracted for falsifying data. Then, will describe how the team scaled up to study a data set of 7000 retracted papers and hundreds of thousands of citations. Finally, Jodi will discuss an ongoing Sloan-funded stakeholder consultation that is bringing editors, publishers, librarians, researchers, and research integrity experts together to address this problem.

Continued citation of bad science and what we can do about it--2021-02-19

jodischneider

Presentation for the Bibliometrics & Research Assessment Symposium 2020 (bibSymp20) https://www.nihlibrary.nih.gov/services/bibliometrics/bibSymp20 October 9, 2020 Retraction is intended to remove articles from the citable literature. However, a series of studies from over 30 years, from 1990 through 2020, have found that many retracted papers continue to be cited, and cited positively, even following misconduct-related retractions. For instance, a fraudulent clinical trial report retracted in 2008 continues to receive citations in 2020, and 96% of post-retraction citations do not mention its citation - perhaps because its retraction not marked on the publisher website and its retraction notice cannot be readily retrieved from 7 out of 8 databases (8 out of 9 database records) we tested. This talk draws an ongoing systematic mapping study of research about retraction and our own research projects to summarize what is known about post-retraction citation in biomedicine. We outline practical steps that authors and reviewers can take to avoid being caught out by poorly marked retracted papers. 20 minutes including Q&A

The problems of post retraction citation - and mitigation strategies that wor...

jodischneider

JCDL2020 full paper. Abstract: Scientific digital libraries speed dissemination of scientific publications, but also the propagation of invalid or unreliable knowledge. Although many papers with known validity problems are highly cited, no auditing process is currently available to determine whether a citing paper’s findings fundamentally depend on invalid or unreliable knowledge. To address this, we introduce a new framework, the keystone framework, designed to identify when and how citing unreliable findings impacts a paper, using argumentation theory and citation context analysis. Through two pilot case studies, we demonstrate how the keystone framework can be applied to knowledge maintenance tasks for digital libraries, including addressing citations of a non-reproducible paper and identifying statements most needing validation in a high-impact paper. We identify roles for librarians, database maintainers, knowledge base curators, and research software engineers in applying the framework to scientific digital libraries. doi:10.1145/3383583.3398514 Preprint: http://jodischneider.com/pubs/jcdl2020.pdf

Towards knowledge maintenance in scientific digital libraries with the keysto...

jodischneider

Keynote talk 2020-08-01 for the JCDL Workshop on Conceptual Models: https://sig-cm.github.io/news/JCDL-2020-CFP/ Discussion points: * Methods are a key part of the Knowledge Organizing Structure for Evidence-Based Medicine. * Methods relate to how we GENERATE evidence. * Different methods generate evidence of different kinds and strength. * I believe Methods can be useful in mining claims and arguments from papers: methods AUTHORIZE claims. * More specialized hierarchies of evidence can be found in medicine * Various groups are complicating the “evidence pyramid” hierarchy of evidence.

Methods Pyramids as an Organizing Structure for Evidence-Based Medicine--SIGC...

jodischneider

Annotation examples. This is an overview of some of the software I have used for annotation (and a few extra features some of this software has.) This was presented in the SwissUniversities Doctoral Programme, Language & Cognition, in the Module: Linguistic and corpus perspectives on argumentative discourse. Screenshots are given of GATE, UAM Corpus Tool, Excel, BRAT, EPPI Reviewer, and a custom tool. In most cases there are references to one of my papers for further details. I briefly describe a typical annotation process: Find text of interest Find phenomena of interest Draft an annotation manual Iteratively test annotation & revise manual Find questionable annotations, check disagreements. Revise the manual. Iterate. Annotate

Annotation examples--Fribourg--2019-09-03

jodischneider

An introduction to argumentation mining for PhD students. This was presented in the SwissUniversities Doctoral Programme, Language & Cognition, in the Module: Linguistic and corpus perspectives on argumentative discourse. The presentation largely follows Chapters 1-4 and Chapter 10 of my book, Argumentation Mining, co-authored with Manfred Stede in the Synthesis Lectures on Human Language Technologies from Morgan & Claypool: https://doi.org/10.2200/S00883ED1V01Y201811HLT040 Topics: My book w/computational linguist Manfred Stede: Argumentation Mining What is argumentation? Argumentation mining: a first look Argumentative language Challenges for argumentation mining Argumentation structures Corpus annotation Why study argumentation mining?

Argumentation mining--an introduction for linguists--Fribourg--2019-09-02

jodischneider

Talk at the 3rd European Conference on Argumentation ABSTRACT: Specialized fields may at any time invent new inference rules—that is, new warrants—to improve on their stock of resources for drawing and defending conclusions. Yet disagreement over the acceptability of an invented warrant can always be re-opened. Randomized Clinical Trial is widely regarded as the gold standard for making inferences about causal relationships between medical treatments and patient outcomes. Once controversial, RCT achieved broad acceptance within the field as a result of warrant-establishing arguments circulating in the medical literature starting in the 1950s. And RCT has accumulated a very impressive track record of generating new conclusions that withstand critical scrutiny. Here we look at two emerging innovations whose purpose is to support reasoning about health, offering ways to generate different classes of conclusions. These innovations could be seen as complementary to RCTs, but for both there are also hints of challenge to the enormous prestige of RCTs. We see this most particularly in the gap that has developed between the RCT-generated fact base and the decisions doctors and health policy officials have to make about treatments for patients. We’ve mentioned before that specialized inference methods that become stabilized within an expert community can meet unexpected challenges when they become components of reasoning by other communities. The two innovations considered here each allow us to explore the tensions that arise from the contrasting perspectives of scientists, clinicians, and patients.

Beyond Randomized Clinical Trials: emerging innovations in reasoning about he...

jodischneider

Publishers are caretakers of science. Part of that work is maintaining the integrity of scientific literature. Science builds directly upon past work, so we need to be sure that we are building upon a solid foundation and not faulty research. Publishers need to take an active role in monitoring and tracking faulty, retracted research and its influence. I'm asking publishers to (1) clearly mark retracted papers; (2) alert authors who have already cited a retracted paper; and (3) before publishing an article, check its bibliography for retracted papers. Retracted papers should be clearly marked everywhere they appear, but today that is not the case. Publishers can also use the CrossRef CrossMark service, which lets readers check for article updates (such as retraction) from a little red ribbon at the top of an article. Checking for citations to retracted articles, and limiting future citations, can help science self-correct by shoring up its foundations.

Problem-citations--CrossrefLive18--2018-11-13

jodischneider

The structure of citation networks provides evidence about how scientific information is diffused. Problematic citation patterns include the selective citation of positive findings, citation bias, as well as the continued citation of retracted literature (i.e. literature formally withdrawn due to error, fraud, or ethical problems). For instance, there is some evidence that positive results tend to receive more citations. The public domain licensing of the Open Citations Corpus makes it possible, in principle, to estimate the likelihood that any network of research papers suffers from problematic citation. To-date, problematic citation been documented ad-hoc, in several striking studies. In Alzheimer's disease research, biased citation, ignoring critical findings, was used to support successful U.S. NIH grant proposals (Greenberg 2009). Mistranslation of obesity research has been used to justify exertion game research (Marshall & Linehan 2017). Citation of fraudulent research about Chronic Obstructive Pulmonary Disease continued after its retraction (Fulton et al. 2015). The data resulting from such studies is of great use to my lab in replicating and determining how to generalize the detection of problematic citation patterns. Previously, the detection of problematic citation patterns has been a side effect of astute researchers, noticing suspicious findings while conducting systematic literature reviews. This talk will describe work-in-progress in my lab detecting problematic citation patterns using natural language processing, combined with network analysis on the Open Citations Corpus.

Problematic citations--Workshop-on-Open-Citations--2018-09-03

jodischneider

Presentation: Jodi Schneider and Novejot Sandhu, “Modeling Alzheimer’s Disease Reseach Claims, Evidence, and Arguments from a Biology Research Paper.” 9th International Conference on Argumentation, International Society for the Society of Argumentation, Amsterdam, Netherlands, July 5, 2018 Abstract: Argument visualization may help make research papers easier to understand, which could both speed quality assessment within a discipline and help build interdisciplinary knowledge networks. This paper presents a case study of the arguments in a single high-profile paper on Alzheimer's disease research. Within this one paper, we analyze and hand-annotate the main claim, which is supported by 4 subclaims, in turn supported by data, methods, and materials. We also investigate how the paper imports and uses knowledge claims from other research papers. We create a specialized argument-based knowledge representation called a micropublication. In future work, we will investigate automatic argumentation mining for experimental biology research papers. Our long-term vision is to create literature-scale claim-argument networks that help more quickly use new knowledge about human health.

Modeling Alzheimer’s Disease research claims, evidence, and arguments from a ...

jodischneider

Presentation: Jodi Schneider and Sally Jackson, “Innovations in Reasoning About Health: The Case of the Randomized Clinical Trial.” 9th International Conference on Argumentation, International Society for the Society of Argumentation, Amsterdam, Netherlands, July 5, 2018 Abstract: Field-dependence in argumentation comes about through forms of inference invented by specialized fields. In recent work we introduced the concept of a "warranting device": (1) an inference license (2) invented for a specialized argumentative purpose and (3) backed by institutional, procedural, and material assurances of the dependability of conclusions generated by the device. Once established, fields employ such devices across many situations without further defense, even as the devices develop in response to newly-noticed problems. Many new warranting devices have appeared over the past century to solve problems in reasoning about health and medicine, replacing and obsolescing earlier forms of medical reasoning. One such device is the Randomized Controlled Trial. This case study traces its historical evolution and discusses some current movements toward competing device types.

Innovations in reasoning about health: the case of the Randomized Clinical Tr...

jodischneider

Viewing universities as landscapes of scholarship, VIVO keynote, 2017-08-04

jodischneider

European Conference on Argumentation talk Jodi Schneider, Graciela Rosemblat, Shabnam Tafreshi and Halil Kilicoglu “Rhetorical moves and audience considerations in the discussion sections of Randomized Controlled Trials of health interventions” [Conference Panel Presentation], 2nd European Conference on Argumentation: Argumentation and Inference, Fribourg, Switzerland, June 20-23 1 of 3 talks in Jodi Schneider and Sally Jackson, organizers, “Innovations in Reasoning and Arguing about Health ”[Conference Panel], 2nd European Conference on Argumentation: Argumentation and Inference, Fribourg, Switzerland, June 20-23.

Rhetorical moves and audience considerations in the discussion sections of ra...

jodischneider

Citation practices and the construction of scientific fact--ECA-facts-preconf...

jodischneider

What WikiCite can learn from biomedical citation networks--Wikicite2017--2017...

jodischneider

Medication safety as a use case for argumentation mining We present a use case for argumentation mining, from biomedical informatics, specifically from medication safety. Tens of thousands of preventable medical errors occur in the U.S. each year, due to limitations in the information available to clinicians. Current knowledge sources about potential drug-drug interactions (PDDIs) often fail to provide essential management recommendations and differ significantly in their coverage, accuracy, and agreement. The Drug Interaction Knowledge Base Project (Boyce, 2006-present; dikb.org) is addressing this problem. Our current work is using knowledge representations and human annotation in order to represent clinically-relevant claims and evidence. Our data model incorporates an existing argumentation-focused ontology, the Micropublications Ontology. Further, to describe more specific information, such as the types of studies that allow inference of a particular type of claim, we are developing an evidence-focused ontology called DIDEO--Drug-drug Interaction and Drug-drug Interaction Evidence Ontology. On the curation side, we will describe how our research team is hand-extracting knowledge claims and evidence from the primary research literature, case reports, and FDA-approved drug labels for 65 drugs. We think that medication safety could be an important domain for applying automatic argumentation mining in the future. In discussions at Dagstuhl, we would like to investigate how current argumentation mining techniques might be used to scale up this work. We can also discuss possible implications for representing evidence from other biomedical domains. Talk for Dagstuhl Seminar 16161: Natural Language Argumentation: Mining, Processing, and Reasoning over Textual Arguments http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=16161

Medication safety as a use case for argumentation mining, Dagstuhl seminar 16...

jodischneider

Acquiring and representing drug-drug interaction knowledge and evidence, Litm...

jodischneider

Acquiring and representing drug-drug interaction knowledge and evidence, TRIA...

jodischneider

A talk for the Moore Institute for Humanities - People and documents are of enduring interest. Documents may be generated by individuals, collective groups, and administrations, on any number of topics. We are particularly interested in the relationships between people and documents. The most important relationships are creation (authors, illustrators, translators, ...), usage (e.g. association copies), and topic-of (e.g. people may be the subjects of biographies). In this lecture, we will talk about several approaches for modeling, or representing, people and documents. We pay particular attention to computer-based approaches to organization, and to organizing information for websites. We will talk briefly about TEI and XML, and the focus on my area of research expertise: modeling "linked data", a widely adopted approach for interlinking data. Adopted by the UK and US governments and search engines such as Google and Yahoo!, linked data has also been widely used in the digital humanities and by libraries, archives, and museums. It consists in naming objects of interest (be they authors, documents, or whatnot) and using standard data formats to enable interlinking.

Persons, documents, models: organising and structuring information for the We...

jodischneider

More from jodischneider (20)