#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015

•

4 likes•969 views

Achieving Expert-Level Annotation Quality with CrowdTruth: The Case of Medical Relation Extraction. Anca Dumitrache, Lora Aroyo and Chris Welty. ==> http://ceur-ws.org/Vol-1428/

Anca Dumitrache, Lora Aroyo, Chris Welty
http://CrowdTruth.org
Achieving Expert-Level
Annotation Quality with the Crowd
The Case of Medical Relation Extraction
Biomedical Data Mining, Modeling & Semantic Integration
@ ISWC2015
#CrowdTruth @anouk_anca @laroyo @cawelty #BDM2I

•  Annotator disagreement is
signal, not noise.
•  It is indicative of the
variation in human
semantic interpretation of
signs
•  It can indicate ambiguity,
vagueness, similarity, over-
generality, etc,
as well as quality
CrowdTruth

http://CrowdTruth.org

•  Goals:
collecting a relation extraction
gold standard
improve the performance of a
relation extraction classifier
•  Approach:
crowdsource 900 medical
sentences
measure disagreement with
CrowdTruth metrics
train & evaluate classifier with
CrowdTruth score
CrowdTruth
for

medical
rela2on

extrac2on

http://CrowdTruth.org

RelEx
TASK
in
CrowdFlower

Pa2ents
with
ACUTE
FEVER
and
nausea
could
be
suﬀering

from
INFLUENZA
AH1N1

Is
ACUTE
FEVER
–
related
to
→
INFLUENZA
AH1N1?

h"p://CrowdTruth.org

1 1 1
Worker
Vector

h"p://CrowdTruth.org

1 1 1
1 1
1
1 1
1 1
1 1
1
1
1
0 1 1 0 0 4 3 0 0 5 1 0
Sentence
Vector

h"p://CrowdTruth.org

0.907,
p
=
0:007

0.844

Annota2on
Quality

of
Expert
vs.
Crowd
Annota2ons

h"p://CrowdTruth.org

0.907,
p
=
0:007

0.844

[0.6
-‐
0.8]
crowd
signiﬁcantly
out-‐performs
expert

with
max
in
0.907
F1
@
0.7
threshold

Annota2on
Quality

of
Expert
vs.
Crowd
Annota2ons

h"p://CrowdTruth.org

0.642,
p
=
0:016

0.638

Relex
CAUSE
Classiﬁer
F1

for
Crowd
vs.
Expert
Annota2ons

h"p://CrowdTruth.org

0.642,
p
=
0:016

0.638

crowd
provides
training
data
that
is
at
least
as
good

if
not
beEer
than
experts

Relex
CAUSE
Classiﬁer
F1

for
Crowd
vs.
Expert
Annota2ons

h"p://CrowdTruth.org

(crowd
with
pos./neg.
threshold
at
0.5)

h"p://CrowdTruth.org

Learning
Curves

Learning
Curves

(crowd
with
pos./neg.
threshold
at
0.5)

above
400
sent.:
crowd
consistently
over
baseline
&
single

above
600
sent.:
crowd
out-‐performs
experts

h"p://CrowdTruth.org

Learning
Curves
Extended

(crowd
with
pos./neg.
threshold
at
0.5)

h"p://CrowdTruth.org

Learning
Curves
Extended

(crowd
with
pos./neg.
threshold
at
0.5)

h"p://CrowdTruth.org

crowd
consistently
performs
beEer
than
baseline

#
of
Workers:
Impact
on
Sentence-‐Rela2on
Score

h"p://CrowdTruth.org

#
of
Workers:
Impact
on
Annota2on
Quality

only
54
sent.
had
15
or
more
workers

h"p://CrowdTruth.org

Experts
vs.
Crowd

in
Human
Annota2on

Overall
Comparison

•  91% of expert annotations covered by the crowd
•  expert annotators reach agreement only in 30%
•  most popular crowd vote covers 95% of this
expert annotation agreement

h"p://CrowdTruth.org

F1 Cost per
sentence
CrowdTruth 0.642 $0.66
Expert Annotator 0.638 $2.00
Single Annotator 0.492 $0.08
h"p://CrowdTruth.org

Expert
vs.
Crowd

in
Human
Annota2on

Cost
Comparison

•  crowd performs just as well as
medical experts
•  crowd is also cheaper
•  crowd is always available
•  using only a few annotators for
ground truth is faulty
•  min 10 workers/sentence are
needed for highest quality
annotations
•  CrowdTruth = a solution to Clinical
NLP Challenge:
•  lack of ground truth for training &
benchmarking
Experiments
proved
that:

http://CrowdTruth.org

#CrowdTruth @anouk_anca @laroyo @cawelty #BDM2I #ISWC2015
CrowdTruth.org
http://data.CrowdTruth.org/medical-relex

Excessive exposure of unprotected skin to sunlight results in sunburn and can also lead to photo-induced oxidation, inflammation, immunosuppression, aging and even carcinogenesis of skin cells. Pre-clinical studies show that typical dietary antioxidant, could reduce such damages. Astaxanthin is believed to protect the skin against UV-light photo-oxidation and the in vitro protective effect of astaxanthin against UV-induced photooxidation was stronger when compared with β-carotene and lutein. These findings suggest that astaxanthin has an excellent potential as an oral sun-protectant. L-Carnitine is Traditionally used as a nutritional supplement in applications such as Weight management programs, Promotion of heart health as well as Enhancement of exercise recovery. L-Carnitine Acts at a cellular level to deliver advanced skin health appearance benefits. Several studies shown that Astaxanthin & L-Carnitine Combination in Astashine silver capsule is supportive of skin health and in particular Contributes to skin strength and elasticity and thus Promotes maintenance of effective skin barrier to maintain healthy skin hydration.

Itinerario campeonato copa alcalde karate

Federación Puertorriqueña de Karate

Клинические испытания в гомеопатии: аргументы для скептиков

Vubuntu Vera

Doedag DA2020 Venlo 9 maart

KING

I will present the CrowdTruth (http://crowdtruth.org/) approach to performing relation extraction from medical data. CrowdTruth exploits inter-annotator disagreement as a useful signal, allowing us to evaluate data quality, such as ambiguity and vagueness at the sentence level, worker quality, and the quality of the target semantics. I will introduce a workflow for generating gold standard annotations for medical relation extraction through a series of crowdsourcing tasks. Then I will present an evaluation of the crowd data by comparing it with the current gold standard in medical relation extraction. The evaluation is performed by training a relation extraction classifier with both datasets, and comparing the results for F1 measure in a cross-validation experiment.

LFS302_Real-World Evidence Platform to Enable Therapeutic Innovation

Amazon Web Services

Historically, there has been an information asymmetry in pharmaceutical R&D where the biopharmaceutical companies had the deepest understanding and knowledge about their products and how they helped and interacted with patients. Now, there's new, real-world data that exists from regulators, health plans, government authorities, and patients, which is helping pharma companies to understand how their therapies and their innovations drive value and impact in patient populations. There are imperatives to leverage that data, create new partnerships in their ecosystem, and get access to that data in an ethical way to derive insights to both fuel innovation and drive discovery. In this session, you learn best practices from Deloitte and Celgene about strategy, operating models, and execution frameworks when implementing a real-world, evidence data platform.

10 Must Know Techniques for Managing Physician Relations in Today's Digital W...

Endeavor Management

cPNI for pni2.org

Harold van Garderen

Open Targets workshop at C4X in 2019

Denise Carvalho-Silva, PhD

CrowdTruth @DIR2015

Anca Dumitrache

ClearWeave White Paper.pdf

RyanCasey60

The Evidence-Based Organization: A Platform for Innovation

Jan Recker @ University of Hamburg

1530 track1 rosenbaum

Rising Media, Inc.

Fore FAIR ISMB 2019

Ian Fore

Evans-Metrics-that-Matter-Inside-Counsel-1.2015 (1)Gareth Evans

Heathcare Communicators Oregon Presentation

CFM Strategic Communications

Impact of Nursing Informatics on Patient Outcomes Care Efficiencies.docx

4934bk

Viewers also liked

Robotics and Embedded Systems

Ankan Naskar

ACTION RESEARCH

Parvathy V

SOCIAL LEARNING THEORY

Parvathy V

Генрі Форд

Marina Hybalo

Kyle cooper titile sequence

Rochelle777

[ETUDE] Les Francais, l'épargne et la retraite

AG2R LA MONDIALE

Viewers also liked (6)

Robotics and Embedded Systems

ACTION RESEARCH

SOCIAL LEARNING THEORY

Генрі Форд

Kyle cooper titile sequence

[ETUDE] Les Francais, l'épargne et la retraite

Similar to #CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015

CrowdTruth Tutorial: Using the Crowd to Understand Ambiguity

Anca Dumitrache

#CrowdTruth: Linked Data for Information Extraction @ISWC2015

Lora Aroyo

TRADELINE_2007_Academic Medical Center Conference

Upali Nanda

CrowdTruth for medical relation extraction - WAI talk

Anca Dumitrache

LFS302_Real-World Evidence Platform to Enable Therapeutic Innovation

Amazon Web Services

10 Must Know Techniques for Managing Physician Relations in Today's Digital W...

Endeavor Management

cPNI for pni2.org

Harold van Garderen

Open Targets workshop at C4X in 2019

Denise Carvalho-Silva, PhD

CrowdTruth @DIR2015

Anca Dumitrache

ClearWeave White Paper.pdf

RyanCasey60

The Evidence-Based Organization: A Platform for Innovation

Jan Recker @ University of Hamburg

1530 track1 rosenbaum

Rising Media, Inc.

Fore FAIR ISMB 2019

Ian Fore

Evans-Metrics-that-Matter-Inside-Counsel-1.2015 (1)Gareth Evans

Heathcare Communicators Oregon Presentation

CFM Strategic Communications

Impact of Nursing Informatics on Patient Outcomes Care Efficiencies.docx

4934bk

Top 25 location R&D HardwareCEB TalentNeuron

FHIR intro and background at HL7 Germany 2014

Ewout Kramer

Automated and Explainable Deep Learning for Clinical Language Understanding a...

Databricks

Closing the Gap: Bringing a Consumer-Like Experience to the Digital Workplace

Lucidworks

Similar to #CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015 (20)

CrowdTruth Tutorial: Using the Crowd to Understand Ambiguity

#CrowdTruth: Linked Data for Information Extraction @ISWC2015

TRADELINE_2007_Academic Medical Center Conference

CrowdTruth for medical relation extraction - WAI talk

LFS302_Real-World Evidence Platform to Enable Therapeutic Innovation

10 Must Know Techniques for Managing Physician Relations in Today's Digital W...

cPNI for pni2.org

Open Targets workshop at C4X in 2019

CrowdTruth @DIR2015

ClearWeave White Paper.pdf

The Evidence-Based Organization: A Platform for Innovation

1530 track1 rosenbaum

Fore FAIR ISMB 2019

Evans-Metrics-that-Matter-Inside-Counsel-1.2015 (1)

Heathcare Communicators Oregon Presentation

Impact of Nursing Informatics on Patient Outcomes Care Efficiencies.docx

Top 25 location R&D Hardware

FHIR intro and background at HL7 Germany 2014

Automated and Explainable Deep Learning for Clinical Language Understanding a...

Closing the Gap: Bringing a Consumer-Like Experience to the Digital Workplace

More from Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

Lora Aroyo

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

Lora Aroyo

Harnessing Human Semantics at Scale (updated)

Lora Aroyo

Data excellence: Better data for better AI

Lora Aroyo

CHIP Demonstrator presentation @ CATCH Symposium

Lora Aroyo

Semantic Web Challenge: CHIP Demonstrator

Lora Aroyo

The Rijksmuseum Collection as Linked Data

Lora Aroyo

Presentation at ISWC2018: http://iswc2018.semanticweb.org/sessions/the-rijksmuseum-collection-as-linked-data/ of our paper published originally in the Semantic Web Journal: http://www.semantic-web-journal.net/content/rijksmuseum-collection-linked-data-2 Many museums are currently providing online access to their collections. The state of the art research in the last decade shows that it is beneficial for institutions to provide their datasets as Linked Data in order to achieve easy cross-referencing, interlinking and integration. In this paper, we present the Rijksmuseum linked dataset (accessible at http://datahub.io/dataset/rijksmuseum), along with collection and vocabulary statistics, as well as lessons learned from the process of converting the collection to Linked Data. The version of March 2016 contains over 350,000 objects, including detailed descriptions and high-quality images released under a public domain license.

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

Lora Aroyo

FAIRview: Responsible Video Summarization @NYCML'18

Lora Aroyo

Presentation at the NYC Media Lab (NYCML2018). There is a growing demand for news videos online, with more consumers preferring to watch the news than read or listen to it. On the publisher side, there is a growing effort to use video summarization technology in order to create easy-to-consume previews (trailers) for different types of broadcast programs. How can we measure the quality of video summaries and their potential to misinform? This workshop will inform participants about automatic video summarization algorithms and how to produce more “representative” video summaries. The research presented is from the FAIRview project and is supported by the Digital News Innovation Fund (DNI Fund), which is part of the Google News Initiative.

Understanding bias in video news & news filtering algorithms

Lora Aroyo

StorySourcing: Telling Stories with Humans & Machines

Lora Aroyo

Data Science with Humans in the Loop

Lora Aroyo

Digital Humanities Benelux 2017: Keynote Lora Aroyo

Lora Aroyo

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

Lora Aroyo

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

Lora Aroyo

The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to the volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, this assumption often creates issues in practice. Previous experiments we performed found that inter-annotator disagreement is usually never captured, either because the number of annotators is too small to capture the full diversity of opinion, or because the crowd data is aggregated with metrics that enforce consensus, such as majority vote. These practices create artificial data that is neither general nor reflects the ambiguity inherent in the data. To address these issues, we proposed the method for crowdsourcing ground truth by harnessing inter-annotator disagreement. We present an alternative approach for crowdsourcing ground truth data that, instead of enforcing an agreement between annotators, captures the ambiguity inherent in semantic annotation through the use of disagreement-aware metrics for aggregating crowdsourcing responses. Based on this principle, we have implemented the CrowdTruth framework for machine-human computation, that first introduced the disagreement-aware metrics and built a pipeline to process crowdsourcing data with these metrics. In this paper, we apply the CrowdTruth methodology to collect data over a set of diverse tasks: medical relation extraction, Twitter event identification, news event extraction and sound interpretation. We prove that capturing disagreement is essential for acquiring a high-quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with a majority vote, a method which enforces consensus among annotators. By applying our analysis over a set of diverse tasks we show that, even though ambiguity manifests differently depending on the task, our theory of inter-annotator disagreement as a property of ambiguity is generalizable.

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

Lora Aroyo

Ambiguity in interpreting signs is not a new idea, yet the vast majority of research in machine interpretation of signals such as speech, language, images, video, audio, etc., tend to ignore ambiguity. This is evidenced by the fact that metrics for quality of machine understanding rely on a ground truth, in which each instance (a sentence, a photo, a sound clip, etc) is assigned a discrete label, or set of labels, and the machine’s prediction for that instance is compared to the label to determine if it is correct. This determination yields the familiar precision, recall, accuracy, and f-measure metrics, but clearly presupposes that this determination can be made. CrowdTruth is a form of collective intelligence based on a vector representation that accommodates diverse interpretation perspectives and encourages human annotators to disagree with each other, in order to expose latent elements such as ambiguity and worker quality. In other words, CrowdTruth assumes that when annotators disagree on how to label an example, it is because the example is ambiguous, the worker isn’t doing the right thing, or the task itself is not clear. In previous work on CrowdTruth, the focus was on how the disagreement signals from low quality workers and from unclear tasks can be isolated. Recently, we observed that disagreement can also signal ambiguity. The basic hypothesis is that, if workers disagree on the correct label for an example, then it will be more diﬃcult for a machine to classify that example. The elaborate data analysis to determine if the source of the disagreement is ambiguity supports our intuition that low clarity signals ambiguity, while high clarity sentences quite obviously express one or more of the target relations. In this talk I will share the experiences and lessons learned on the path to understanding diversity in human interpretation and the ways to capture it as ground truth to enable machines to deal with such diversity.

Data Science with Human in the Loop @Faculty of Science #Leiden University

Lora Aroyo

Software systems are becoming ever more intelligent and more useful, but the way we interact with these machines too often reveals that they don’t actually understand people. Knowledge Representation and Semantic Web focus on the scientific challenges involved in providing human knowledge in machine-readable form. However, we observe that various types of human knowledge cannot yet be captured by machines, especially when dealing with wide ranges of real-world tasks and contexts. The key scientific challenge is to provide an approach to capturing human knowledge in a way that is scalable and adequate to real-world needs. Human Computation has begun to scientifically study how human intelligence at scale can be used to methodologically improve machine-based knowledge and data management. My research is focusing on understanding human computation for improving how machine-based systems can acquire, capture and harness human knowledge and thus become even more intelligent. In this talk I will show how the CrowdTruth framework (http://crowdtruth.org) facilitates data collection, processing and analytics of human computation knowledge. Some project links: - http://controcurator.org/ - http://crowdtruth.org/ - http://diveproject.beeldengeluid.nl/ - http://vu-amsterdam-web-media-group.github.io/linkflows/

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

Lora Aroyo

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Lora Aroyo

"Video Killed the Radio Star": From MTV to Snapchat

Lora Aroyo

More from Lora Aroyo (20)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning

Harnessing Human Semantics at Scale (updated)

Data excellence: Better data for better AI

CHIP Demonstrator presentation @ CATCH Symposium

Semantic Web Challenge: CHIP Demonstrator

The Rijksmuseum Collection as Linked Data

Keynote at International Conference of Art Libraries 2018 @Rijksmuseum

FAIRview: Responsible Video Summarization @NYCML'18

Understanding bias in video news & news filtering algorithms

StorySourcing: Telling Stories with Humans & Machines

Data Science with Humans in the Loop

Digital Humanities Benelux 2017: Keynote Lora Aroyo

DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...

Crowdsourcing ambiguity aware ground truth - collective intelligence 2017

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

Data Science with Human in the Loop @Faculty of Science #Leiden University

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

"Video Killed the Radio Star": From MTV to Snapchat

Recently uploaded

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

91mobiles

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

UiPathCommunity

In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni. 📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath: Autopilot per Studio Web Autopilot per Studio Autopilot per Apps Clipboard AI GenAI applicata alla Document Understanding 👨‍🏫👨‍💻 Speakers: Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath Andrei Tasca, RPA Solutions Team Lead @NTT Data

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Thierry Lestable

Bits & Pixels using AI for Good.........

Alison B. Lowndes

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

nkrafacyberclub

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Peter Spielvogel

Building better applications for business users with SAP Fiori. • What is SAP Fiori and why it matters to you • How a better user experience drives measurable business benefits • How to get started with SAP Fiori today • How SAP Fiori elements accelerates application development • How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities • How SAP Fiori paves the way for using AI in SAP apps

Accelerate your Kubernetes clusters with Varnish Caching

Thijs Feryn

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Free Complete Python - A step towards Data Science

RinaMondal9

Monitoring Java Application Security with JDK Tools and JFR Events

Ana-Maria Mihalceanu

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

FIDO Alliance Osaka Seminar: Overview.pdf

FIDO Alliance

Recently uploaded (20)

Leading Change strategies and insights for effective change management pdf 1.pdf

Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf

When stars align: studies in data quality, knowledge graphs, and machine lear...

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

Elevating Tactical DDD Patterns Through Object Calisthenics

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Empowering NextGen Mobility via Large Action Model Infrastructure (LAMI): pav...

Bits & Pixels using AI for Good.........

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Accelerate your Kubernetes clusters with Varnish Caching

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Free Complete Python - A step towards Data Science

Monitoring Java Application Security with JDK Tools and JFR Events

UiPath Test Automation using UiPath Test Suite series, part 3

FIDO Alliance Osaka Seminar: Overview.pdf

#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015

1. Anca Dumitrache, Lora Aroyo, Chris Welty http://CrowdTruth.org Achieving Expert-Level Annotation Quality with the Crowd The Case of Medical Relation Extraction Biomedical Data Mining, Modeling & Semantic Integration @ ISWC2015 #CrowdTruth @anouk_anca @laroyo @cawelty #BDM2I

2. •  Annotator disagreement is signal, not noise. •  It is indicative of the variation in human semantic interpretation of signs •  It can indicate ambiguity, vagueness, similarity, over- generality, etc, as well as quality CrowdTruth http://CrowdTruth.org

3. •  Goals: collecting a relation extraction gold standard improve the performance of a relation extraction classifier •  Approach: crowdsource 900 medical sentences measure disagreement with CrowdTruth metrics train & evaluate classifier with CrowdTruth score CrowdTruth for medical rela2on extrac2on http://CrowdTruth.org

4. RelEx TASK in CrowdFlower Pa2ents with ACUTE FEVER and nausea could be suﬀering from INFLUENZA AH1N1 Is ACUTE FEVER – related to → INFLUENZA AH1N1? h"p://CrowdTruth.org

5. 1 1 1 Worker Vector h"p://CrowdTruth.org

6. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 4 3 0 0 5 1 0 Sentence Vector h"p://CrowdTruth.org

7. 0.907, p = 0:007 0.844 Annota2on Quality of Expert vs. Crowd Annota2ons h"p://CrowdTruth.org

8. 0.907, p = 0:007 0.844 [0.6 -‐ 0.8] crowd signiﬁcantly out-‐performs expert with max in 0.907 F1 @ 0.7 threshold Annota2on Quality of Expert vs. Crowd Annota2ons h"p://CrowdTruth.org

9. 0.642, p = 0:016 0.638 Relex CAUSE Classiﬁer F1 for Crowd vs. Expert Annota2ons h"p://CrowdTruth.org

10. 0.642, p = 0:016 0.638 crowd provides training data that is at least as good if not beEer than experts Relex CAUSE Classiﬁer F1 for Crowd vs. Expert Annota2ons h"p://CrowdTruth.org

11. (crowd with pos./neg. threshold at 0.5) h"p://CrowdTruth.org Learning Curves

12. Learning Curves (crowd with pos./neg. threshold at 0.5) above 400 sent.: crowd consistently over baseline & single above 600 sent.: crowd out-‐performs experts h"p://CrowdTruth.org

13. Learning Curves Extended (crowd with pos./neg. threshold at 0.5) h"p://CrowdTruth.org

14. Learning Curves Extended (crowd with pos./neg. threshold at 0.5) h"p://CrowdTruth.org crowd consistently performs beEer than baseline

15. # of Workers: Impact on Sentence-‐Rela2on Score h"p://CrowdTruth.org

16. # of Workers: Impact on Annota2on Quality only 54 sent. had 15 or more workers h"p://CrowdTruth.org

17. Experts vs. Crowd in Human Annota2on Overall Comparison •  91% of expert annotations covered by the crowd •  expert annotators reach agreement only in 30% •  most popular crowd vote covers 95% of this expert annotation agreement h"p://CrowdTruth.org

18. F1 Cost per sentence CrowdTruth 0.642 $0.66 Expert Annotator 0.638 $2.00 Single Annotator 0.492 $0.08 h"p://CrowdTruth.org Expert vs. Crowd in Human Annota2on Cost Comparison

19. •  crowd performs just as well as medical experts •  crowd is also cheaper •  crowd is always available •  using only a few annotators for ground truth is faulty •  min 10 workers/sentence are needed for highest quality annotations •  CrowdTruth = a solution to Clinical NLP Challenge: •  lack of ground truth for training & benchmarking Experiments proved that: http://CrowdTruth.org

20. #CrowdTruth @anouk_anca @laroyo @cawelty #BDM2I #ISWC2015 CrowdTruth.org http://data.CrowdTruth.org/medical-relex

#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (6)

Similar to #CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015

Similar to #CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015 (20)

More from Lora Aroyo

More from Lora Aroyo (20)

Recently uploaded

Recently uploaded (20)

#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2015) @ISWC2015