1. Microtask crowdsourcing for
disease mention annotation
in PubMed abstracts
Benjamin Good, Max Nanis, Andrew Su
The Scripps Research Institute
@bgood

2. • Rapid growth of text
Long term goal: improve
information extraction from text
2
• Existing computational
methods
- are not perfect
- need training data
pubs/year
>100/hour

3. Information Extraction
1. Find mentions of high level concepts in text
2. Map mentions to specific terms in ontologies
3. Identify relationships between concepts
3

4. Crowdsourcing
There is accumulating evidence that many
non-expert members of ‘the crowd’ can
read English well enough to help with many
information extraction tasks - even in
complex biomedical text
4 Zhai 2013, Aroyo 2013, Burger 2014

5. Microtask Crowdsourcing
• Distribute discrete units of work
(aka “human intelligence tasks” or
HITs) to many workers in parallel
who are paid to solve them.
5
Reported 500,000
registered workers in
2011 [1]
[1] Paritosh P, Ipeirotis P, Cooper M, Suri S: The computer is the new sewing
machine: benefits and perils of crowdsourcing. WWW '11 2011:325–326.

6. AMT, how it works
6
Requester Tasks
Amazon
For each task, specify:
• a qualification test
• how many workers per
task
• how much we will pay
per task
• in this case, a link to a
website that we host
where they can
complete the task.
Interact directly with
Amazon system
Manages:
• parallel execution of jobs
• worker access to tasks
via qualification tests
• payments
• task advertising
Workers

7. How well can AMT workers, in aggregate,
reproduce a gold standard disease mention
corpus within the text of PubMed abstracts?
7

8. Corpus used for comparison
NCBI Disease corpus
• 793 PubMed abstracts
• (100 development, 593 training, 100 test)
• 12 expert annotators (2 annotate each abstract)
6,900 “disease” mentions
8
Doğan, Rezarta, and Zhiyong Lu. "An improved corpus of disease mentions in PubMed citations." Proceedings of the 2012
Workshop on Biomedical Natural Language Processing. Association for Computational Linguistics.

9. Disease
Phrase is a disease IF:
• it can be mapped to a unique UMLS metathesaurus
concept in one of these semantic types
9
Doğan, Rezarta, and Zhiyong Lu. "An improved corpus of disease mentions in PubMed citations." Proceedings of the 2012
Workshop on Biomedical Natural Language Processing. Association for Computational Linguistics.
• and it contains information helpful to physicians

10. 10
• Specific Disease:
• “Diastrophic dysplasia”
• Disease Class:
• “Cancers”
• Composite Mention:
• “prostatic , skin , and lung cancer”
• Modifier:
• ..the “familial breast cancer” gene , BRCA2..
Disease
mentions

11. Instructions
• Task: You will be presented with text from the biomedical literature which we believe may help
resolve some important medical questions. The task is to highlight words and phrases in that
text which are diseases, disease groups, or symptoms of diseases. This work will help
advance research in cancer and many other diseases!
• Highlight all diseases and disease abbreviations !
• “...are associated with Huntington disease ( HD )... HD patients
received...”
• “The Wiskott-Aldrich syndrome ( WAS ) , an X-linked immunodeficiency…”
• Highlight the longest span of text specific to a disease !
• “... contains the insulin-dependent diabetes mellitus locus …”
• and not just ‘diabetes’.
• Highlight disease conjunctions as single, long spans.
• “... a significant fraction of familial breast and ovarian cancer , but
undergoes…”
• Highlight symptoms - physical results of having a disease!
• “XFE progeroid syndrome can cause dwarfism, cachexia, and microcephaly.
Patients often display learning disabilities, hearing loss, and visual impairment.
11

12. Qualification task: Q1
Select all and only the terms that should be
highlighted for each text segment:
12
1. “Myotonic dystrophy ( DM ) is associated with a ( CTG ) n trinucleotide repeat expansion in
the 3-untranslated region of a protein kinase-encoding gene , DMPK , which maps to
chromosome 19q13 . 3 . ”
• Myotonic
• dystrophy
• Myotonic dystrophy
• DM
• CTG
• trinucleotide repeat expansion
• kinase-encoding gene
• DMPK

13. Qualification task: Q2
13
2. “Germline mutations in BRCA1 are responsible for most cases of inherited breast
and ovarian cancer . However , the function of the BRCA1 protein has remained
elusive . As a regulated secretory protein , BRCA1 appears to function by a
mechanism not previously described for tumour suppressor gene products.”
• Germline mutations
• BRCA1
• breast
• ovarian cancer
• inherited breast and ovarian cancer
• cancer
• tumour
• tumour suppressor

14. Qualification task: Q3
14
3. “We report about Dr . Kniest , who first described the condition in 1952 , and his patient ,
who , at the age of 50 years is severely handicapped with short stature , restricted joint
mobility , and blindness but is mentally alert and leads an active life . This is in accordance
with molecular findings in other patients with Kniest dysplasia and…”
• age of 50 years
• severely handicapped
• short
• short stature
• restricted joint mobility
• blindness
• mentally alert
• molecular findings
• Kniest dysplasia
• dysplasia

15. Qualification task results
15
Threshold
for passing
33/194 passed
17%
Workers
qualified
workers

16. Tagging interface
16
Click to see instructions
Highlight
mentions

17. Experiment
17
Identify the disease mentions in the 593
abstracts from the NCBI disease corpus
• 6 cents per HIT
• HIT = annotate one abstract from PubMed
• 5 workers annotate each abstract

18. AMT, how it really works
18
Requester
Tasks
Amazon
Aggregation
function
Workers
http://www.thesheepmarket.com/

19. Increase precision with voting
19
1 or more votes (K=1)
This molecule inhibits the growth of a broad
panel of cancer cell lines, and is particularly
efficacious in leukemia cells, including
orthotopic leukemia preclinical models as well
as in ex vivo acute myeloid leukemia (AML)
and chronic lymphocytic leukemia (CLL)
patient tumor samples. Thus, inhibition of
CDK9 may represent an interesting approach
as a cancer therapeutic target especially in
hematologic malignancies.
K=2
This molecule inhibits the growth of a broad
panel of cancer cell lines, and is particularly
efficacious in leukemia cells, including
orthotopic leukemia preclinical models as well
as in ex vivo acute myeloid leukemia (AML)
and chronic lymphocytic leukemia (CLL)
patient tumor samples. Thus, inhibition of
CDK9 may represent an interesting approach
as a cancer therapeutic target especially in
hematologic malignancies.
K=3
This molecule inhibits the growth of a broad
panel of cancer cell lines, and is particularly
efficacious in leukemia cells, including
orthotopic leukemia preclinical models as well
as in ex vivo acute myeloid leukemia (AML)
and chronic lymphocytic leukemia (CLL)
patient tumor samples. Thus, inhibition of
CDK9 may represent an interesting approach
as a cancer therapeutic target especially in
hematologic malignancies.
K=4
This molecule inhibits the growth of a broad
panel of cancer cell lines, and is particularly
efficacious in leukemia cells, including
orthotopic leukemia preclinical models as well
as in ex vivo acute myeloid leukemia (AML)
and chronic lymphocytic leukemia (CLL)
patient tumor samples. Thus, inhibition of
CDK9 may represent an interesting approach
as a cancer therapeutic target especially in
hematologic malignancies.
Aggregation
function

20. Results 593 abstracts
compared to gold standard
• 7 days
• $192.90
• 17 workers
20
F = 0.81, k = 2

21. Inter-Annotator agreement among
experts, NCBI Disease corpus
21
Doğan, Rezarta, and Zhiyong Lu. "An improved corpus of disease mentions in PubMed citations." Proceedings of
the 2012 Workshop on Biomedical Natural Language Processing. Association for Computational Linguistics, 2012.
0.76
0.87
Average level
of agreement
between expert
annotators
(stage 1)

22. In aggregate, our worker ensemble is faster,
cheaper and as accurate as a single expert
annotator for this task
• experts had consistency (F) with other experts =
0.76.
• The turker ensemble had consistency with the
finalized standard = 0.81
22

23. Summary
• Some members of the crowd can tag “disease”
mentions in PubMed abstracts with comparable
accuracy to experts
• This was nontrivial to set up
• We can now generate disease mention
annotations at a rate of about 500 abstracts and
$150 per week
• Next step: mentions to concepts…
23

24. The Future
• It looks like, if we want to, we can have access
to much larger sets of annotated corpora than
ever before
• The annotations are different
• New ways of using and evaluating IE algorithms
are needed [1].
24
[1] Aroyo, Lora, and Chris Welty. Harnessing disagreement in crowdsourcing a relation
extraction gold standard. Tech. Rep. RC25371 (WAT1304-058), IBM Research, 2013.

25. Thanks
25
Max Nanis Andrew Su
Mechanical Turk Workers!
@bgood
bgood@scipps.edu

26. Try it yourself!
• GATE crowdsourcing plugin.
http://gate.ac.uk/wiki/crowdsourcing.html
• Or you can try our code at
https://bitbucket.org/sulab/mark2cure/
!
• And present your findings at the crowdsourcing
session at the Pacific Symposium on
Biocomputing January 2015, Big Island, Hawaii
26

27. Clarification…
• This is NOT a replacement for
professional annotators
• This IS a tool that could be used by
professional annotators
27

28. Related work
• [1] Zhai et al 2013, used similar protocol to tag medication
names in clinical trials descriptions. F = 0.88 compared to
gold standard
• [2] Burger et al, using microtask workers to identify
relationships between genes and mutations.
• [3] Aroyo & Welty, used workers to identify relations
between concepts in medical text.
28
[1] Zhai H. et al (2013) ”Web 2.0-Based Crowdsourcing for High-Quality Gold Standard
Development in Clinical Natural Language Processing” J Med Internet Res
[2] Burger, John, et al. (2014) "Hybrid curation of gene-mutation relations combining automated
extraction and crowdsourcing.” Mitre technical report
[3] Aroyo, Lora, and Chris Welty. Harnessing disagreement in crowdsourcing a relation extraction
gold standard. Tech. Rep. RC25371 (WAT1304-058), IBM Research, 2013.