Presentation on crowdsourcing disagreement on open-domain questions

1. CrowdsourcingDisagreement on Open-Domain QuestionsJuly 18th 2014
Crowdsourcing Disagreement
on
Open-Domain Questions
By Benjamin Timmermans,
Lora Aroyo (VU) & Chris Welty (IBM)

2. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
● Natural Language
● Generate and evaluate hypothesis
● Learn from evidence
Cognitive Systems

3. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Gold Standard
● Distant supervision incomplete (Min et al., 2013)
● Crowdsourcing ground-truth (Inel et al., 2013)
● Large diverse volume of annotations
● Crowd Truth

4. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Closed Questions
Is rice a grain? (Yes)
Are all snakes Brown? (No)
Is it hard to get into Stanford?
Is MO polar?
Do isotopes of the same element have the same atonic weight?

5. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Do isotopes of the same element have the same atonic weight?
1. 1913: Henry Moseley, working fromVan den Broek's earlier idea, introduces concept of atomic number to fix inadequacies of Mendeleev's periodic table, which had been based on atomic weight,. ;1913:Frederick Soddyproposes
theconcept of isotopes, that elements with thesamechemical properties may havediffering atomic weights.
2. A chemical element consists of atomswith a specific number of protonsin their nuclei, but different atomic weights owing to variationsin thenumber of neutrons. Atoms of thesameelement with differing atomic weights are
called isotopes.
3. A primary isotopeshift is thechangein chemical shift observed between spectra of different isotopes of thesameelement, such as proton and deuterium. Primary isotopeshiftsbetween 1H and 2Dand areusually less than 0.1
ppm.
4. A radioisotopedoes not "rot";it decays by turning intoanother isotopeof thesameelement-or even into another element entirely.
5. Although receiving considerablesupport it was eventually rejected when it was found that many elementshavenon-integral weights (e.g. chlorine: 35.453). Frederick Soddy in 1913had introduced theidea of isotopes; that is, the
samechemical element in different forms having differing weights. Aston established thatisotopes arenot restricted to radioactiveelements but arecommonthroughouttheperiodic table.
6. An alternateform of an element that has theusual number of protons but a nonstandard number of neutrons; thefewer or additional neutronsgivetheisotopea different atomic weight than theregular element and may make
theisotoperadioactive, but otherwisean isotopehas thesamechemical actionas theregular element. Becauseof this, isotopes (such as radioactivecarbon) areused as tracers in biological systemsor processes.
7. An isotopeis oneof several kinds of atoms of thesameelement that havedifferent masses. Theseatoms havethesamenumber of protons in their nuclei, but different numbers of neutrons, and thereforedifferent mass
numbers.
8. As noted in the discussion of deuterium, tritiumcan only beseparated from protiumdueto thedifferences in mass. Thechemical properties of isotopes with thesameparent element makethem otherwiseindistinguishable, and
hence purely chemical means cannot beused to separatethem.
9. At theturn of thenineteenth century Dalton extended and refined Prout s remarkableconclusion thatcompounds wereof fixed composition by proposing thatatomsof thesameelement had thesameatomic weight. Today,
following thediscovery of isotopes in this century, wedefinean atomic weight for a blend of isotopes of an element as the ratio of the average mass per atom of theelement to one-twelfth themass of an atom of 12C
10. Atoms of the same atomic number but different atomic weights are called isotopes. Elements can
exist in both stable and unstable (radioactive) forms.
11. Atoms of thesameelement whosenuclei contain a different number of neutrons aresaid to be different isotopes of theelement. A pure element can exist as monatomic units or as diatomic or polyatomic unitscomprising the
samekind of atoms.
12. Isotopes areforms of the sameelement having different atomic weights
13. Atoms of thesameelement with differing atomic weights arecalled isotopes. Radioactivedecay is a spontaneous processin which an isotope(theparent) loses particles from its nucleusto form an isotopeof a new element (the
daughter)
14. By the way, an isotopemeans theforms of an element having different atomic weights becauseof thedifference in number of neutrons.
15. Carbon has atomic number 6 and atomic weight 12.011, and is represented by thesymbol C. It occurs in two different isotopes. Isotopes sharethesameatomic number, hencethesameidentity as elements and samechemical
behavior, but havedifferent atomic weights. Put differently, isotopes havethesamenumber of protonsbut a different number of neutrons. Theisotopes of carbon arecarbon-12 (six protons plussix neutrons) and carbon-14 (six
protons plus eight neutrons)
16. Deuterium is an isotopeof Hydrogen. An isotopeis any of two or moreforms of an element having thesameatomic number but with different weights (mass). Hydrogen has 2 isotopes, Deuterium (stable, i.e. non-radioactive) and
Tritium (radioactive)
17. Dr. W. D. Harkins, Professor of Physical Chemistryat theUniversity of Chicago, who earlier in theweek had announced his discovery of Zeta rays, described his original work with isotopes, or elements having thesamechemical
structurebut different atomic weights.
18. During this time, heworked with American chemist and physicist HaroldUrey at Columbia Universityon gaseous diffusiontechniquesfor theseparation of uranium isotopes (isotopes aredifferent forms of thesameelement
having thesameatomic number but different atomic weights)
19. During this time, heworked with American chemist and physicist HaroldUrey at Columbia Universityon gaseous diffusiontechniquesfor theseparation of uranium isotopes (isotopes aredifferent forms of thesameelement
having thesameatomic number but different atomic weights). After thewar, he accepted an appointment as a professor of chemistryat theUniversity of Chicago and began to conduct research at theInstituteof Nuclear Studies.
20. Fiveelements haveseven stableisotopes, eight havesix stableisotopes, ninehavefivestableisotopes, ninehavefour stable isotopes, ninehavethreestableisotopes, 16 havetwo stableisotopes (counting m as stable), and 26
elements haveonly a singlestableisotope(of these, 19 are so-called mononuclidic elements, having a singleprimordial stable isotopethat dominates and fixes theatomic weight of thenatural element to high precision;
21. For an element with threenaturally occurring isotopes themethodis thesame: sum themasses of theisotopes weighted by atom-fraction.
22. For an element with threenaturally occurring isotopes themethodis thesame: sum themasses of theisotopes weighted by atom-fraction. This method of calculating theaveragemass takes into account therelativeabundance
of all of theisotopes of an element, so that this mass number always gives thesametotal number of atoms, for a natural sampleof any element.
23. He called atoms of thesecond group isotopes, atomsof thesameelement with different atomic weights. In any natural sample of an element, theremay beseveral types of isotopes. As a result, theatomic weight of an element
that was calculated by Berzelius was actually an averageof all theisotopeweights for that element. This was thereason that someelements did not fall into theright order on Mendeleev's Periodic Table--theaverageatomic
weight depended on how much of each kind of isotopewas present. Soddy suggested placing theelements in thePeriodic Tableby similarity of chemical reactionsand then numbering them in order.
24. He called such chemically identical elements, with slightly differing atomic weights, isotopes (from theGreek words meaning in thesameplace)
25. His assistant, Francis Aston, developed Thomson's instrumentfurther and with theimproved version was ableto discover isotopes-atomsof thesameelement with different atomic weights-in a largenumber of nonradioactive
elements.
26. In 1913, Richards proved theexistenceof lead isotopes (two or moreatoms of thesameelement that differ in atomic weight) by investigating theweight of lead from various sources.
27. In 1914 hereceived theNobel Prizein Chemistry for accurately determining theatomic weightsof morethan 25 chemical elementsand ascertaining theexistenceof isotopes, chemical elementsthat havethesameatomic
number and position in theperiodic tablebut different atomic masses and physical properties.
28. In a short letter to theeditor of Nature, published on December 4, 1913, hefirst proposed theterm isotopeto designatechemically identicalelements with different atomic weights (in modern terms, elements with thesame
atomic numbers but different mass numbers)
29. In a short letter to theeditor of Nature, published on December 4, 1913, hefirst proposed theterm isotopeto designatechemically identicalelements with different atomic weights (in modern terms, elements with thesame

6. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Machine way of thinking
Crowdsourcing Architecture

7. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
High amount of annotations
40
Answer Passages
3000
Questions
10
Workers
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
??????????????????????????????
0 - 20
Matching Terms

8. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Passage Alignment Template

9. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Result of Alignment
Answer Passage Terms
Question
Terms

10. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Annotated Relation Types

11. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Negation in Passages vs
Wiki Answers
Is MO polar?
o Yes
x No Question ID

12. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Passage Alignment not perfect
● No expertise required
● High variety and volume of annotations
● Difficult to find the answer
● Expensive at $0.06 per task for 10 workers for 120.000 passages =
$72.000

13. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Question Answering
Architecture

14. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Question Answering Template

15. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Question Answering Template

16. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Question Answering Template

17. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Answer Justification
Question ID
JustifyingPassages

18. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Reduce dataset
120.000 -10% -75% 27.000

19. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
AggregatedWorkerAnswer
Crowdsourcing Answers vs
Wiki Answers
Wiki Answer is Yes Wiki Answer is No
Is rex burkhead a senior?
Is MO polar?
Question IDQuestion ID
AggregatedWorkerAnswer

20. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Tracking low quality workers
No Answer Found
Unanswerable
Yes
No
Other Answer
Yes
No
Closed Question
Other AnswerOpen Question

21. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
AMT vs CF worker efficiency
Low Quality
Workers

22. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Question Answering Task
● No expertise required
● Judgements on:
● Answer Justification
● Question Type
● Question Answer
120.000 -10% -75% 27.000

23. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Future Work
● Gamification
● Domain specific expert annotators
● Improve preprocessing
● Relation tasks

24. July 18th 2014 CrowdsourcingDisagreement on Open-Domain Questions
Conclusions
● Crowd can answer open-domain questions
● Aligning passages via Crowdsourcing is expensive
● Crowd performs better on AMT than on CF