Watson at RPI - Summer 2013

1. TECHNICAL PROJECT REVIEW 14 TH AUGUST, 2013 WATSON @ RPI WATSON RESEARCH LAB PROFESSOR JIM HENDLER SIMON ELLIS KATE MCGUIRE  NICOLE NEGEDLY DILLON BURNS  MATT KLAWONN AVI WEINSTOCK

2. WATSON RPI Simon Ellis INTRODUCTION

3. ???IBM Watson

4. ???Watson is…  … a piece of software that will run on your laptop  Though very slowly  Specialised hardware and control platform  … an implementation of the DeepQA concept  … the first iteration of the „cognitive computing‟ platform  … a very clever artificial intelligence  A very clever application of human intelligence

5. ???Background  IBM agrees to give RPI a version of Watson  Watson team is set up to undertake summer research on the Watson system  Watson hardware/software configuration not ready at beginning of summer session  So what do we do with: 10 weeks, 5 undergraduates and 1 graduate…

6. ???Challenge accepted!  Build a new version of Watson  Based on research published in IBM J Res & Dev  With support and input from IBM Research  Use open source libraries wherever possible  Faster development  No IP issues  Turns out to be a very useful project  Trains team in the details of the operation of Watson system  Can be used in education, training, testing, evaluation

7. ???Sample output  Demo run of RPI version of Watson  Shows output representing most of the “pipeline”

8. ???Inside Watson Watson pipeline as published by IBM; see IBM J Res & Dev 56 (3/4), May/July 2012, p. 15:2

9. WATSON RPI Nicole Negedly QUESTION ANALYSIS

10. ???Question Analysis

11. ???Question Analysis  What is the question asking for?  What structured information can be determined from the unstructured text of the question?  Topics  Parsers  Syntactic and Semantic Analysis Tools  Focus and Lexical Answer Type Detection  Future Work

12. ???Parsing  Open-source parsers  Stanford Parser  Berkeley Parser  Functions  Determine grammatical structure of text  Parse trees, part-of-speech Tags, dependency relations

13. ???Coreference Resolution  What terms in the question refer to the same entity?

14. ???Named Entity Extraction  Identifies people, places, organizations, and time spans.

15. ???Focus and Lexical Answer Type POETS & POETRY He was a bank clerk in the Yukon before he published Songs of a Sourdough in 1907  Focus: “he”  LAT: “he”, “clerk”, “poet”

16. ???Future Work  Adding additional parsers to the system  Comparison of parser output  Relation extraction  Prolog code and database  Improved focus and LAT detection  Princeton WordNet

17. WATSON RPI Dillon Burns PRIMARY SEARCH

18. ???Primary Search

19. ???Primary Search & Corpus Generation  Primary search is used to generate our corpus of information from which to take candidate answers, passages, supporting evidence, and essentially all textual input to the system.  Search Wikipedia for the focus identified during the Question Analysis phase.  Grab first 5 documents returned back as corpus.  Uses Jsoup library to collect and parse HTML.

20. ???JSoup String[] results = {“/wiki/Snapple”,”/wiki/Dr_Pepper_Snapple_Group”,”/wiki/Snapple_Theater….”

21. ???JSoup String[] results = {“/wiki/Snapple”,”/wiki/Dr_Pepper_Snapple_Group”,”/wiki/Snapple_Theater….” To Cache

22. ???DBpedia

23. ???DBpedia  As of 2011 it had 3.64 million things categorized in its database  URLs are a direct map to Wikipedia‟s  Wikipedia redirect lists help with alternate names for entities and closely related concepts to certain entities or people

24. ???Future Directions  Use DBpedia to fact-check answers about entities in the database  Making use of the DBpedia subject matching

25. WATSON RPI Kate McGuire CANDIDATE GENERATION

26. ???Search Result Processing and Candidate Generation

27. ???Search Result Processing and Candidate Generation

28. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile-2">[2]</a><a href="#cite_note-AAAS-3">[3]</a> <div id="toc" class="toc"> <div id="toctitle"> <h2>Contents</h2> </div> <ul> <li class="toclevel-1 tocsection-1"><a href="#Early_life_and_schooling">1 Early life and schooling</a></li> <li class="toclevel-1 tocsection-2"><a href="#Career">2 Career</a> <ul> <li class="toclevel-2 tocsection-3"><a href="#Rensselaer_Polytechnic_Institute">2.1 Rensselaer Polytechnic Institute</a></li> </ul> </li> <li class="toclevel-1 tocsection-4"><a href="#Honors_and_distinctions">3 Honors and distinctions</a> <ul> <li class="toclevel-2 tocsection-5"><a href="#Boards_of_directors">3.1 Boards of directors</a></li> </ul> </li> <li class="toclevel-1 tocsection-6"><a href="#Personal">4 Personal</a></li> <li class="toclevel-1 tocsection-7"><a href="#References">5 References</a></li> <li class="toclevel-1 tocsection-8"><a href="#External_links">6 External links</a></li> </ul> </div> <h2>Early life and schooling[<a href="/w/index.php?title=Shirley_Ann_Jackson&action=edit&section=1" title="Edit section: Early life and schooling">edit source</a> | <a href="/w/index.php?title=Shirley_Ann_Jackson&veaction=edit&section=1" title="Edit section: Early life and schooling" class="mw-editsection-visualeditor">edit</a>]</h2> Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note-diaspora-4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note-diaspora-4">[4]</a>

29. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile- 2">[2]</a><a href="#cite_note-AAAS- 3">[3]</a> <div id="toc" class="toc"> <div id="toctitle"> <h2>Contents</h2> </div> <ul> <li class="toclevel-1 tocsection-1"><a href="#Early_life_and_schooling">1 Early life and schooling</a></li> <li class="toclevel-1 tocsection-2"><a href="#Career">2 Career</a> <ul> <li class="toclevel-2 tocsection-3"><a href="#Rensselaer_Polytechnic_Institute">2.1 Rensselaer Polytechnic Institute</a></li> </ul> </li> <li class="toclevel-1 tocsection-4"><a href="#Honors_and_distinctions">3 Honors and distinctions</a> <ul> <li class="toclevel-2 tocsection-5"><a href="#Boards_of_directors">3.1 Boards of directors</a></li> </ul> </li> <li class="toclevel-1 tocsection-6"><a href="#Personal">4 Personal</a></li> <li class="toclevel-1 tocsection-7"><a href="#References">5 References</a></li> <li class="toclevel-1 tocsection-8"><a href="#External_links">6 External links</a></li> </ul> </div> <h2>Early life and schooling[<a href="/w/index.php?title=Shirley_Ann_Jackson&action=edit&section=1" title="Edit section: Early life and schooling">edit source</a> | <a href="/w/index.php?title=Shirley_Ann_Jackson&veaction=edit&section=1" title="Edit section: Early life and schooling" class="mw-editsection-visualeditor">edit</a>]</h2> Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note-diaspora- 4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note-diaspora- 4">[4]</a>

30. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile- 2">[2]</a><a href="#cite_note-AAAS- 3">[3]</a> <div id="toc" class="toc">">edit</a></h2> Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note- diaspora-4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note-diaspora- 4">[4]</a>

31. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile- 2">[2]</a><a href="#cite_note-AAAS- 3">[3]</a>Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note-diaspora- 4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note- diaspora-4">[4]</a>

32. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile- 2">[2]</a><a href="#cite_note-AAAS- 3">[3]</a>Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note-diaspora- 4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note- diaspora-4">[4]</a>

33. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Text: Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>.

34. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Text: Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. Cleaned Text: Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>.

35. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Text: Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. Cleaned Text: Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>.

36. ???Search Result Processing  Passage Retrieval  Watson: Indri and Lucene  Identifies each HTML sentence and adds both the HTML and the clean text to the passage type  Adds information about each passage  Passage Parsing  Forms parse trees for each individual sentence  Add an array of passages to each document Text: Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. Cleaned Text: Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. Parse Tree: (ROOT (S (NP (NP (NNP ) (NNP Shirley) (NNP Ann) (NNP Jackson) (NNP )) (PRN (-LRB- -LRB-) (VP (VBN born) (NP (NNP August) (CD 5) (, ,) (CD 1946))) (-RRB- -RRB-))) (VP (VBZ is) (NP (NP (DT an) (JJ ) (NNP American) (NNP ) (NNP ) (NN physicist) (NNS )) (, ,) (CC and) (NP (NP (DT the) (JJ 18th) (NN president)) (PP (IN of) (NP (NNP ) (NNP Rensselaer) (NNP Polytechnic) (NNP Institute) (NNP )))))) (. .)))

37. ???Candidate Generation  Using each document, and the passages created by Search Result Processing, we generate candidates using three techniques: 1. Title of Document (T.O.D.): Adds the title of the document as a candidate. 2. Wikipedia Title Candidate Generation: Adds any noun phrases within the document‟s passage texts that are also the titles of Wikipedia articles. 3. Anchor Text Candidate Generation: Adds candidates based on the hyperlinks and metadata within the document.

38. ???Wikipedia Title Candidate Generation  Runs on the passage array from each search result.  Using the parse tree, retrieves all the noun phrases in each passage.  Checks if each Noun Phrase is the title of a Wikipedia Article  Adds the verified candidates along with an array of the passages that contained them Array of Passages Retrieving Noun Phrases Check against Previous Data Wikipedia URL Check Candidate and Containing Passages (ROOT (S (NP (NP (NNP ) (NNP Shirley) (NNP Ann) (NNP Jackson) (NNP )) (PRN (- LRB- -LRB-) (VP (VBN born) (NP (NNP August) (CD 5) (, ,) (CD 1946))) (-RRB- -RRB- ))) (VP (VBZ is) (NP (NP (DT an) (JJ ) (NNP American) (NNP ) (NNP ) (NN physicist) (NNS )) (, ,) (CC and) (NP (NP (DT the) (JJ 18th) (NN president)) (PP (IN of) (NP (NNP ) (NNP Rensselaer) (NNP Polytechnic) (NNP Institute) (NNP )))))) (. .))) b>Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>.

39. ???Wikipedia Title Candidate Generation  Runs on the passage array from each search result.  Using the parse tree, retrieves all the noun phrases in each passage.  Checks if each Noun Phrase is the title of a Wikipedia Article  Adds the verified candidates along with an array of the passages that contained them Array of Passages Retrieving Noun Phrases Check against Previous Data Wikipedia URL Check Candidate and Containing Passages (ROOT (S (NP (NP (NNP ) (NNP Shirley) (NNP Ann) (NNP Jackson) (NNP )) (PRN (- LRB- -LRB-) (VP (VBN born) (NP (NNP August) (CD 5) (, ,) (CD 1946))) (-RRB- -RRB- ))) (VP (VBZ is) (NP (NP (DT an) (JJ ) (NNP American) (NNP ) (NNP ) (NN physicist) (NNS )) (, ,) (CC and) (NP (NP (DT the) (JJ 18th) (NN president)) (PP (IN of) (NP (NNP ) (NNP Rensselaer) (NNP Polytechnic) (NNP Institute) (NNP )))))) (. .))) Shirley Ann Jackson Shirley Ann Jackson (born August 5, 1946) August 5, 1946 An American Physicist An American Physicist, and the 18th president of Rensselaer Polytechnic Institute The 18th president The 18th president of Rensselaer Polytechnic Institute Rensselaer Polytechnic Institute

40. ???Wikipedia Title Candidate Generation  Runs on the passage array from each search result.  Using the parse tree, retrieves all the noun phrases in each passage.  Checks if each Noun Phrase is the title of a Wikipedia Article  Adds the verified candidates along with an array of the passages that contained them Array of Passages Retrieving Noun Phrases Check against Previous Data Wikipedia URL Check Candidate and Containing Passages (ROOT (S (NP (NP (NNP ) (NNP Shirley) (NNP Ann) (NNP Jackson) (NNP )) (PRN (- LRB- -LRB-) (VP (VBN born) (NP (NNP August) (CD 5) (, ,) (CD 1946))) (-RRB- -RRB- ))) (VP (VBZ is) (NP (NP (DT an) (JJ ) (NNP American) (NNP ) (NNP ) (NN physicist) (NNS )) (, ,) (CC and) (NP (NP (DT the) (JJ 18th) (NN president)) (PP (IN of) (NP (NNP ) (NNP Rensselaer) (NNP Polytechnic) (NNP Institute) (NNP )))))) (. .))) Shirley Ann Jackson Shirley Ann Jackson (born August 5, 1946) August 5, 1946 An American Physicist An American Physicist, and the 18th president of Rensselaer Polytechnic Institute The 18th president The 18th president of Rensselaer Polytechnic Institute Rensselaer Polytechnic Institute

41. ???Wikipedia Title Candidate Generation  Runs on the passage array from each search result.  Using the parse tree, retrieves all the noun phrases in each passage.  Checks if each Noun Phrase is the title of a Wikipedia Article  Adds the verified candidates along with an array of the passages that contained them Array of Passages Retrieving Noun Phrases Check against Previous Data Wikipedia URL Check Candidate and Containing Passages http://en.wikipedia.org/w/index.php?title=Special%3ASearch&profile=default&search= Shirley+Ann+Jackson Shirley Ann Jackson Shirley Ann Jackson (born August 5, 1946) August 5, 1946 An American Physicist An American Physicist, and the 18th president of Rensselaer Polytechnic Institute The 18th president The 18th president of Rensselaer Polytechnic Institute Rensselaer Polytechnic Institute

42. ???Wikipedia Title Candidate Generation  Runs on the passage array from each search result.  Using the parse tree, retrieves all the noun phrases in each passage.  Checks if each Noun Phrase is the title of a Wikipedia Article  Adds the verified candidates along with an array of the passages that contained them Array of Passages Retrieving Noun Phrases Check against Previous Data Wikipedia URL Check Candidate and Containing Passages http://en.wikipedia.org/w/index.php?title=Special%3ASearch&profile=default&search= Shirley+Ann+Jackson Shirley Ann Jackson Shirley Ann Jackson (born August 5, 1946) August 5, 1946 An American Physicist An American Physicist, and the 18th president of Rensselaer Polytechnic Institute The 18th president The 18th president of Rensselaer Polytechnic Institute Rensselaer Polytechnic Institute

43. ???Anchor Text Candidate Generation  Runs on the passage array from each search result.  Checks for hyperlinks within the HTML text of each passage.  Adds the title of the hyperlinked article as a candidate  Adds each passage containing the candidate to an array Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile- 2">[2]</a><a href="#cite_note-AAAS- 3">[3]</a>Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note- diaspora-4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note-diaspora- 4">[4]</a>

44. ???Anchor Text Candidate Generation  Runs on the passage array from each search result.  Checks for hyperlinks within the HTML text of each passage.  Adds the title of the hyperlinked article as a candidate  Adds each passage containing the candidate to an array Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile- 2">[2]</a><a href="#cite_note-AAAS- 3">[3]</a>Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note- diaspora-4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note-diaspora- 4">[4]</a>

45. ???Search Result Processing Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile-2">[2]</a><a href="#cite_note-AAAS-3">[3]</a> <div id="toc" class="toc"> <div id="toctitle"> <h2>Contents</h2> </div> <ul> <li class="toclevel-1 tocsection-1"><a href="#Early_life_and_schooling">1 Early life and schooling</a></li> <li class="toclevel-1 tocsection-2"><a href="#Career">2 Career</a> <ul> <li class="toclevel-2 tocsection-3"><a href="#Rensselaer_Polytechnic_Institute">2.1 Rensselaer Polytechnic Institute</a></li> </ul> </li> <li class="toclevel-1 tocsection-4"><a href="#Honors_and_distinctions">3 Honors and distinctions</a> <ul> <li class="toclevel-2 tocsection-5"><a href="#Boards_of_directors">3.1 Boards of directors</a></li> </ul> </li> <li class="toclevel-1 tocsection-6"><a href="#Personal">4 Personal</a></li> <li class="toclevel-1 tocsection-7"><a href="#References">5 References</a></li> <li class="toclevel-1 tocsection-8"><a href="#External_links">6 External links</a></li> </ul> </div> <h2>Early life and schooling[<a href="/w/index.php?title=Shirley_Ann_Jackson&action=edit&section=1" title="Edit section: Early life and schooling">edit source</a> | <a href="/w/index.php?title=Shirley_Ann_Jackson&veaction=edit&section=1" title="Edit section: Early life and schooling" class="mw-editsection- visualeditor">edit</a>]</h2> Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note-diaspora-4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note-diaspora-4">[4]</a>

46. ???Search Result Processing Shirley Ann Jackson (born August 5, 1946) is an <a href="/wiki/United_States" title="United States">American</a> <a href="/wiki/Physicist" title="Physicist">physicist</a>, and the 18th president of <a href="/wiki/Rensselaer_Polytechnic_Institute" title="Rensselaer Polytechnic Institute">Rensselaer Polytechnic Institute</a>. She received her <a href="/wiki/Doctor_of_Philosophy" title="Doctor of Philosophy">Ph.D.</a> in physics from the <a href="/wiki/Massachusetts_Institute_of_Technology" title="Massachusetts Institute of Technology">Massachusetts Institute of Technology</a> in 1973, becoming the first <a href="/wiki/African_American" title="African American">African American</a> woman to earn a doctorate from MIT in nuclear physics.<a href="#cite_note-profile-2">[2]</a><a href="#cite_note-AAAS-3">[3]</a> <div id="toc" class="toc"> <div id="toctitle"> <h2>Contents</h2> </div> <ul> <li class="toclevel-1 tocsection-1"><a href="#Early_life_and_schooling">1 Early life and schooling</a></li> <li class="toclevel-1 tocsection-2"><a href="#Career">2 Career</a> <ul> <li class="toclevel-2 tocsection-3"><a href="#Rensselaer_Polytechnic_Institute">2.1 Rensselaer Polytechnic Institute</a></li> </ul> </li> <li class="toclevel-1 tocsection-4"><a href="#Honors_and_distinctions">3 Honors and distinctions</a> <ul> <li class="toclevel-2 tocsection-5"><a href="#Boards_of_directors">3.1 Boards of directors</a></li> </ul> </li> <li class="toclevel-1 tocsection-6"><a href="#Personal">4 Personal</a></li> <li class="toclevel-1 tocsection-7"><a href="#References">5 References</a></li> <li class="toclevel-1 tocsection-8"><a href="#External_links">6 External links</a></li> </ul> </div> <h2>Early life and schooling[<a href="/w/index.php?title=Shirley_Ann_Jackson&action=edit&section=1" title="Edit section: Early life and schooling">edit source</a> | <a href="/w/index.php?title=Shirley_Ann_Jackson&veaction=edit&section=1" title="Edit section: Early life and schooling" class="mw-editsection- visualeditor">edit</a>]</h2> Jackson was born in Washington D.C. Her parents, Beatrice and George Jackson, strongly valued education and encouraged her in school.<a href="#cite_note-diaspora-4">[4]</a> Her father spurred on her interest in science by helping her with projects for her science classes. At Roosevelt High School, Jackson attended accelerated programs in both math and science, and graduated in 1964 as valedictorian.<a href="#cite_note-diaspora-4">[4]</a>

47. ???Future Work  Search Result Processing  Improve methods of imitating Indri or Lucene passage retrieval without a corpus.  Create a passage score and rank.  Candidate Generation  Continue to improve the speed and quality of Candidate Generation  Research and implement Candidate Generation from Structured Sources (Prismatic, Answer Lookup)  Record and measure recall in comparison with Watson and other Question answering software.

48. WATSON RPI Matt Klawonn SCORING & RANKING

49. ???Scoring & Ranking

50. ???Differentiating between answers  Making sense of candidates  Filtering  Supporting Evidence Retrieval (SER)  Scoring (passage-based)

51. ???Scorers  Passage Term Match  Textual Alignment  Skip-Bigram  Each of these scores supportive evidence  These scores are then merged to produce a single candidate score

52. ???Passage Term Search  Question Terms Extracted  Passage is searched for those terms  Score calculated for that passage  Done per passage  “Where is Toronto?” “Where” “is” “Toronto”  “Toronto is in Southern Ontario” “Toronto is ”  Score = IDF(Toronto) + IDF(is)

53. ???Textual Alignment  Finds an optimal alignment of a question and a passage  Assigns “partial credit” for close matches  “Who is the President of RPI?”  Shirley Ann Jackson is the President of RPI.  Who is the President of RPI.

54. ???Skip-Bigram  Constructs a graph  Nodes represent terms (syntactic objects)  Edges represent relations  Extracts skip-bigrams  A skip-bigram is a pair of nodes either directly connected or which have only one intermediate node  Skip-bigrams represent close relationships between terms  Scores based on number of common skip-bigrams

55. ???Example  Who authored “The Good Earth”?  “Pearl Buck, author of the good earth…”

56. ???Future Directions  More algorithms  Logical form answer candidate scoring  Improved Type Coercion scoring  Begin implementing machine learning  Temporal/Spatial reasoning

57. WATSON RPI Avi Weinstock UIMA PIPELINE

58. ???UIMA  The DeepQA architecture is built on top of another architecture, UIMA (Unstructured Information Management Architecture).  A UIMA CAS (Common Analysis Structure) contains a contiguous block of data (normally text), and annotations, which contain start & end indexes into the data, and additional data (strings, integers, doubles, arrays, annotation references).

59. ???UIMA  CAS Multipliers output multiple CASes based on the data in the input CAS; this facilitates parallelization, which is the key to Watson‟s response time.

60. ???DeepQA Architecture  The DeepQA architecture, which both IBM Watson and RPI MiniDeepQA implement, is a QA (Question Answering) system that answers questions by generating as many potential answers as is practical, then filtering them with multiple evidence scorers in parallel.

61. ???Data cache  IBM Watson has a pre-processed corpus of information, generated automatically by a subset of the DeepQA pipeline from an enormous volume of raw text, which the remainder of the pipeline uses at question time.  As our system retrieves information from the internet on a per-question basis, it cannot (practically) process the whole corpus in advance.

62. ???Data cache  Since parsing the documents takes a large amount of time, in order to test/demonstrate the system, it is beneficial to store webpages and associated parses locally. This allows a question that has been asked before, and candidates that come up for multiple questions, to be processed faster.  As a side-benefit of the caching, if a website is temporarily down, its data can still be used (if it was not down at some point in the past).

63. ???Graphical User Interface  Towards the start of the project, we ran our system using the Document Analyzer (a UIMA-provided tool).  While it was useful, once we had the entire pipeline set up, testing the full system required more input than necessary.  Additionally, there wasn't a convenient way to display just the intended output, nor intermediate output at a level suitable for monitoring progress/giving demonstrations.

64. ???Graphical User Interface  The GUI addresses these concerns, and has additionally been extensively tweaked to be demonstration-friendly.

Watson at RPI - Summer 2013

Watson at RPI - Summer 2013

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(20)

Similar to Watson at RPI - Summer 2013

Similar to Watson at RPI - Summer 2013(20)

More from James Hendler

More from James Hendler(20)

Recently uploaded

Recently uploaded(20)

Watson at RPI - Summer 2013

Editor's Notes