IBM Watson in the
Classroom
Wlodek Zadrozny (UNC Charlotte/formerly IBM Research)
Sean Gallagher (UNC Charlotte)
Watson Po...
Background papers
Similar work on Watson in the classroom
RPI
Columbia U.
UT Austin
CMU(?)
???
“Watson became possibly the first nonhuman
millionaire by besting its human competition”
Arguably, this event started the ...
The Jeopardy! Challenge: Solved in 2011
No replication of the solution as of 2015
Broad/Open
Domain
Complex
Language
High
...
Two Research Challenges:
Replication of IBM Watson performance
Understanding why Watson works.
Watson heterogeneity
is per...
Watson in the classroom: Spring 2014
 Motivation: teaching computer science using solved challenges
 Content: “Semantic ...
Teaching Objective: Learning IR and NLP
All students should learn all the technologies involved.
Grading should be based o...
The Resulting System
as of Spring 2014
Shaded components were more complete.
WatsonSim Components Changed
Watsonsim Accuracy
- Bing, Lucene and Indri as search
- Wikipedia, Wikiquotes, and Shakespeare as sources
- Using n-gram s...
Integrating contributions of different
teams was often challenging
- We logged over 3500 runs, recording accuracies
- The ...
Current Status: Individual Study
 Started the Watson MOOC
 Reading Watson papers
 Extending WatsonSim as a practicum fo...
Plans
 Cognitive computing class
 Use as a vehicle for individual study
and undergrad/MS research
 Research (proposal) ...
Two Research Challenges:
Replication of IBM Watson performance
Understanding why Watson works.
Deeper Research
Questions:
...
Why should the NLP researchers care?
 We have a mismatch between the academic/scientific
theory of meaning and NLU and wh...
A challenge in representing meaning
of text?
Schuetze 2013 argues that
“meaning is heterogeneous and
that semantic theory ...
Some questions for a formal model
of Watson?
 What is the role of search in computing meaning?
In Watson, formally speaki...
Other sample research topics
 Role of formal semantics and theorem proving
“the overall performance of QA systems is dire...
Conclusions
 Formal model of Watson might help in building more
realistic theories of natural language understanding
 Po...
Upcoming SlideShare
Loading in …5
×

Ibm cognitive seminar march 2015 watsonsim final

643 views

Published on

Wlodek Zadronzny's presentation for Cognitive Systems Institute Group Speaker Series March 12, 2015.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
643
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Ibm cognitive seminar march 2015 watsonsim final

  1. 1. IBM Watson in the Classroom Wlodek Zadrozny (UNC Charlotte/formerly IBM Research) Sean Gallagher (UNC Charlotte) Watson Polymath Ideas Valeria de Paiva (Nuance) Lawrence S. Moss (Indiana University) WatsonSim development Walid Shalaby Adarsh Avadhani and others
  2. 2. Background papers
  3. 3. Similar work on Watson in the classroom RPI Columbia U. UT Austin CMU(?) ???
  4. 4. “Watson became possibly the first nonhuman millionaire by besting its human competition” Arguably, this event started the era of “cognitive computing”
  5. 5. The Jeopardy! Challenge: Solved in 2011 No replication of the solution as of 2015 Broad/Open Domain Complex Language High Precision Accurate Confidence High Speed $600 In cell division, mitosis splits the nucleus & cytokinesis splits this liquid cushioning the nucleus $200 If you're standing, it's the direction you should look to check out the wainscoting. $2000 Of the 4 countries in the world that the U.S. does not have diplomatic relations with, the one that’s farthest north $1000 The first person mentioned by name in ‘The Man in the Iron Mask’ is this hero of a previous book by the same author. Based on a slide from IBM
  6. 6. Two Research Challenges: Replication of IBM Watson performance Understanding why Watson works. Watson heterogeneity is perfect for introducing students to IR and NLP Opportunity for MS and Undergrad Student Research Data Sets are Available on J-Archive Deeper Research Questions: We know how Watson works, but we don’t know why it works Watson Architecture was described in details in IBM J. R&D + patents
  7. 7. Watson in the classroom: Spring 2014  Motivation: teaching computer science using solved challenges  Content: “Semantic Technologies in IBM Watson” (provided by IBM)  Students (20): Upper level undergrad(1/3), MS(2/3), 1PhD  Since we didn’t have any code we decided to build a Watson simulator Students followed the idea of IBM Watson architecture, simplifying whenever possible, e.g. no UIMA  Used as a way to learn: Information Retrieval (Bing, Google, Lucene, Indri) Elements of machine learning (using Weka, logistic regression, ) Elements of NLP: why NLU is difficult, POS tagging, parsing (with OpenNLP) Data preparation: regular expressions, polite data crawling, etc. …
  8. 8. Teaching Objective: Learning IR and NLP All students should learn all the technologies involved. Grading should be based on the degree of mastery.
  9. 9. The Resulting System as of Spring 2014 Shaded components were more complete.
  10. 10. WatsonSim Components Changed
  11. 11. Watsonsim Accuracy - Bing, Lucene and Indri as search - Wikipedia, Wikiquotes, and Shakespeare as sources - Using n-gram scores, parse tree comparisons, LAT matching - SVM based score aggregation
  12. 12. Integrating contributions of different teams was often challenging - We logged over 3500 runs, recording accuracies - The peak is around 26.6% top accuracy - 36.6% for the top three candidates
  13. 13. Current Status: Individual Study  Started the Watson MOOC  Reading Watson papers  Extending WatsonSim as a practicum for individual studies  Adding new scorers  Adding question analyzers and classifiers  Adding new sources  Code available on github: https://github.com/SeanTater/uncc2014watsonsim
  14. 14. Plans  Cognitive computing class  Use as a vehicle for individual study and undergrad/MS research  Research (proposal) around the “why” question
  15. 15. Two Research Challenges: Replication of IBM Watson performance Understanding why Watson works. Deeper Research Questions: We know how Watson works, but we don’t know why it works Main points: 1. Despite IBM Watson’s success, we don’t know why it works? 2. Figuring it out can be an open collaborative project: Polymath style ?
  16. 16. Why should the NLP researchers care?  We have a mismatch between the academic/scientific theory of meaning and NLU and what technical experience seems to be telling us  Inadequate theories might be limiting our ability to make progress in NLP  There are interesting research questions, given Watson unorthodox approach to QA
  17. 17. A challenge in representing meaning of text? Schuetze 2013 argues that “meaning is heterogeneous and that semantic theory will always consist of distinct modules that are governed by different principles”  We don’t have a formal theory that would support this view  A formal model of Watson might be a starting point The(?) meaning is constructed in multiple steps
  18. 18. Some questions for a formal model of Watson?  What is the role of search in computing meaning? In Watson, formally speaking, it constrains the entities that end up in the correct discourse model. Can any collection of text be reorganized that way? Dynamically?  Semantics of the deferred type evaluation/deferred meaning computation?  Formal model of scoring? Meaning through interaction? Or intersection of constraints?  Is it Watson-like model fundamental? Or an engineering feat?
  19. 19. Other sample research topics  Role of formal semantics and theorem proving “the overall performance of QA systems is directly related to the depth of NLP resources” “the prover boosts the performance of the QA system on TREC questions by 30%” D.Moldovan in 2003 Similar results reported by MacCartney and Manning 2009  Could interactive theorem provers, e.g. Coq and HOL, be adapted to improve performance of QA systems, and NLP systems in general? (Same question for automated provers).  Role of natural logics
  20. 20. Conclusions  Formal model of Watson might help in building more realistic theories of natural language understanding  Polymath model might be useful in creating a formal theory/model of Watson  More realistic theories might help in building better NLP systems  Experiments with Watson-models are possible  Now, experiments with IBM Watson in the ‘cloud’ are possible  Watson can be the basis for teaching NLP

×