The document outlines the capabilities and workings of IBM's Watson cognitive system, emphasizing its reliance on information retrieval, natural language processing, and supervised learning techniques. It discusses various processes such as ingestion of data, question analysis, query decomposition, and scoring methodologies for generating and ranking responses. Additionally, it highlights future developments of Watson and its application in closed domains based on specialized training sets.

1. Swiss Group for Artificial Intelligence and Cognitive Science
Intelligent Systems and Applications Workshop 2014, University of Basel
Watson Technical Deep Dive
© 2013 IBM Corporation
@RomeoKienzler, IBM Innovation Center Zurich

2. (part of) my role @IBM
● Accelerate cognitive computing
● In Switzerland
● Through
● Academia
● Startups/ISV's
● Cloud
Watson in the cloud: bit.ly/go4bluemix
2 © 2013 IBM Corporation

3. What Watson is not
● Search Engine
● Database System
● HAL9000
3 © 2013 IBM Corporation

4. 4 © 2013 IBM Corporation

5. What Watson is
● Cognitive System (Marketing)
● Combination of
● Information Retrieval
● NLP
● Structured + Unstructured Data !
● Runs on UIMA
● Based on supervised learning
5 © 2013 IBM Corporation

6. What is a parser?
● Annotate sentence with
● Tags
● Relationships
● Probabilistic (e.g. Stanford)
● Rule based
● (E) Slot Grammar
6 © 2013 IBM Corporation

7. Slot Grammar
● Simplified
● Lexicalist character
● High focus on words
● Low focus on structure
● Assign words to slots
I can resist everything except temptation
● Subject (I)
● Verb (can resist)
● Object (everything except temptation)
7 © 2013 IBM Corporation

8. PAS - Builder
● Predicate-Argument Structure
● Downstream to ESG
● Reduces complexity of ESG
“John opened Bill's door (with his key)
John's key opened Bill's door
Bill's door opened
Bill's door was opened (by John)”
OPEN (John door key)
| | |
Agent Theme Instrument
Many ESG trees reduce to same PAS
8 © 2013 IBM Corporation

9. Relationships
● Relationship Extractor
● Combination of
● Manual pattern specifications
~30 types, high precision
● Statistical methods
~7000 types, low precision
● SVM's on DBPedia/Wikipedia
9 © 2013 IBM Corporation

10. Relationships (2)
“The Screwtape Letters” from a senior devil to
an under devil are by this man better known for
children’s books
author(“this man”,“The Screwtape Letters”)
10 © 2013 IBM Corporation

11. Ingestion
11 © 2013 IBM Corporation

12. Ingestion
● Corpus creation
● Input format: TREC
(Text Retrieval Conference)
● Multiple HTML pages in one
HDFS file
● Parallel ingestion process
(LiteScale)
12 © 2013 IBM Corporation

13. Dictionary
● started w/ Wikipedia copus
● Keyword → Text structure
● Transformation of free text
● into Keyword → Text
● optimization objective
13 © 2013 IBM Corporation

14. Knowledge Expansion
● Follow links in content
● Identify content keywords and link
to new content
● → generate more content in
Keyword → Text form
14 © 2013 IBM Corporation

15. Question Analysis
15 © 2013 IBM Corporation

16. Question Analysis
● Named entity recognition
● Type identification /Extract focus
● ESG/PAS
● Relationship detection
16 © 2013 IBM Corporation

17. Question Analysis
1) Extract focus
2) Map to LAT
3) Broad Type classification
4) Detect if special handling is
needed (e.g. nested question)
17 © 2013 IBM Corporation

18. Query Decomposition
18 © 2013 IBM Corporation

19. Query Decomposition
● Keyword identification
● LAT (Lexical Answer Type)
● IBM Pat. US20120078890 for
confidence estimation of LAT
● optimization objective: choosing
keywords out of nontrivial set of
words based on ML
19 © 2013 IBM Corporation

20. Query Decomposition
In 1894 C.W. Post created his warm cereal
drink Postum in this Michigan city
● Focus: this Michigan City
● LAT: Michigan
● Keywords: 1894, C.W. Post,
created, warm, cereal, drink,
Postum, Michigan, City
20 © 2013 IBM Corporation

21. Query Decomposition
21 © 2013 IBM Corporation

22. Primary Search
22 © 2013 IBM Corporation

23. Primary Search
● Lucene and Indri search engine
● Preprocessing generated
keyword->text based documents
● Keyword associated with found
document added to candidate
answer list
23 © 2013 IBM Corporation

24. Hypothesis generation
24 © 2013 IBM Corporation

25. Supporting Evidence Retrieval
Unlike most sea animals, in the sea horse this pair
of sense organs can move independently of one
another
Question decomposition:
Which [sense organ] of [Sea Horse] move independently?
Hypothesis generation:
A Sea Horse can move its eyes independently.
A Sea Horse can move its ears independently.
A Sea Horse can move its skin independently.
A Sea Horse can move its nose independently.
A Sea Horse can move its tung independently.
25 © 2013 IBM Corporation

26. http://angelalmassey.com/SHC/about.html
26 © 2013 IBM Corporation

27. Supporting Evidence
● Generated Candidate Answer is
● ESG'd
● PAS'd
● searched against corpus
● LATs used to determine whether
a candidate answer is an
instance of the answer types
27 © 2013 IBM Corporation

28. Supporting Evidence
28 © 2013 IBM Corporation

29. Scoring
29 © 2013 IBM Corporation

30. Scoring
● Optimization objective
(confidence estimation framework)
● Relational (PRISMATIC, Dbpedia)
● Taxonomic,Geospacial
● Temporal, Source Reliability
● Gender, Name consistency
● Passage Support
● 30 Theory consistency
© 2013 IBM Corporation

31. Scoring challenges
● Feature significance different for
● Different questions
● Different question classes
● Very heterogeneous features
● Normalization problem
● Missing features
● Class imbalance
31 © 2013 IBM Corporation

32. Merging and ranking
32 © 2013 IBM Corporation

33. Merging and ranking
1. John Fitzgerald Kennedy 2. Kennedy, 3. JFK
● Different Scores
● Merge to canonical form
● Morphological
● Pattern-based
● Table Lookup
● Partially generated from Wikipedia
disabiguation pages
33 © 2013 IBM Corporation

34. Example
MYTHING IN ACTION: One legend says this
was given by the Lady of the Lake & thrown
back in the lake on King Arthur’s death.
●Watson merged sword + Excalibur
to “sword” (canonical form)
● Preserved relation
● more_specific(sword)->Excalibur
34 © 2013 IBM Corporation

35. ML in Ranking
● Experiments with logistic regression, support
vector machines, linear and nonlinear
kernels, ranking SVM, boosting, single and
multilayer neural nets, decision trees, locally
weighted learning
● Finally:
regularized logistic regression
35 © 2013 IBM Corporation

36. Normalization
● Q set of all candidate answers
● Feature x_ij
● j feature, i answer
● missing values imputed
36 © 2013 IBM Corporation

37. Ranking
● Based on training set n > 10K
● IBM SPSS Modeler
37 © 2013 IBM Corporation

38. Evidence Sources
38 © 2013 IBM Corporation

39. Automatic Learning
● Read through text semantically
● Statistically rank annotated text
● generate new knowledge
● Inventors patent inventions 0.8
● officials submit resignations 0.7
● people earn degrees at schools 0.9
● fluid is a liquid 0.6
● liquid is a fluid 0.5
● vessels sink 0.7
● people sink 8-balls (0.5) (in pool/0.8)
39 © 2013 IBM Corporation

40. Next steps
● “Jeopardy!” - Watson was
● Open domain
● Large training set
● New “Watsons” are
● Closed domain
● Small, but growing training set
40 © 2013 IBM Corporation

41. Demo
● Bit.ly/go4bluemix
41 © 2013 IBM Corporation

42. References
[1] Jeffrey Kabot, “Deep Parsing”
[2] Richard Nordquist, “slot and
filler”
[3] The Journal of Research and
Development, Vol 56, 2012
42 © 2013 IBM Corporation