© 2011 IBM Corporation
Watson and the Social Web
Chris Welty
IBM Watson Group
ibmwatson.com
Do Not Record. Do Not Distribu...
© 2011 IBM Corporation
What is Cognitive Computing?
§  Increasingly, machines are being asked to add their computational
...
© 2011 IBM Corporation
Examples of Cognitive Computing
§ Web Search
§ Image Search
§ Event Search
§ Recommendations
§...
© 2011 IBM Corporation
What is Watson?
§  Open Domain Question-Answering Machine
§  Given
–  Rich Natural Language Quest...
© 2011 IBM Corporation
What is Jeopardy?
§ Jeopardy! is an American quiz
show
– 1964 – Today
– Household name in U.S.
§ ...
© 2011 IBM Corporation
Social Computing: What’s the connection?
§ Social Web as Data Source:
– The vast majority of sourc...
© 2011 IBM Corporation
$200
If you are looking at
the wainscoating,
you are looking in
this direction.
$1000
The first per...
© 2011 IBM Corporation
The Winner’s Cloud
What It Takes to compete against Top Human Jeopardy! Players
Winning Human
Perfo...
© 2011 IBM Corporation
2007 QA Computer System
In 2007, we committed to
making a Huge Leap!
More Confident Less Confident
...
© 2011 IBM Corporation
DeepQA: The Technology Behind Watson
An example of a new software paradigm
. . .
Answer
Scoring
Mod...
© 2011 IBM Corporation
Example Question
In 1894 C.W. Post
created his warm
cereal drink Postum in
this Michigan city
Relat...
© 2011 IBM Corporation
Planet Fitness
Role	
  of	
  Answer	
  Typing	
  in	
  QA	
  
Type Information - a crucial hint to ...
© 2011 IBM Corporation
§  This	
  fish	
  was	
  thought	
  to	
  be	
  exLnct	
  millions	
  of	
  years	
  ago	
  	
  
u...
© 2011 IBM Corporation
Broad Domain
Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text.
Stru...
© 2011 IBM Corporation
Sources for typing evidence
§ DbPedia & Freebase
– Wide coverage of well-known entities
– Taxonomy...
© 2011 IBM Corporation
Typing	
  Impact	
  on	
  Jeopardy!	
  clues	
  
61.5%
62.0%
62.5%
63.0%
63.5%
64.0%
64.5%
65.0%
65...
© 2011 IBM Corporation
Many sources of evidence
In 1894 C.W. Post
created his warm
cereal drink Postum in
this Michigan ci...
© 2011 IBM Corporation
Watson as part of a social machine
§  Watson makes mistakes:
– This woman was the first to witness...
© 2011 IBM Corporation
Cut to the chase…..
Watson emerges victorious
© 2011 IBM Corporation
Technology marches forward…
© 2011 IBM Corporation
Adapt Watson
Models
Answer &
Confidence
Question
Evidence
Sources
Models
Models
Models
Models
Model...
© 2011 IBM Corporation
Integrating Watson in Social Interaction?
Did you hear about Bob?
No
He’s taking a year off to clim...
© 2011 IBM Corporation
Privacy – a blessing and a curse
Need to protect our data, but…
Crime on the web, the social web, i...
© 2011 IBM Corporation
The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of
over 100 indep...
© 2011 IBM Corporation
The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of
over 100 indep...
© 2011 IBM Corporation
The arrival of Cognitive Computing
Assist & augment human cognition. Watson
depended on primarily a...
© 2011 IBM Corporation
The arrival of Cognitive Computing
Speed&Scale. Watson used big data, as well as a
3000 node cluste...
© 2011 IBM Corporation
The arrival of Cognitive Computing
Interact in a natural way. Watson was a significant
step forward...
© 2011 IBM Corporation
The arrival of Cognitive Computing
Learn & improve. The core of Watson is a group of
over 100 indep...
© 2011 IBM Corporation
…and for Social Web
§  First and foremost, social web analytics (e.g. recommendations) and Social
...
Upcoming SlideShare
Loading in …5
×

Lecture 6: Watson and the Social Web (2014), Chris Welty

1,933 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Lecture 6: Watson and the Social Web (2014), Chris Welty

  1. 1. © 2011 IBM Corporation Watson and the Social Web Chris Welty IBM Watson Group ibmwatson.com Do Not Record. Do Not Distribute.
  2. 2. © 2011 IBM Corporation What is Cognitive Computing? §  Increasingly, machines are being asked to add their computational power to problems which are not inherently solvable §  Traditionally, these problems came from AI – The hardest AI problems are the easiest for human intelligence: vision, speech, natural language – these are not actually associated with “being intelligent” – Human intelligence provides solutions, but does not scale §  Cognitive Computing is founded on four principles Learn & improve. Cognitive computing systems focus on inexact solutions to unsolvable problems that utilize machine learning and improve over time. Often they combine multiple approaches and must integrate them effectively. They must learn from humans, in more and more seamless ways. Speed&Scale. Cognitive computing harnesses the clear advantage machines have over humans in their ability to perform mundane tasks of arbitrary complexity repeatedly, whether it is the scale of the data or the complexity of the task. Interact in a natural way. Cognitive computing provides technologies that support a higher level of human cognition by adapting to human approaches and interfaces...over the next several decades it will incorporate essentially all the ways humans sense and interact. Assist & augment human cognition. Cognitive computing addresses problems that lie squarely in the province of human intelligence, but where we can't handle the volume of information, penetrate the complexity or otherwise extend our reach (physically). The goal is to be useful, not universally correct. or Computers can be incorrect and still prove useful!
  3. 3. © 2011 IBM Corporation Examples of Cognitive Computing § Web Search § Image Search § Event Search § Recommendations § Natural Language Processing
  4. 4. © 2011 IBM Corporation What is Watson? §  Open Domain Question-Answering Machine §  Given –  Rich Natural Language Questions –  Over a Broad Domain of Knowledge §  Delivers –  Precise Answers: Determine what is being asked & give precise response –  Accurate Confidences: Determine likelihood answer is correct –  Consumable Justifications: Explain why the answer is right –  Fast Response Time: Precision & Confidence in <3 seconds –  At the level of human experts – Proved its mettle in a televised match –  Won a 2-game Jeopardy match against the all-time winners –  viewed by over 50,000,000 4
  5. 5. © 2011 IBM Corporation What is Jeopardy? § Jeopardy! is an American quiz show – 1964 – Today – Household name in U.S. § answer-and-question format – contestants are presented with clues in the form of answers – must phrase their responses in question form. – Open domain trivia questions, speed is a big factor §  Example –  Category: General Science –  Clue: When hit by electrons, a phosphor gives off electromagnetic energy in this form –  Answer: What is light?
  6. 6. © 2011 IBM Corporation Social Computing: What’s the connection? § Social Web as Data Source: – The vast majority of sources Watson used to answer questions came from community-created data –  Adapting Watson to a new problem requires the same kind of information about that problem §  Social Machines: –  Watson combined with people is a powerful proposition §  Social Web as Application: –  Watson’s major advance is in understanding natural language, the technology can be useful to augment social interaction
  7. 7. © 2011 IBM Corporation $200 If you are looking at the wainscoating, you are looking in this direction. $1000 The first person mentioned by name in ‘The Man in the Iron Mask’ is this hero of a previous book by the same author. 7 The Jeopardy! Challenge Hard for humans, hard for machines Broad/Open Domain Complex Language High Precision Accurate Confidence High Speed $600 In cell division, mitosis splits the nucleus & cytokinesis splits this liquid cushioning the nucleus $800 The conspirators against this man were wounded by each other while they stabbed at him But hard for different reasons. For people, the challenge is knowing the answer For machines, the challenge is understanding the question What is down? Who is D’Artagnan? What is cytoplasm? Who is Julius Caesar?
  8. 8. © 2011 IBM Corporation The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players Winning Human Performance 2007 QA Computer System Grand Champion Human Performance Top human players are remarkably good. Each dot – actual historical human Jeopardy! games More Confident Less Confident Develop against a metric!
  9. 9. © 2011 IBM Corporation 2007 QA Computer System In 2007, we committed to making a Huge Leap! More Confident Less Confident Each dot – actual historical human Jeopardy! games Computers? Not So Good. Winning Human Performance Grand Champion Human Performance The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players
  10. 10. © 2011 IBM Corporation DeepQA: The Technology Behind Watson An example of a new software paradigm . . . Answer Scoring Models Answer & Confidence Question Evidence Sources Models Models Models Models ModelsPrimary Search Candidate Answer Generation Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Synthesis Answer Sources Question & Topic Analysis Question Decomposition Evidence Retrieval Deep Evidence Scoring Hypothesis Generation Hypothesis and Evidence Scoring Learned Models help combine and weigh the Evidence DeepQA generates and scores many hypotheses using an extensible collection of Natural Language Processing, Machine Learning and Reasoning Algorithms. These gather and weigh evidence over both unstructured and structured content to determine the answer with the best confidence. Content from Community Resources!
  11. 11. © 2011 IBM Corporation Example Question In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city Related Content (Structured & Unstructured) Primary Search 1985 Post Foods aramour General Foods Grand Rapids … Battle Creek … … Candidate Answer Generation 1)  Battle Creek (0.85) 2)  Post Foods ( 0.20) 3)  1985 (0.05) Merging & Ranking Evidence Retrieval Question Analysis Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) … [0.58 0 -1.3 … 0.97] [0.71 1 13.4 … 0.72] [0.12 0 2.0 … 0.40] [0.84 1 10.6 … 0.21] [0.33 0 6.3 … 0.83] [0.21 1 11.1 … 0.92] [0.91 0 -8.2 … 0.61] [0.91 0 -1.7 … 0.60] Evidence Scoring Need thousands of Q/A pairs for training!
  12. 12. © 2011 IBM Corporation Planet Fitness Role  of  Answer  Typing  in  QA   Type Information - a crucial hint to get the correct answer ASTRONOMY:  In  1610  Galileo  named  the  moons  of  this  planet   for  the  Medici  brothers   Telescope Giovanni Medici Sidereus Nuncius Jupiter Ganymede Telescope (Instrument) Giovanni Medici (Person) Sidereus Nuncius (Book) Jupiter (Planet) Ganymede (Moon) Terms  Associated  with  Clue  Context    (e.g.  via  Keyword  Search)   Planet Fitness (Planet)
  13. 13. © 2011 IBM Corporation §  This  fish  was  thought  to  be  exLnct  millions  of  years  ago     unLl  one  was  found  off  South  Africa  in  1938       §  Category:  ENDS  IN  "TH"     §  Answer:   §  When  hit  by  electrons,  a  phosphor  gives  off  electromagneLc   energy  in  this  form   §  Category:  General  Science   §  Answer:     §  Secy.  Chase  just  submiXed  this  to  me  for  the  third  Lme-­‐-­‐guess   what,  pal.  This  Lme  I'm  accepLng  it     §  Category:  Lincoln  Blogs   §  Answer:     The type of thing being asked for is often indicated but can go from specific to very vague coelacanth   light  (or  photons)   his  resigna4on   13   Answer Typing for Jeopardy!?
  14. 14. © 2011 IBM Corporation Broad Domain Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text. We do NOT attempt to anticipate all questions and build databases. We do NOT try to build a formal model of the world
  15. 15. © 2011 IBM Corporation Sources for typing evidence § DbPedia & Freebase – Wide coverage of well-known entities – Taxonomy (MountainsOfNepal → Mountain) – Good type coverage, but not many synonyms •  E.g. what about “summit” §  Wikpedia Categories –  Wide coverage of entities and type name synonyms –  Noisy (many errors) §  Wikipedia Intro –  First sentence always indicates the most common type of the entity –  Highly reliable, low coverage of types Communities can scale data collection!
  16. 16. © 2011 IBM Corporation Typing  Impact  on  Jeopardy!  clues   61.5% 62.0% 62.5% 63.0% 63.5% 64.0% 64.5% 65.0% 65.5% 66.0% 66.5% An ensemble of TyCor components + ~10%
  17. 17. © 2011 IBM Corporation Many sources of evidence In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city Related Content (Structured & Unstructured) Primary Search 1985 Post Foods aramour General Foods Grand Rapids … Battle Creek … … Candidate Answer Generation 1)  Battle Creek (0.85) 2)  Post Foods ( 0.20) 3)  1985 (0.05) Merging & Ranking Evidence Retrieval Question Analysis Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) … [0.58 0 -1.3 … 0.97] [0.71 1 13.4 … 0.72] [0.12 0 2.0 … 0.40] [0.84 1 10.6 … 0.21] [0.33 0 6.3 … 0.83] [0.21 1 11.1 … 0.92] [0.91 0 -8.2 … 0.61] [0.91 0 -1.7 … 0.60] Evidence Scoring
  18. 18. © 2011 IBM Corporation Watson as part of a social machine §  Watson makes mistakes: – This woman was the first to witness her husband resign from the U.S. Presidency. – This U.S. City’s largest airport is named for a world-war II hero; its second largest for a world-war II battle. §  These mistakes are typically obvious to people – Even when they don’t know the answer – Watson isn’t stupid, it solves problems differently – Often these multiple perspectives can combine productively •  E.g. add a “dismiss” button to the answer interface Richard Nixon Dolly Madison Pat Nixon Watson can adapt and learn from its users!
  19. 19. © 2011 IBM Corporation Cut to the chase….. Watson emerges victorious
  20. 20. © 2011 IBM Corporation Technology marches forward…
  21. 21. © 2011 IBM Corporation Adapt Watson Models Answer & Confidence Question Evidence Sources Models Models Models Models Models Answer Sources . . . Answer Scoring Primary Search Candidate Answer Generation Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Synthesis Question & Topic Analysis Question Decomposition Evidence Retrieval Deep Evidence Scoring Hypothesis Generation Hypothesis and Evidence Scoring Learned Models help combine and weigh the Evidence What does it take to use Watson in a new domain? (medical diagnosis, call centers, etc...) Gathering significant numbers of question-answer pairs is proving to be one of the most significant challenges for adapting Watson. Can the social web help? Community created!
  22. 22. © 2011 IBM Corporation Integrating Watson in Social Interaction? Did you hear about Bob? No He’s taking a year off to climb the tallest mountain! The tallest mountain is Mount Everest. Wow. me me Jeff Watson Jeff
  23. 23. © 2011 IBM Corporation Privacy – a blessing and a curse Need to protect our data, but… Crime on the web, the social web, is very real Identity theft Credit card, bank, insurance fraud Terrorist networks Medical diagnosis Monitoring your profile for health-related information ICT for depression Calendar, appointments, traffic, spreading disease
  24. 24. © 2011 IBM Corporation The arrival of Cognitive Computing Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs). Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range. Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse… Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.
  25. 25. © 2011 IBM Corporation The arrival of Cognitive Computing Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs). 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % Answered
  26. 26. © 2011 IBM Corporation The arrival of Cognitive Computing Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities. UTI Diabetes Influenza hypokalemia Renal failure esophogitis Diagnosis  Models   Confidence   Most  Confident  Diagnosis:  UTI     Symptoms   Tests/Findings   Medica4ons   Family  History   Notes/Hypotheses   Huge  Volumes  of  Texts,   Journals,  References,  DBs   etc.   Pa4ent  History  
  27. 27. © 2011 IBM Corporation The arrival of Cognitive Computing Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range.
  28. 28. © 2011 IBM Corporation The arrival of Cognitive Computing Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse…
  29. 29. © 2011 IBM Corporation The arrival of Cognitive Computing Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs). Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range. Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse… Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.
  30. 30. © 2011 IBM Corporation …and for Social Web §  First and foremost, social web analytics (e.g. recommendations) and Social Computing in general lie clearly in the realm of Cognitive Computing – Uncertainty, natural language, human intelligence – Inexact solutions that can improve with time, training – Problems & solutions need metrics to be solvable §  All cognitive computing systems require ground truth data – This data is expensive to collect – Crowdsourcing is a key new technology/approach §  The user interface moving closer to people – Natural language, speech, gestures – In addition, integrating the collection of training data seamlessly into the interface is a key development §  Cognitive computing systems require integration of multiple, disparate, data sources – Structured, unstructured, semi-structured – curated, crowdsourced

×