Lecture 6: Watson and the Social Web (2014), Chris Welty

2. © 2011 IBM Corporation What is Cognitive Computing? §  Increasingly, machines are being asked to add their computational power to problems which are not inherently solvable §  Traditionally, these problems came from AI – The hardest AI problems are the easiest for human intelligence: vision, speech, natural language – these are not actually associated with “being intelligent” – Human intelligence provides solutions, but does not scale §  Cognitive Computing is founded on four principles Learn & improve. Cognitive computing systems focus on inexact solutions to unsolvable problems that utilize machine learning and improve over time. Often they combine multiple approaches and must integrate them effectively. They must learn from humans, in more and more seamless ways. Speed&Scale. Cognitive computing harnesses the clear advantage machines have over humans in their ability to perform mundane tasks of arbitrary complexity repeatedly, whether it is the scale of the data or the complexity of the task. Interact in a natural way. Cognitive computing provides technologies that support a higher level of human cognition by adapting to human approaches and interfaces...over the next several decades it will incorporate essentially all the ways humans sense and interact. Assist & augment human cognition. Cognitive computing addresses problems that lie squarely in the province of human intelligence, but where we can't handle the volume of information, penetrate the complexity or otherwise extend our reach (physically). The goal is to be useful, not universally correct. or Computers can be incorrect and still prove useful!

4. © 2011 IBM Corporation What is Watson? §  Open Domain Question-Answering Machine §  Given –  Rich Natural Language Questions –  Over a Broad Domain of Knowledge §  Delivers –  Precise Answers: Determine what is being asked & give precise response –  Accurate Confidences: Determine likelihood answer is correct –  Consumable Justifications: Explain why the answer is right –  Fast Response Time: Precision & Confidence in <3 seconds –  At the level of human experts – Proved its mettle in a televised match –  Won a 2-game Jeopardy match against the all-time winners –  viewed by over 50,000,000 4

5. © 2011 IBM Corporation What is Jeopardy? § Jeopardy! is an American quiz show – 1964 – Today – Household name in U.S. § answer-and-question format – contestants are presented with clues in the form of answers – must phrase their responses in question form. – Open domain trivia questions, speed is a big factor §  Example –  Category: General Science –  Clue: When hit by electrons, a phosphor gives off electromagnetic energy in this form –  Answer: What is light?

6. © 2011 IBM Corporation Social Computing: What’s the connection? § Social Web as Data Source: – The vast majority of sources Watson used to answer questions came from community-created data –  Adapting Watson to a new problem requires the same kind of information about that problem §  Social Machines: –  Watson combined with people is a powerful proposition §  Social Web as Application: –  Watson’s major advance is in understanding natural language, the technology can be useful to augment social interaction

7. © 2011 IBM Corporation $200 If you are looking at the wainscoating, you are looking in this direction. $1000 The first person mentioned by name in ‘The Man in the Iron Mask’ is this hero of a previous book by the same author. 7 The Jeopardy! Challenge Hard for humans, hard for machines Broad/Open Domain Complex Language High Precision Accurate Confidence High Speed $600 In cell division, mitosis splits the nucleus & cytokinesis splits this liquid cushioning the nucleus $800 The conspirators against this man were wounded by each other while they stabbed at him But hard for different reasons. For people, the challenge is knowing the answer For machines, the challenge is understanding the question What is down? Who is D’Artagnan? What is cytoplasm? Who is Julius Caesar?

8. © 2011 IBM Corporation The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players Winning Human Performance 2007 QA Computer System Grand Champion Human Performance Top human players are remarkably good. Each dot – actual historical human Jeopardy! games More Confident Less Confident Develop against a metric!

9. © 2011 IBM Corporation 2007 QA Computer System In 2007, we committed to making a Huge Leap! More Confident Less Confident Each dot – actual historical human Jeopardy! games Computers? Not So Good. Winning Human Performance Grand Champion Human Performance The Winner’s Cloud What It Takes to compete against Top Human Jeopardy! Players

10. © 2011 IBM Corporation DeepQA: The Technology Behind Watson An example of a new software paradigm . . . Answer Scoring Models Answer & Confidence Question Evidence Sources Models Models Models Models ModelsPrimary Search Candidate Answer Generation Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Synthesis Answer Sources Question & Topic Analysis Question Decomposition Evidence Retrieval Deep Evidence Scoring Hypothesis Generation Hypothesis and Evidence Scoring Learned Models help combine and weigh the Evidence DeepQA generates and scores many hypotheses using an extensible collection of Natural Language Processing, Machine Learning and Reasoning Algorithms. These gather and weigh evidence over both unstructured and structured content to determine the answer with the best confidence. Content from Community Resources!

11. © 2011 IBM Corporation Example Question In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city Related Content (Structured & Unstructured) Primary Search 1985 Post Foods aramour General Foods Grand Rapids … Battle Creek … … Candidate Answer Generation 1)  Battle Creek (0.85) 2)  Post Foods ( 0.20) 3)  1985 (0.05) Merging & Ranking Evidence Retrieval Question Analysis Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) … [0.58 0 -1.3 … 0.97] [0.71 1 13.4 … 0.72] [0.12 0 2.0 … 0.40] [0.84 1 10.6 … 0.21] [0.33 0 6.3 … 0.83] [0.21 1 11.1 … 0.92] [0.91 0 -8.2 … 0.61] [0.91 0 -1.7 … 0.60] Evidence Scoring Need thousands of Q/A pairs for training!

12. © 2011 IBM Corporation Planet Fitness Role of Answer Typing in QA Type Information - a crucial hint to get the correct answer ASTRONOMY: In 1610 Galileo named the moons of this planet for the Medici brothers Telescope Giovanni Medici Sidereus Nuncius Jupiter Ganymede Telescope (Instrument) Giovanni Medici (Person) Sidereus Nuncius (Book) Jupiter (Planet) Ganymede (Moon) Terms Associated with Clue Context (e.g. via Keyword Search) Planet Fitness (Planet)

13. © 2011 IBM Corporation §  This fish was thought to be exLnct millions of years ago unLl one was found off South Africa in 1938 §  Category: ENDS IN "TH" §  Answer: §  When hit by electrons, a phosphor gives off electromagneLc energy in this form §  Category: General Science §  Answer: §  Secy. Chase just submiXed this to me for the third Lme-‐-‐guess what, pal. This Lme I'm accepLng it §  Category: Lincoln Blogs §  Answer: The type of thing being asked for is often indicated but can go from specific to very vague coelacanth light (or photons) his resigna4on 13 Answer Typing for Jeopardy!?

14. © 2011 IBM Corporation Broad Domain Our Focus is on reusable NLP technology for analyzing vast volumes of as-is text. Structured sources (DBs and KBs) provide background knowledge for interpreting the text. We do NOT attempt to anticipate all questions and build databases. We do NOT try to build a formal model of the world

15. © 2011 IBM Corporation Sources for typing evidence § DbPedia & Freebase – Wide coverage of well-known entities – Taxonomy (MountainsOfNepal → Mountain) – Good type coverage, but not many synonyms •  E.g. what about “summit” §  Wikpedia Categories –  Wide coverage of entities and type name synonyms –  Noisy (many errors) §  Wikipedia Intro –  First sentence always indicates the most common type of the entity –  Highly reliable, low coverage of types Communities can scale data collection!

17. © 2011 IBM Corporation Many sources of evidence In 1894 C.W. Post created his warm cereal drink Postum in this Michigan city Related Content (Structured & Unstructured) Primary Search 1985 Post Foods aramour General Foods Grand Rapids … Battle Creek … … Candidate Answer Generation 1)  Battle Creek (0.85) 2)  Post Foods ( 0.20) 3)  1985 (0.05) Merging & Ranking Evidence Retrieval Question Analysis Keywords: 1894, C.W. Post, created … Lexical AnswerType: (Michingan city) Date(1894) Relations: Create(Post, cereal drink) … [0.58 0 -1.3 … 0.97] [0.71 1 13.4 … 0.72] [0.12 0 2.0 … 0.40] [0.84 1 10.6 … 0.21] [0.33 0 6.3 … 0.83] [0.21 1 11.1 … 0.92] [0.91 0 -8.2 … 0.61] [0.91 0 -1.7 … 0.60] Evidence Scoring

18. © 2011 IBM Corporation Watson as part of a social machine §  Watson makes mistakes: – This woman was the first to witness her husband resign from the U.S. Presidency. – This U.S. City’s largest airport is named for a world-war II hero; its second largest for a world-war II battle. §  These mistakes are typically obvious to people – Even when they don’t know the answer – Watson isn’t stupid, it solves problems differently – Often these multiple perspectives can combine productively •  E.g. add a “dismiss” button to the answer interface Richard Nixon Dolly Madison Pat Nixon Watson can adapt and learn from its users!

21. © 2011 IBM Corporation Adapt Watson Models Answer & Confidence Question Evidence Sources Models Models Models Models Models Answer Sources . . . Answer Scoring Primary Search Candidate Answer Generation Hypothesis Generation Hypothesis and Evidence Scoring Final Confidence Merging & Ranking Synthesis Question & Topic Analysis Question Decomposition Evidence Retrieval Deep Evidence Scoring Hypothesis Generation Hypothesis and Evidence Scoring Learned Models help combine and weigh the Evidence What does it take to use Watson in a new domain? (medical diagnosis, call centers, etc...) Gathering significant numbers of question-answer pairs is proving to be one of the most significant challenges for adapting Watson. Can the social web help? Community created!

22. © 2011 IBM Corporation Integrating Watson in Social Interaction? Did you hear about Bob? No He’s taking a year off to climb the tallest mountain! The tallest mountain is Mount Everest. Wow. me me Jeff Watson Jeff

23. © 2011 IBM Corporation Privacy – a blessing and a curse Need to protect our data, but… Crime on the web, the social web, is very real Identity theft Credit card, bank, insurance fraud Terrorist networks Medical diagnosis Monitoring your profile for health-related information ICT for depression Calendar, appointments, traffic, spreading disease

24. © 2011 IBM Corporation The arrival of Cognitive Computing Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs). Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range. Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse… Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.

25. © 2011 IBM Corporation The arrival of Cognitive Computing Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs). 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% % Answered

26. © 2011 IBM Corporation The arrival of Cognitive Computing Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities. UTI Diabetes Influenza hypokalemia Renal failure esophogitis Diagnosis Models Conﬁdence Most Conﬁdent Diagnosis: UTI Symptoms Tests/Findings Medica4ons Family History Notes/Hypotheses Huge Volumes of Texts, Journals, References, DBs etc. Pa4ent History

28. © 2011 IBM Corporation The arrival of Cognitive Computing Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse…

29. © 2011 IBM Corporation The arrival of Cognitive Computing Learn & improve. The core of Watson is a group of over 100 independent algorithms that approximate a solution to the “is this the right answer to the question” problem. Achieving winning (human expert) performance, required two hallmarks of cognitive computing systems: a metric to measure improvements to the system (the winners cloud), and a significant ground truth (over 200K Q-A pairs). Speed&Scale. Watson used big data, as well as a 3000 node cluster for massive computation to get answering speeds down into the 2s range. Interact in a natural way. Watson was a significant step forward in natural language understanding, the most basic interface for humans. Say goodbye to your mouse… Assist & augment human cognition. Watson depended on primarily a set of background documents (the corpus). The value of having access to this kind of fact-finding power over a large (and possibly changing) corpus provides a clear augmentation to human abilities.

30. © 2011 IBM Corporation …and for Social Web §  First and foremost, social web analytics (e.g. recommendations) and Social Computing in general lie clearly in the realm of Cognitive Computing – Uncertainty, natural language, human intelligence – Inexact solutions that can improve with time, training – Problems & solutions need metrics to be solvable §  All cognitive computing systems require ground truth data – This data is expensive to collect – Crowdsourcing is a key new technology/approach §  The user interface moving closer to people – Natural language, speech, gestures – In addition, integrating the collection of training data seamlessly into the interface is a key development §  Cognitive computing systems require integration of multiple, disparate, data sources – Structured, unstructured, semi-structured – curated, crowdsourced

Lecture 6: Watson and the Social Web (2014), Chris Welty

Lecture 6: Watson and the Social Web (2014), Chris Welty

Recommended

More Related Content

Similar to Lecture 6: Watson and the Social Web (2014), Chris Welty

Similar to Lecture 6: Watson and the Social Web (2014), Chris Welty(20)

More from Lora Aroyo

More from Lora Aroyo(20)

Recently uploaded

Recently uploaded(20)

Lecture 6: Watson and the Social Web (2014), Chris Welty