“Towards Building a Cognitive System to Fight for National College Admission Challenge”

60 views

Published on

Yansong Feng from Peking University presented “Towards Building a Cognitive System to Fight for National College Admission Challenge” as part of the Cognitive Systems Institute Speaker Series.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
60
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

“Towards Building a Cognitive System to Fight for National College Admission Challenge”

  1. 1. Towards Building a Cognitive System to Fight for National College Admission Challenge Yansong Feng Joint work with Kun Xu, Songfang Huang, Dongyan Zhao Peking University IBM China Research Lab December 1, 2016 Feng et al. (PKU) Question Answering December 1, 2016 1 / 25
  2. 2. Pass the Exam: A New AI Challenge The Todai Robot Project National Institute of Informatics and collaborators National Center Test for University Admissions ( 2016) Entrance Exam of University of Tokyo (2021) Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
  3. 3. Pass the Exam: A New AI Challenge The Todai Robot Project National Institute of Informatics and collaborators National Center Test for University Admissions ( 2016) Entrance Exam of University of Tokyo (2021) Japanese, Social Science, Math, Physics Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
  4. 4. Pass the Exam: A New AI Challenge The Todai Robot Project National Institute of Informatics and collaborators National Center Test for University Admissions ( 2016) Entrance Exam of University of Tokyo (2021) Japanese, Social Science, Math, Physics The Project Aristo and Euclid The Allen Institute for Artificial Intelligence Elementary School: High School: Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
  5. 5. Pass the Exam: A New AI Challenge The Todai Robot Project National Institute of Informatics and collaborators National Center Test for University Admissions ( 2016) Entrance Exam of University of Tokyo (2021) Japanese, Social Science, Math, Physics The Project Aristo and Euclid The Allen Institute for Artificial Intelligence Elementary School: Science and Math High School: Geometry Feng et al. (PKU) Question Answering December 1, 2016 2 / 25
  6. 6. The GaoKao Challenge Gaokao in China National College Entrance Examination Chinese, Math, English, History, Geography, Politics, Physics, Chemistry, Biology over 9,400,000 students in 2016 Feng et al. (PKU) Question Answering December 1, 2016 3 / 25
  7. 7. The GaoKao Challenge Gaokao in China National College Entrance Examination Chinese, Math, English, History, Geography, Politics, Physics, Chemistry, Biology over 9,400,000 students in 2016 The China Gaokao Challenge Prompt research in Artificial Intelligence Team: national research institutes, universities and companies Real National College Entrance Examinations Chinese, History, Math, Geography Feng et al. (PKU) Question Answering December 1, 2016 3 / 25
  8. 8. The GaoKao Challenge Gaokao in China National College Entrance Examination Chinese, Math, English, History, Geography, Politics, Physics, Chemistry, Biology over 9,400,000 students in 2016 The China Gaokao Challenge Prompt research in Artificial Intelligence Team: national research institutes, universities and companies Real National College Entrance Examinations Chinese, History, Math, Geography Feng et al. (PKU) Question Answering December 1, 2016 3 / 25
  9. 9. Is that Difficult? Which is the correct ranking of provinces according to their average altitutes, from highest to lowest? Xiang, Liao, Ning Tai, Lu, Su Qing, Yue, Jin Gui, Gan, Yu Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
  10. 10. Is that Difficult? Which is the correct ranking of provinces according to their average altitutes, from highest to lowest? Xiang, Liao, Ning → Hunan, Liaoning, Ningxia Tai, Lu, Su →Taiwan, Shandong, Jiangsu Qing, Yue, Jin →Qinghua, Guangdong, Shanxi Gui, Gan, Yu → Guangxi, Gansu, Henan 1 Knowledge: short names of provinces average altitudes of provinces Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
  11. 11. Is that Difficult? Which is the correct ranking of provinces according to their average altitutes, from highest to lowest? Xiang, Liao, Ning → Hunan, Liaoning, Ningxia Tai, Lu, Su →Taiwan, Shandong, Jiangsu Qing, Yue, Jin →Qinghua, Guangdong, Shanxi Gui, Gan, Yu → Guangxi, Gansu, Henan 1 Knowledge: short names of provinces average altitudes of provinces 2 Reasoning relative comparisons of provinces’ altitude ranking Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
  12. 12. Is that Difficult? Which is the correct ranking of provinces according to their average altitutes, from highest to lowest? Xiang, Liao, Ning → Hunan, Liaoning, Ningxia Tai, Lu, Su →Taiwan, Shandong, Jiangsu Qing, Yue, Jin →Qinghua, Guangdong, Shanxi Gui, Gan, Yu → Guangxi, Gansu, Henan 1 Knowledge: short names of provinces average altitudes of provinces 2 Reasoning relative comparisons of provinces’ altitude ranking Not very challenging? Feng et al. (PKU) Question Answering December 1, 2016 4 / 25
  13. 13. What about this one? Missouri River Valley is an important agricultural area of the United States. Aerial figure 1 shows the winter of the Missouri River, where the white part is snow. 1. Why is the farmland on the peninsula shaped as circular? their watering approach rugged terrain their farming approach lack of farmland Feng et al. (PKU) Question Answering December 1, 2016 5 / 25
  14. 14. What about this one? 2. Could you have a guess what is the main crop in this area? Winter wheat Corn Rice Potato Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
  15. 15. What about this one? 2. Could you have a guess what is the main crop in this area? Winter wheat Corn Rice Potato 1 Knowledge: agriculture climate longitude and latitude read maps and images Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
  16. 16. What about this one? 2. Could you have a guess what is the main crop in this area? Winter wheat Corn Rice Potato 1 Knowledge: agriculture climate longitude and latitude read maps and images 2 Reasoning relationship among those factors find analogies common sense knowledge Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
  17. 17. What about this one? 2. Could you have a guess what is the main crop in this area? Winter wheat Corn Rice Potato 1 Knowledge: agriculture climate longitude and latitude read maps and images 2 Reasoning relationship among those factors find analogies common sense knowledge Not very challenging? Feng et al. (PKU) Question Answering December 1, 2016 6 / 25
  18. 18. What We Need? 1 Solid knowledge: every aspects about the Syllabus 2 Math 3 Reasoning: logical inference use common sense knowledge textual entailment Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
  19. 19. What We Need? 1 Solid knowledge: every aspects about the Syllabus 2 Math 3 Reasoning: logical inference use common sense knowledge textual entailment A practical starting point: Answering Factoid Questions with Knowledge Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
  20. 20. What We Need? 1 Solid knowledge: every aspects about the Syllabus 2 Math 3 Reasoning: logical inference use common sense knowledge textual entailment A practical starting point: Answering Factoid Questions with Knowledge Information Retrieval Based Question Answering Answering Factoid Questions with Structured Knowledge Base Answering Factoid Questions with both Structured and Unstructured KBs Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
  21. 21. What We Need? 1 Solid knowledge: every aspects about the Syllabus (Knowledge Bases) 2 Math 3 Reasoning: logical inference use common sense knowledge (a little bit) textual entailment A practical starting point: Answering Factoid Questions with Knowledge Information Retrieval Based Question Answering Answering Factoid Questions with Structured Knowledge Base Answering Factoid Questions with both Structured and Unstructured KBs Feng et al. (PKU) Question Answering December 1, 2016 7 / 25
  22. 22. The Task What else did the director of the movie Interstellar direct ? fb:m.0fkf28 fb:object.type fb:film.film fb:m.0fkf28 fb:film.film.directed_by ?x [ ] select ?y ?x fb:film.director.fim ?y ?y fb:object.type fb:film.film Inception, The Dark Knight Rises The Dark Knight Batman Begins ….. Feng et al. (PKU) Question Answering December 1, 2016 8 / 25
  23. 23. The Task What else did the director of the movie Interstellar direct ? fb:m.0fkf28 fb:object.type fb:film.film fb:m.0fkf28 fb:film.film.directed_by ?x [ ] select ?y ?x fb:film.director.fim ?y ?y fb:object.type fb:film.film Inception, The Dark Knight Rises The Dark Knight Batman Begins ….. Feng et al. (PKU) Question Answering December 1, 2016 8 / 25
  24. 24. Question Answering over Structured Knowledge Bases Goal Answer Natural Language Questions against Structured Knowledge Bases Feng et al. (PKU) Question Answering December 1, 2016 9 / 25
  25. 25. Related Work Information Retrieval Community Natural Language Processing Community Semantic Parsing Based PCCG style: (Zettlemoyer and Collins, 2005, Cai and Yates, 2013; Kwiatkowski et al. 2013, Reddy et al., 2014) Syntactic parsing style: (Clarke et al., 2010, Liang et al., 2011, Berant et al. 2013, Berant and Liang, 2014, Xu et al., 2014) Information Extraction Based (Yao and van Durme 2014, Bao et al., 2014, Yih et al., 2015, Dong et al., 2015) Deep Learning, End2End style (Bordes et al., 2014a, 2014b, Yang et al., 2014, Bordes et al., 2015, Zhang et al., 2016 ) Feng et al. (PKU) Question Answering December 1, 2016 10 / 25
  26. 26. Related Work Information Retrieval Community Natural Language Processing Community Semantic Parsing Based PCCG style: (Zettlemoyer and Collins, 2005, Cai and Yates, 2013; Kwiatkowski et al. 2013, Reddy et al., 2014) Syntactic parsing style: (Clarke et al., 2010, Liang et al., 2011, Berant et al. 2013, Berant and Liang, 2014, Xu et al., 2014) Information Extraction Based (Yao and van Durme 2014, Bao et al., 2014, Yih et al., 2015, Dong et al., 2015) Deep Learning, End2End style (Bordes et al., 2014a, 2014b, Yang et al., 2014, Bordes et al., 2015, Zhang et al., 2016 ) Feng et al. (PKU) Question Answering December 1, 2016 10 / 25
  27. 27. Semantic Parsing Based Methods Challenges: 1 Convert questions into proper meaning representations 2 Ground the meaning representation into a database query Feng et al. (PKU) Question Answering December 1, 2016 11 / 25
  28. 28. Semantic Parsing Based Methods Challenges: 1 Convert questions into proper meaning representations 2 Ground the meaning representation into a database query Previously: 1 Search space is huge 2 Difficult to adapt to other KBs Feng et al. (PKU) Question Answering December 1, 2016 11 / 25
  29. 29. Motivation 1 Meaning representation should be KB-independent [what] did the director of] [the movie] [Interstellar]else movies] [direct] Feng et al. (PKU) Question Answering December 1, 2016 12 / 25
  30. 30. Motivation 1 Meaning representation should be KB-independent [what] did the director of] [the movie] [Interstellar]else movies] [direct] 2 Separation of meaning representation and instantiation fb:m.0fkf28 fb:object.type fb:film.film fb:m.0fkf28 fb:film.film.directed_by ?x [ ] select ?y ?x fb:film.director.fim ?y ?y fb:object.type fb:film.film ns:Interstellar dbo:type dbo:film dbp:director ?x [ ] select ?y ?y ?y ns:Interstellar dbp:director ?x dbo:filmdbo:type Feng et al. (PKU) Question Answering December 1, 2016 12 / 25
  31. 31. Framework what else movies did the director of the movie Interstellar direct Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
  32. 32. Framework what else movies did the director of the movie Interstellar direct Parsing Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
  33. 33. Framework what else movies did the director of the movie Interstellar direct Parsing [what] did the director of] [the movie] [Interstellar] variable relation category entity else movies] [direct] category relation Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
  34. 34. Framework what else movies did the director of the movie Interstellar direct Parsing [what] did the director of] [the movie] [Interstellar] variable relation category entity else movies] [direct] category relation Instantiation Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
  35. 35. Framework what else movies did the director of the movie Interstellar direct Parsing [what] did the director of] [the movie] [Interstellar] variable relation category entity else movies] [direct] category relation Instantiation fb:m.0fkf28 fb:object.type fb:film.film fb:m.0fkf28 fb:film.film.directed_by ?x [ ] select ?y ?x fb:film.director.fim ?y ?y fb:object.type fb:film.film Feng et al. (PKU) Question Answering December 1, 2016 13 / 25
  36. 36. Phrase Dependency Graph [what] did the director of] [the movie] [Interstellar] variable relation category entity else movies] [direct] category relation Node A phrase with a semantic label l ∈ {entity, category, variable, relation} Edge A predicate-argument dependency between phrases unary predicate binary predicate Feng et al. (PKU) Question Answering December 1, 2016 14 / 25
  37. 37. Structure Prediction Input: a natural language question Output: a phrase dependency graph A pipeline framework to predict the structure 1 Phrase Detection what did the director of the movie Interstellar variable relation category entity else movies direct category relation 2 Phrase Dependency Parsing [what] did the director of] [the movie] [Interstellar] variable relation category entity else movies] [direct] category relation Feng et al. (PKU) Question Answering December 1, 2016 15 / 25
  38. 38. Instantiation [what] did the director of] [the movie] [Interstellar] variable relation category entity else movies] [direct] category relation 1 Converting Phrase Dependency Graph into Structured Queries 2 Instantiating Structured Query against KB Feng et al. (PKU) Question Answering December 1, 2016 16 / 25
  39. 39. Applying Rules [what] did the director of] [the movie] [Interstellar] variable relation category entity else movies] [direct] category relation variable category variablecategory rule#1 rule#1 variablerelationrelationentity rule#8 ?y type movies type ?x ?x ?y moviesInterstellar Interstellar direct the director of [ ] select ?y type moviesInterstellar ?xInterstellar the director of ?x ?ydirect ?y type movies Feng et al. (PKU) Question Answering December 1, 2016 17 / 25
  40. 40. Probabilistic Model [ ] select ?y type moviesInterstellar ?xInterstellar the director of ?x ?ydirect ?y type movies Qind Qd ns:Interstellar dbo:type dbo:film dbp:director ?x [ ] select ?y ?y ?y ns:Interstellar dbp:director ?x dbo:filmdbo:type Q∗ d = arg max P(Qd |Qind ) P(Qd |Qind ) = n i=1 P(sdi |sindi )P(odi |oindi )P(pdi |pindi ) Feng et al. (PKU) Question Answering December 1, 2016 18 / 25
  41. 41. P(sdi |sindi )P(odi |oindi ) Freebase Search API ⇒ wikipedia ID ⇒ DBpedia Entity P(pd |pind ) We construct the co-occurrence matrix from the patty relation phrase dataset which includes 1,631,530 relation phrases Feng et al. (PKU) Question Answering December 1, 2016 19 / 25
  42. 42. Results on QALDs Question Answering over Linked Data Processed Right Partial Recall Precision F-1 2014 40 34 6 0.71 0.72 0.72 2015 42 26 7 0.72 0.74 0.73 Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
  43. 43. Results on QALDs Question Answering over Linked Data Processed Right Partial Recall Precision F-1 2014 40 34 6 0.71 0.72 0.72 2015 42 26 7 0.72 0.74 0.73 First Place in CLEF QALD 4 and 5 Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
  44. 44. Results on QALDs Question Answering over Linked Data Processed Right Partial Recall Precision F-1 2014 40 34 6 0.71 0.72 0.72 2015 42 26 7 0.72 0.74 0.73 Nice for longer/complex sentences Efficient: around 0.33 sec per sentence Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
  45. 45. Results on QALDs Question Answering over Linked Data Processed Right Partial Recall Precision F-1 2014 40 34 6 0.71 0.72 0.72 2015 42 26 7 0.72 0.74 0.73 Nice for longer/complex sentences Efficient: around 0.33 sec per sentence Consistent performances 0.76 of F-1 on Free917 0.41 of F-1 on WebQuestions Feng et al. (PKU) Question Answering December 1, 2016 20 / 25
  46. 46. Further Improvement Semantic interpretation for superlatives Nile is the longest river in the world. Keys: the target: Nile the comparison set: all rivers in the world the comparison dimension: the length of a river →/geography/river/length the ranking order: descending Feng et al. (PKU) Question Answering December 1, 2016 21 / 25
  47. 47. A Little More Extraction For simple sentences: Who does michael keaton play in cars Who michael keaton cars Michael Keaton The Merry Gentleman Penthouse North cvt1 Cars cvt2 Chick Hicks film starring starring filmdirect direct by ctv3 spouse_s Caroline McWilliams spousespouse_s spouse 6/5/1982 from 1/29/1990 to Marriage type of union character portrayer film character starring charactercharacter Question: Star Graph: Freebase Graph: cvt3 film actor Leona Elizabeth Loftus George A. Douglas parents child Feng et al. (PKU) Question Answering December 1, 2016 22 / 25
  48. 48. A Little More Extraction For simple sentences: Join Entitly Linking and Relation Extraction gives 0.49 (+0.06) F1 on WebQuestions. Feng et al. (PKU) Question Answering December 1, 2016 22 / 25
  49. 49. With Hybrid Knowledge Base Resources a little bit complex... Where should a visitor see in Germany ? Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  50. 50. With Hybrid Knowledge Base Resources a little bit complex... Where should a visitor see in Germany ? What is the most popular crop during 1900s in USA ? Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  51. 51. With Hybrid Knowledge Base Resources a little bit complex... Where should a visitor see in Germany ? What is the most popular crop during 1900s in USA ? Who did Shaq first play for ? Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  52. 52. With Hybrid Knowledge Base Resources Either subjective, or hard to map against existing Structured KBs Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  53. 53. With Hybrid Knowledge Base Resources Either subjective, or hard to map against existing Structured KBs Using both structured knowledge bases and texts, e.g., Wikipedia or existing community QA archives. Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  54. 54. With Hybrid Knowledge Base Resources who did shaq first play for KB-QA Entity Linking Relation Extraction Joint Inference shaq: m.012xdf shaq: m.05n7bp shaq: m.06_ttvh sports.pro_athlete.teams..sports.sports_team_roster.team basketball.player.statistics..basketball.player_stats.team …… m.012xdf sports.pro_athlete.teams..sports.sports_team_roster.team Los Angeles Lakers, Boston Celtics, Orlando Magic, Miami Heat Freebase Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  55. 55. With Hybrid Knowledge Base Resources Answer Refinement Los Angeles Lakers, Boston Celtics, Orlando Magic, Miami Heat Freebase Shaquille O'Neal O'Neal signed as a free agent with the Los Angeles Lakers Shaquille O'Neal O'Neal played for the Boston Celtics in the 2010-11 season before retiring Shaquille O'Neal O'Neal was drafted in the 1992 NBA draft by the Orlando Magic with the first overall pick Los Angeles Lakers Boston Celtics Orlando Magic O’Neal was drafted by the Orlando Magic with the first overall pick in the 1992 NBA draft O’Neal played for the Boston Celtics in the 2010-11 season before retiring O’Neal signed as a free agent with the Los Angeles Lakers Refinement Model +- - Orlando Magic Wikipedia Dump Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  56. 56. With Hybrid Knowledge Base Resources Textual Relation Extraction Triple SolverTriple Solver who is the front man of the band that wrote Coffee & TV Question Decomposition < ans, is the front man of, var1 > < var1 , is a , band > < var1 , wrote , Coffee & TV > < var1 , wrote, Coffee & TV > Triple Solver Entity Linking Multi-Channel Neural Network Paraphrase Model Wikipedia Dump Textual KB KB-based Relation Extraction DBpedia Freebase DBpedia Lookup Coffee & TV Bitter_Coffee_(Iranian_video _series) Irish_Coffee_(TV_series) Coffee & TV influencedBy associatedMusicalArtist associatedBand writer front man of is written by lead vocalist of Open Information Extractor wrote is the front man of Joint Inference Damon Albarn Feng et al. (PKU) Question Answering December 1, 2016 23 / 25
  57. 57. Conclusion At this point... still a lot to do for GaoKao but, a flexible QA framework With multiple resources, e.g., structured knowledge bases, Wikipedia, text books, exercises, even news papers, etc. Our collaborations with IBM China Research Lab The Keys From natural languages to knowledge bases Inference over structured knowledge Answer with common-sense knowledge Understand various images, tables, figures, diagrams... Feng et al. (PKU) Question Answering December 1, 2016 24 / 25
  58. 58. Conclusion At this point... still a lot to do for GaoKao but, a flexible QA framework With multiple resources, e.g., structured knowledge bases, Wikipedia, text books, exercises, even news papers, etc. Our collaborations with IBM China Research Lab contribute to the Watson Competitions contribute to a Multi-Modal QA system (with Vision China) The Keys From natural languages to knowledge bases → on the way Inference over structured knowledge Answer with common-sense knowledge Understand various images, tables, figures, diagrams... Feng et al. (PKU) Question Answering December 1, 2016 24 / 25
  59. 59. Conclusion At this point... still a lot to do for GaoKao but, a flexible QA framework With multiple resources, e.g., structured knowledge bases, Wikipedia, text books, exercises, even news papers, etc. Our collaborations with IBM China Research Lab contribute to the Watson Competitions contribute to a Multi-Modal QA system (with Vision China) The Keys From natural languages to knowledge bases → on the way Inference over structured knowledge → challenging Answer with common-sense knowledge Understand various images, tables, figures, diagrams... Feng et al. (PKU) Question Answering December 1, 2016 24 / 25
  60. 60. Conclusion At this point... still a lot to do for GaoKao but, a flexible QA framework With multiple resources, e.g., structured knowledge bases, Wikipedia, text books, exercises, even news papers, etc. Our collaborations with IBM China Research Lab contribute to the Watson Competitions contribute to a Multi-Modal QA system (with Vision China) The Keys From natural languages to knowledge bases → on the way Inference over structured knowledge → challenging Answer with common-sense knowledge → still missing Understand various images, tables, figures, diagrams... → still missing Feng et al. (PKU) Question Answering December 1, 2016 24 / 25
  61. 61. Thanks & Questions Feng et al. (PKU) Question Answering December 1, 2016 25 / 25

×