Learning Multilingual Semantics from Big Data on the Web

1. Learning Multilingual Semantics from Big Data on the Web Gerard de Melo Assistant Professor, Tsinghua University http://gerard.demelo.org Learning Multilingual Semantics from Big Data on the Web Gerard de Melo Assistant Professor, Tsinghua University http://gerard.demelo.org

2. Life at Tsinghua

3. Life at Tsinghua

4. Big Data on the WebBig Data on the WebBig Data on the WebBig Data on the Web Matej Kren: Idiom. Prague Municipal Library https://www.flickr.com/photos/ill-padrino/6437837857/

5. From Big Data toFrom Big Data to Multilingual Semantics?Multilingual Semantics? From Big Data toFrom Big Data to Multilingual Semantics?Multilingual Semantics? Image: Brett Ryder

6. Manual Knowledge OrganizationManual Knowledge Organization Image: http://commons.wikimedia.org/wiki/File:Mundaneum_Tir%C3%A4ng_Karteikaarten.jpg Universal Bibliographic Repertory (Repertoire Bibliographique Universel, RBU) by Paul Otlet and Henri La Fontaine in 1895 index cards with answers to queries Universal Bibliographic Repertory (Repertoire Bibliographique Universel, RBU) by Paul Otlet and Henri La Fontaine in 1895 index cards with answers to queries

7. Manual Knowledge OrganizationManual Knowledge Organization Image: Mundaneum Universal Bibliographic Repertory (Repertoire Bibliographique Universel, RBU) by Paul Otlet and Henri La Fontaine in 1895 index cards with answers to queries Universal Bibliographic Repertory (Repertoire Bibliographique Universel, RBU) by Paul Otlet and Henri La Fontaine in 1895 index cards with answers to queries Alex Wright: This was a sort of “analog search engine” Alex Wright: This was a sort of “analog search engine”

8. Zipfian DistributionZipfian DistributionZipfian DistributionZipfian Distribution https://commons.wikimedia.org/wiki/File:Moby_Dick_Words.gif

9. Big Data on the WebBig Data on the WebBig Data on the WebBig Data on the Web +

10. Goal: Large YetGoal: Large Yet Reasonably Clean KnowledgeReasonably Clean Knowledge Goal: Large YetGoal: Large Yet Reasonably Clean KnowledgeReasonably Clean Knowledge Theological Hall, Strahov Monastery Library, Prague

11. OutlineOutline Large-Scale Knowledge Graphs Semantics in Action Models for the Future

13. Lexical Knowledge Portuguese-Chinese Dictionary by Ruggieri et al. (1580s) The first European-Chinese dictionary https://commons.wikimedia.org/wiki/File:Ricci-Ruggieri-Portuguese-Chinese-dictionary-page-1.png

14. Provides translations, antonyms, etc. WiktionaryWiktionary

15. WiktionaryWiktionary

16. WiktionaryWiktionary

17. e.g. “salary” < Lat. “salarius” < Lat. “sal” (salt) Etymological WordnetEtymological Wordnet

18. LREC 2014LREC 2014 Etymological WordnetEtymological Wordnet

19. LREC 2014LREC 2014 Etymological WordnetEtymological Wordnet

20. Etymological WordnetEtymological Wordnet Old English Example Old English Example

21. Lexical AmbiguitiesLexical Ambiguities

22. Hipsters in London Images: https://www.flickr.com/photos/poisonbabyfood/4274634681 https://www.facebook.com/alexander.balabanov.82 Lexical AmbiguitiesLexical Ambiguities

23. Reunion Lexical AmbiguitiesLexical Ambiguities

24. Reunion Images: https://commons.wikimedia.org/wiki/File:Reunions_Class_of_82_2007.jpg https://commons.wikimedia.org/wiki/File:Riviere_Langevin_Trou_Noir_P1440224-35.jpg and many more...and many more... Lexical AmbiguitiesLexical Ambiguities

25. Lexical Knowledge Bases

26. Multilingual Lexical Knowledge UWN (de Melo & Weikum 2009)

27. UWN: Universal Wordnet Before: manual work over two decades but not many large wordnets Before: manual work over two decades but not many large wordnets Our Approach: ● Exploit translation resources on the Web ● Learn regression model with sophisticated graph-based features Our Approach: ● Exploit translation resources on the Web ● Learn regression model with sophisticated graph-based features Gerard de Melo

28. UWN: Universal Wordnet Gerard de Melo

29. UWN: Universal Wordnet over 1,000,000 words in over 100 languages CIKM 2009CIKM 2009 ICGL 2008ICGL 2008 Best Paper AwardBest Paper Award ICGL 2008ICGL 2008 Best Paper AwardBest Paper Award Gerard de Melo

30. UWN: Taxonomy

31. UWN: Getting StartedUWN: Getting Started Simple API for JVM Languages val uwn = new UWN(new File("plugins/")) for (m <- uwn.getMeanings("souris", "fra")) println(m) Or Just Download the TSV File Simple API for JVM Languages val uwn = new UWN(new File("plugins/")) for (m <- uwn.getMeanings("souris", "fra")) println(m) Or Just Download the TSV File

32. Adding Other Sources Gerard de Melo Language-specific,Language-specific, Domain-specific,Domain-specific, Arbitrary DatabasesArbitrary Databases Language-specific,Language-specific, Domain-specific,Domain-specific, Arbitrary DatabasesArbitrary Databases

33. Adding Other SourcesAdding Other SourcesAdding Other SourcesAdding Other Sources https://commons.wikimedia.org/wiki/File:Encyclopedia_Britannica_in_the_library_of_The_Kings_School,_Goa.jpg

34. Adding Other SourcesAdding Other SourcesAdding Other SourcesAdding Other Sources Rob Matthews: printed small sample of Wikipedia Actually, a printed Wikipedia corresponds to 2000 Britannica volumes Source: http://www.labnol.org/internet/wikipedia-printed-book/9136/ Actually, a printed Wikipedia corresponds to 2000 Britannica volumes Source: http://www.labnol.org/internet/wikipedia-printed-book/9136/

35. ACL 2010 AAAI 2013 ACL 2010 AAAI 2013 Use Identity Links to connect What is equivalent Merging Structured DataMerging Structured DataMerging Structured DataMerging Structured Data

36. Merging Structured DataMerging Structured DataMerging Structured DataMerging Structured Data ACL 2010 AAAI 2013 ACL 2010 AAAI 2013

37. Merging Structured DataMerging Structured Data Trentino Trentino- Alto Adige

38. Merging Structured DataMerging Structured DataMerging Structured DataMerging Structured Data One bad link isOne bad link is enough to make aenough to make a connected componentconnected component inconsistentinconsistent One bad link isOne bad link is enough to make aenough to make a connected componentconnected component inconsistentinconsistent ACL 2010 AAAI 2013 ACL 2010 AAAI 2013

39. Source: Peter Mika Entity Integration: Challenges Entity Integration: Challenges

40. Merging Structured DataMerging Structured Data Distinctness Assertions Di = ({en: Province of Trento, en:Trentino}, {en:Trentino-South Tyrol, en:Trentino-Alto Adige/Südtirol}) Distinctness Assertions Di = ({en: Province of Trento, en:Trentino}, {en:Trentino-South Tyrol, en:Trentino-Alto Adige/Südtirol}) ACL 2010 AAAI 2013 ACL 2010 AAAI 2013

41. How to reconcileHow to reconcile equivalenceequivalence andand distinctnessdistinctness evidence?evidence? How to reconcileHow to reconcile equivalenceequivalence andand distinctnessdistinctness evidence?evidence? a) ignore somea) ignore some equivalence informationequivalence information (delete certain edges)(delete certain edges) a) ignore somea) ignore some equivalence informationequivalence information (delete certain edges)(delete certain edges) b) ignore someb) ignore some distinctness informationdistinctness information (remove node from(remove node from distinctness assertion)distinctness assertion) b) ignore someb) ignore some distinctness informationdistinctness information (remove node from(remove node from distinctness assertion)distinctness assertion) Merging Structured DataMerging Structured DataMerging Structured DataMerging Structured Data ACL 2010 AAAI 2013 ACL 2010 AAAI 2013

42. Min. cost solution:Min. cost solution: NP-hardNP-hard APX-hardAPX-hard Min. cost solution:Min. cost solution: NP-hardNP-hard APX-hardAPX-hard Merging Structured DataMerging Structured DataMerging Structured DataMerging Structured Data ACL 2010 AAAI 2013 ACL 2010 AAAI 2013

43. Finally, use region growingFinally, use region growing algorithm in the spiritalgorithm in the spirit of Leighton & Rao 1988of Leighton & Rao 1988 Finally, use region growingFinally, use region growing algorithm in the spiritalgorithm in the spirit of Leighton & Rao 1988of Leighton & Rao 1988 Linear Program RelaxationLinear Program RelaxationLinear Program RelaxationLinear Program Relaxation Approximation Guarantee:Approximation Guarantee: 4ln(nq+1)4ln(nq+1) for n distinctness assertions,for n distinctness assertions, q=max |Dq=max |Di,ji,j || but independent of |Dbut independent of |Dii | !| ! Approximation Guarantee:Approximation Guarantee: 4ln(nq+1)4ln(nq+1) for n distinctness assertions,for n distinctness assertions, q=max |Dq=max |Di,ji,j || but independent of |Dbut independent of |Dii | !| ! Merging Structured DataMerging Structured DataMerging Structured DataMerging Structured Data

44. Linear Program RelaxationLinear Program RelaxationLinear Program RelaxationLinear Program Relaxation Nice:Nice: This generalizes theThis generalizes the Hungarian AlgorithmHungarian Algorithm to various advancedto various advanced types of non-standardtypes of non-standard matchingsmatchings (cf. de Melo. AAAI 2013)(cf. de Melo. AAAI 2013) Nice:Nice: This generalizes theThis generalizes the Hungarian AlgorithmHungarian Algorithm to various advancedto various advanced types of non-standardtypes of non-standard matchingsmatchings (cf. de Melo. AAAI 2013)(cf. de Melo. AAAI 2013) Merging Structured DataMerging Structured DataMerging Structured DataMerging Structured Data

45. Separated ConceptsSeparated Concepts (Multilingual Wikipedia)(Multilingual Wikipedia) Separated ConceptsSeparated Concepts (Multilingual Wikipedia)(Multilingual Wikipedia)

46. Application: Lexvo.org Semantic WebSemantic Web Journal 2014Journal 2014 Semantic WebSemantic Web Journal 2014Journal 2014

47. Lexvo.orgLexvo.org Semantic WebSemantic Web Journal 2014Journal 2014 Semantic WebSemantic Web Journal 2014Journal 2014

48. Semantic WebSemantic Web Journal 2014Journal 2014 Semantic WebSemantic Web Journal 2014Journal 2014 InterdisciplinaryInterdisciplinary Work, e.g. inWork, e.g. in Digital HumanitiesDigital Humanities InterdisciplinaryInterdisciplinary Work, e.g. inWork, e.g. in Digital HumanitiesDigital Humanities Lexvo.orgLexvo.org

49. Taxonomic Organization a user wants a list of „Art Schools in Europe“

50. Multilingual Taxonomies a Swedish user wants a list of „Konstskolor i Europa“

51. De Melo & Weikum (2010). CIKM Best Interdisciplinary Paper Award De Melo & Weikum (2010). CIKM Best Interdisciplinary Paper Award Taxonomic Integration:Taxonomic Integration: MENTA ApproachMENTA Approach Taxonomic Integration:Taxonomic Integration: MENTA ApproachMENTA Approach

55. De Melo & Weikum (2010). CIKM Best Interdisciplinary Paper Award De Melo & Weikum (2010). CIKM Best Interdisciplinary Paper Award Predict Individual Taxonomic Links: Article → Category Category → WordNet Predict Individual Taxonomic Links: Article → Category Category → WordNet Taxonomic Integration:Taxonomic Integration: MENTAMENTA Taxonomic Integration:Taxonomic Integration: MENTAMENTA

56. Predict Individual Taxonomic Links: Article → Category Category → WordNet Predict Individual Taxonomic Links: Article → Category Category → WordNet Taxonomic Integration:Taxonomic Integration: MENTAMENTA Taxonomic Integration:Taxonomic Integration: MENTAMENTA

57. Taxonomic Integration:Taxonomic Integration: MENTAMENTA Taxonomic Integration:Taxonomic Integration: MENTAMENTA

58. Taxonomic Integration:Taxonomic Integration: MENTAMENTA Taxonomic Integration:Taxonomic Integration: MENTAMENTA Image: https://de.wikipedia.org/wiki/Datei:Bersntol_palae.jpg Fersental (Bersntol, Valle dei Mòcheni) Fersental (Bersntol, Valle dei Mòcheni)

60. Taxonomic Integration:Taxonomic Integration: MENTAMENTA Taxonomic Integration:Taxonomic Integration: MENTAMENTA https://de.wikipedia.org/wiki/Datei:Language_distribution_Trentino_2011.png Fersental (Bersntol, Valle dei Mòcheni) Fersental (Bersntol, Valle dei Mòcheni)

62. Use Identity Constraint Algorithm to form equivalence classes Use Identity Constraint Algorithm to form equivalence classes Markov Chain Random Walk with Restarts to Rank Parents Markov Chain Random Walk with Restarts to Rank Parents Taxonomic Integration:Taxonomic Integration: MENTAMENTA Taxonomic Integration:Taxonomic Integration: MENTAMENTA

65. Bansal et al.Bansal et al. ACL 2014. Best Paper Runner-UpACL 2014. Best Paper Runner-Up Bansal et al.Bansal et al. ACL 2014. Best Paper Runner-UpACL 2014. Best Paper Runner-Up Bansal et al.Bansal et al. ACL 2014. Best Paper Runner-UpACL 2014. Best Paper Runner-Up Bansal et al.Bansal et al. ACL 2014. Best Paper Runner-UpACL 2014. Best Paper Runner-Up Belief PropagationBelief Propagation exploiting Kirchhoff’sexploiting Kirchhoff’s Matrix Tree TheoremMatrix Tree Theorem for efficient handling offor efficient handling of tree factortree factor Belief PropagationBelief Propagation exploiting Kirchhoff’sexploiting Kirchhoff’s Matrix Tree TheoremMatrix Tree Theorem for efficient handling offor efficient handling of tree factortree factor Chu-Liu-EdmondsChu-Liu-Edmonds directed spanning treedirected spanning tree algorithm for decodingalgorithm for decoding Chu-Liu-EdmondsChu-Liu-Edmonds directed spanning treedirected spanning tree algorithm for decodingalgorithm for decoding New Algorithm: Structured Output Prediction New Algorithm: Structured Output Prediction

66. UWN/MENTA CIKM 2010CIKM 2010 Best Paper AwardBest Paper Award CIKM 2010CIKM 2010 Best Paper AwardBest Paper Award Biggest (ontological)Biggest (ontological) taxonomytaxonomy Biggest (ontological)Biggest (ontological) taxonomytaxonomy

67. UWN/MENTA multilingual extension of WordNet for word senses and taxonomical information over 200 languages Gerard de Melo

69. Language EducationLanguage EducationLanguage EducationLanguage Education

70. UWNUWNUWNUWN http://www.lexvo.org/uwn/http://www.lexvo.org/uwn/http://www.lexvo.org/uwn/http://www.lexvo.org/uwn/

74. Application: Sense-DisambiguatedApplication: Sense-Disambiguated Example SentencesExample Sentences Application: Sense-DisambiguatedApplication: Sense-Disambiguated Example SentencesExample Sentences

77. Application: Monolingual Language Users Application: Monolingual Language Users

78. Application: Monolingual Language Users Application: Monolingual Language Users

79. ThesauriThesauri See also: de Melo & Weikum (2008).

80. ThesauriThesauri Borin, Allwood, de Melo. LREC 2014.

81. Application: Machine TranslationApplication: Machine Translation OpenWN-PT: Used by Google Translate OpenWN-PT: Used by Google Translate

82. Machine LearningMachine Learning Examples Incorrect Correct

83. Machine LearningMachine Learning Examples LearningLearning Incorrect Correct ClassifierModel

84. Machine LearningMachine Learning Examples LearningLearning Incorrect Correct ClassifierModel ???

85. Machine LearningMachine Learning Examples Probably Incorrect! LearningLearning PredictionPrediction Incorrect Correct ClassifierModel

86. Better Machine LearningBetter Machine Learning Examples Probably Incorrect! LearningLearning PredictionPrediction Incorrect Correct ClassifierModel Better Classifier! + Better Labels for Test Data

87. MT?

88. MT?

89. MT?

90. UWN Senses in MT? Issue: Senses should be less fine-grained Issue: Senses should be less fine-grained

91. No Word Left Behind Web page: http://www.buzzfeed.com/paulf24/24-signs-youre-in-a-pretty-rad-relationship-b5ra#.txpBGq4p4

95. Similar: Part-Of-Speech TaggingSimilar: Part-Of-Speech Tagging ● British fans gathered at the stadium to... ADJECTIVE “Didgeridoo” is similar to: “horn” (NOUN) “drums” (NOUN) “accordion” (NOUN) “Didgeridoo” is similar to: “horn” (NOUN) “drums” (NOUN) “accordion” (NOUN) Didgeridoo fans gathered at the park to... ???

96. Similar: Part-Of-Speech TaggingSimilar: Part-Of-Speech Tagging ● British fans gathered at the stadium to... ADJECTIVE Gaelic “didiridiú” translates to “didgeridoo” (NOUN) in English Gaelic “didiridiú” translates to “didgeridoo” (NOUN) in English ...Astrálach is ea an didiridiú ???

97. Sentence LevelSentence Level

100. What about Document-Level Tasks? What about Document-Level Tasks? Public Domain Image from https://pixabay.com/en/book-text-read-paper-education-451067/

101. “new” 1.0 “york” 1.0 “jaguar” 1.0 “automobile” 0.0 “car” 0.0 “10th” 1.0 “street” 1.0 “show” 1.0 ... ... New_York 1.0 Jaguar (car) 0.0 Jaguar (animal) 1.0 Automobile/Car 0.0 10th Street 1.0 Performance 1.0 ... ... “10th street new york jaguar show” Similar: “10th New show in York” “New Jaguar show” “Show New Street in York” “10th street new york jaguar show” Similar: “10th street nyc jaguar show” Document LevelDocument Level

102. “new” 1.0 “york” 1.0 “jaguar” 1.0 “automobile” 0.0 “car” 0.0 “10th” 1.0 “street” 1.0 “show” 1.0 ... ... New_York 1.0 Jaguar (car) 0.0 Jaguar (animal) 1.0 Automobile/Car 0.0 10th Street 1.0 Performance 1.0 ... ... Animal 0.5 Vehicle 0.0 “10th street new york jaguar show” Similar: “10th New show in York” “New Jaguar show” “Show New Street in York” “10th street new york jaguar show” Similar: “10th street nyc jaguar show” “10th street nyc animal show” “Exposición de jaguares Nueva York” Expansion (de Melo & Siersdorfer 2007) Document LevelDocument Level

103. Given: training documents with class labels Goal: guess class labels for test documents in some other language Result: better than plain machine translation. See de Melo & Siersdorfer 2007. Multilingual Tasks: Cross-Lingual Text Classification Multilingual Tasks: Cross-Lingual Text Classification

104. Underlying frame: Commercial transfer Capture the “who-did-what-to-whom” Microsoft bought the patent from Nokia. Nokia sold the patent to Microsoft. The patent was acquired by Microsoft [from Nokia]. The patent was sold [by Nokia] to Microsoft. Sentence-Level SemanticsSentence-Level Semantics Buyer: Microsoft Seller: Nokia Product: The patent

105. FrameBase.org Bringing knowledge into a standard form based on natural language (FrameNet) Bringing knowledge into a standard form based on natural language (FrameNet) ESWC 2015 Best Student Paper Nominee ESWC 2015 Best Student Paper Nominee

106. Relation IntegrationRelation Integration X isAuthorOf Y Y writtenBy X X wrote Y Y writtenInYear Z ESWC 2015 Best Student Paper Nominee ESWC 2015 Best Student Paper Nominee

107. Relation IntegrationRelation Integration YAGO: isMarriedTo predicateYAGO: isMarriedTo predicate Freebase: Marriage EntityFreebase: Marriage Entity Challenge: Modelling Differences Challenge: Modelling Differences

108. Search Interfaces “Which companies were created during the last century in Silicon Valley ?” YAGO2: WWW 2011 Best Demo Award YAGO2: WWW 2011 Best Demo Award Gerard de Melo

109. Answering Questions IBM's Jeopardy!-winning Watson system Gerard de Melo

110. Answering Questions IBM's Jeopardy!-winning Watson system Gerard de Melo

111. What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors? Jiaqiang Chen and Gerard de Melo 2015

112. What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors? The Roman Empire was remarkably multicultural, with ”a rather astonishing cohesive capacity” to create a sense of shared identity while encompassing diverse peoples within its political system over a long span of time. Jiaqiang Chen and Gerard de Melo 2015

113. What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors? The Roman Empire was remarkably multicultural, with ”a rather astonishing cohesive capacity” to create a sense of shared identity while encompassing diverse peoples within its political system over a long span of time. syntactic Jiaqiang Chen and Gerard de Melo 2015

114. What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors? The Roman Empire was remarkably multicultural, with ”a rather astonishing cohesive capacity” to create a sense of shared identity while encompassing diverse peoples within its political system over a long span of time. syntactic semantic! Jiaqiang Chen and Gerard de Melo 2015

115. What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors? The Roman Empire was remarkably multicultural, with ”a rather astonishing cohesive capacity” to create a sense of shared identity while encompassing diverse peoples within its political system over a long span of time. semantic!syntactic syntactic? Jiaqiang Chen and Gerard de Melo 2015

116. What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors? The Roman Empire was remarkably multicultural, with ”a rather astonishing cohesive capacity” to create a sense of shared identity while encompassing diverse peoples within its political system over a long span of time. semantic!syntactic syntactic? ? Jiaqiang Chen and Gerard de Melo 2015

117. What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors?What Goes into Word Vectors? The Roman Empire was remarkably multicultural, with ”a rather astonishing cohesive capacity” to create a sense of shared identity while encompassing diverse peoples within its political system over a long span of time. semantic!syntactic ? Word2Vec Solution: Subsampling Word2Vec Solution: Subsampling syntactic? Jiaqiang Chen and Gerard de Melo 2015

118. Word2Vec ApproachWord2Vec ApproachWord2Vec ApproachWord2Vec Approach Alexandre Duret-Lutz https://www.flickr.com/photos/gadl/110845690/ Take everything we can get Take everything we can get

119. Our Proposal:Our Proposal: Extract the Most Valuable PartsExtract the Most Valuable Parts Our Proposal:Our Proposal: Extract the Most Valuable PartsExtract the Most Valuable Parts Theological Hall, Strahov Monastery Library, Prague

120. …Greek and Roman mythology... Our Proposal:Our Proposal: Extract the Most Valuable PartsExtract the Most Valuable Parts Our Proposal:Our Proposal: Extract the Most Valuable PartsExtract the Most Valuable Parts semantic! look for semantically salient contexts in text! look for semantically salient contexts in text! Jiaqiang Chen and Gerard de Melo 2015

121. Two WorldsTwo Worlds Jiaqiang Chen and Gerard de Melo 2015 Distributional Semantics: Use all available text (Symbolic) Information Extraction: Look for valuable connections

122. Proposed Research Program: Joint Training Proposed Research Program: Joint Training Better Word Embeddings Joint Training Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

123. Proposed Research Program: Joint Training Proposed Research Program: Joint Training Better Word Embeddings Joint Training Jiaqiang Chen and Gerard de Melo 2015 Use parallel threads Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

124. Preliminary Experiments:Preliminary Experiments: Joint TrainingJoint Training Preliminary Experiments:Preliminary Experiments: Joint TrainingJoint Training Recently lots of related work: E.g. Faruqui et al., Hill & Korhonen, Wang et al., Johansson & Nieto Piña Recently lots of related work: E.g. Faruqui et al., Hill & Korhonen, Wang et al., Johansson & Nieto Piña Jiaqiang Chen and Gerard de Melo 2015

125. Preliminary Experiments:Preliminary Experiments: Joint TrainingJoint Training Preliminary Experiments:Preliminary Experiments: Joint TrainingJoint Training Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

126. Preliminary Experiments:Preliminary Experiments: Joint TrainingJoint Training Preliminary Experiments:Preliminary Experiments: Joint TrainingJoint Training Use negative samplingUse negative sampling Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

127. Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Jiaqiang Chen and Gerard de Melo 2015 Variant 1: Definition ExtractionVariant 1: Definition Extraction Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

128. Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Jiaqiang Chen and Gerard de Melo 2015 Definitions befuddle: to becloud and confuse as with liquor befuddled: dazed by alcoholic drink befuddled: confused and vague used especially of thinking beg: to ask earnestly for, to entreat or supplicate for, to beseech Variant 1: Definition ExtractionVariant 1: Definition Extraction Source: GCIDE Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

129. Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Synonyms effectual: effectual efficacious effective effectuality: effectiveness effectivity effectualness efficacious: effectual efficaciousness: efficacy Jiaqiang Chen and Gerard de Melo 2015 Variant 1: Definition ExtractionVariant 1: Definition Extraction Source: GCIDE Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

130. Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Jiaqiang Chen and Gerard de Melo 2015 Variant 2: List ExtractionVariant 2: List Extraction Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

131. Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Jiaqiang Chen and Gerard de Melo 2015 ● Look for repeated occurrences of commas ● Short units of roughly equal length ● noun phrases, adjectives Variant 2: List ExtractionVariant 2: List Extraction Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

132. Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Jiaqiang Chen and Gerard de Melo 2015 ● Look for repeated occurrences of commas ● Short units of roughly equal length ● noun phrases, adjectives ● Also: Hearst patterns, e.g. “cities such as New York, London, ...” Variant 2: List ExtractionVariant 2: List Extraction Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

133. Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Preliminary Experiments:Preliminary Experiments: Information ExtractionInformation Extraction Jiaqiang Chen and Gerard de Melo 2015 Extracted Lists player captain manager director vice-chairman group race culture religion organisation person person Italian Mexican Chinese Creole French Self-Portraits Portraits iris Still-Lives with Sunflowers view from the Asylum Works after Millet Vineyards ballscrews leadscrews worm gear screwjacks linear actuator Cleveland Essex Lincolnshire Northamptonshire Nottinghamshire Thames Valley South Wales ant.py dimdriver.py dimdriverdatafile.py dimdriverdatasetdef.py dimexception.py dimmaker.py dimoperators.py dimparser.py dimrex.py dimension.py Variant 2: List ExtractionVariant 2: List Extraction

134. Preliminary Experiments:Preliminary Experiments: SetupSetup Preliminary Experiments:Preliminary Experiments: SetupSetup Wikipedia 2010 normalize to lower case and remove special characters Contain 1,205,009,010 words Select words appearing at least 50 times Vocabulary size 220,521 Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

135. Preliminary Experiments:Preliminary Experiments: SetupSetup Preliminary Experiments:Preliminary Experiments: SetupSetup Wikipedia 2010 normalize to lower case and remove special characters Contain 1,205,009,010 words Select words appearing at least 50 times Vocabulary size 220,521 Balance Components simply by controlling starting learning rates: 0.05 for CBOW, varying rates for extracted information Balance Components simply by controlling starting learning rates: 0.05 for CBOW, varying rates for extracted information Vector dim. 300 Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

136. Preliminary Experiments:Preliminary Experiments: Results on WS353Results on WS353 Preliminary Experiments:Preliminary Experiments: Results on WS353Results on WS353 Positive effect from 0.001 until around 0.04 Positive effect from 0.001 until around 0.04 Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

137. Preliminary Experiments:Preliminary Experiments: ExampleExample Preliminary Experiments:Preliminary Experiments: ExampleExample Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

138. Preliminary Experiments:Preliminary Experiments: ExampleExample Preliminary Experiments:Preliminary Experiments: ExampleExample Jiaqiang Chen and Gerard de Melo 2015 Best Paper Award at NAACL 2015 Vector Space Modeling Workshop Best Paper Award at NAACL 2015 Vector Space Modeling Workshop

140. History Repeating?History Repeating?History Repeating?History Repeating? SMTSMT NMTNMT Phrase-Based SMT Hierarchical Phrases WSD, MEANT etc. Phrase-Based SMT Hierarchical Phrases WSD, MEANT etc. Extended NMT?Extended NMT?

141. Well-Known IssuesWell-Known IssuesWell-Known IssuesWell-Known Issues

142. Source: The New Yorker Future:Future: Learning Common-SenseLearning Common-Sense Future:Future: Learning Common-SenseLearning Common-Sense

143. Learning Common-SenseLearning Common-SenseLearning Common-SenseLearning Common-Sense WebChild AAAI 2014 WSDM 2014 AAAI 2011 WebChild AAAI 2014 WSDM 2014 AAAI 2011

144. Lexical Intensity OrderingsLexical Intensity Orderings hothot warmwarm fieryfiery scorchingscorching < < < weak strong TACL 2013TACL 2013

145. Knowlywood: Human ActivitiesKnowlywood: Human Activities CIKM 2015CIKM 2015

146. Extension to RelationshipsExtension to RelationshipsExtension to RelationshipsExtension to Relationships http://www.wikihow.com/Read-a-Book-to-a-Baby-or-Infant#/Image:Read-a-Book-to-a-Baby-or-Infant-Step-5.jpg

147. Extension to RelationshipsExtension to RelationshipsExtension to RelationshipsExtension to Relationships x x x x petronia sparrow parched arid xdry x bird http://www.wikihow.com/Read-a-Book-to-a-Baby-or-Infant#/Image:Read-a-Book-to-a-Baby-or-Infant-Step-5.jpg

148. Extension to RelationshipsExtension to RelationshipsExtension to RelationshipsExtension to Relationships x x x x petronia sparrow parched arid xdry x bird http://www.wikihow.com/Read-a-Book-to-a-Baby-or-Infant#/Image:Read-a-Book-to-a-Baby-or-Infant-Step-5.jpg Should account for relationships (incl. affordances, causality, etc.) Should account for relationships (incl. affordances, causality, etc.)

149. Extension to RelationshipsExtension to RelationshipsExtension to RelationshipsExtension to Relationships Assume that she is learning just from text Assume that she is learning just from text

150. 1. Gather large amounts of Patterns 2. Use Web-Scale Data (Google N-Grams, derived from 10^12 words of text) Hearst-style Bootstrapping with large numbers of seeds Gerard de Melo Information Extraction from TextInformation Extraction from Text

151. Extension to RelationshipsExtension to Relationships Commonsense word relationships extracted from Google 1T n-grams 24 relations bootstrapped via ConceptNet → 1,158,141 triples Jiaqiang Chen, Niket Tandon, Gerard de Melo. WI 2015

152. Extension to RelationshipsExtension to Relationships earring hasProperty gorgeous concept definedAs theory sonar partOf submarine predator desires food Commonsense word relationships extracted from Google 1T n-grams 24 relations bootstrapped via ConceptNet → 1,158,141 triples Jiaqiang Chen, Niket Tandon, Gerard de Melo. WI 2015

153. Extension to RelationshipsExtension to RelationshipsExtension to RelationshipsExtension to Relationships Jiaqiang Chen, Niket Tandon, Gerard de Melo. WI 2015

154. Extension to RelationshipsExtension to RelationshipsExtension to RelationshipsExtension to Relationships What causes cancer?

155. Extension to RelationshipsExtension to RelationshipsExtension to RelationshipsExtension to Relationships Can cats fly?

156. Summary Large-Scale Knowledge Graphs ► Universal WordNet/MENTA: large multilingual taxonomy ► Etymological WordNet Semantics in Action, e.g. ► Lexvo.org ► Question Answering with YAGO Future Perspectives ► Vector Representations ► Common-Sense for NLU More Information: www.demelo.org gdm@demelo.org More Information: www.demelo.org gdm@demelo.org Gerard de Melo

Learning Multilingual Semantics from Big Data on the Web

You are reading a preview.

Check these out next

Learning Multilingual Semantics from Big Data on the Web

More Related Content

Slideshows for you (20)

Similar to Learning Multilingual Semantics from Big Data on the Web (20)

More from Gerard de Melo (15)

Recently uploaded (20)