Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
SensEmbed: Learning Sense Embeddings
for Word and Relational Similarity
Ignacio Iacobacci, Mohammad Taher Pilehvar, Robert...
Vector Space Model
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Word Embeddings
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Bengio et al. (2003)
Mikolov et al. (2013)
Collobert & Weston (2008)
Pennington et al. (2014)
SensEmbed: Learning Sense Em...
Problem I
Word Embeddings cannot capture polysemy
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ...
Bankn
1: Sloping land (especially the
slope beside a body of water).
Bankn
2: A financial institution that
accepts deposits...
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Problem II
Word Embeddings do not take advantage of
existing semantic resources
SensEmbed: Learning Sense Embeddings for W...
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Our Solution
Sense Embeddings
+
Semantic Knowledge
=
SensEmbed
SensEmbed: Learning Sense Embeddings for Word and Relationa...
What are Sense Embeddings?
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Building SensEmbed
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Software: Word2vec - CBOW architecture
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP ...
SensEmbed construction
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed construction
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed construction
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed construction
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed construction
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Associating words with senses
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Associating words with senses
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Associating words with senses
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Associating words with senses
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Measuring Similarity
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Measuring Similarity
Vector Similarity
Cosine Similarity
S(w1, w2) =
w1 · w2
w1 · w2
(1)
Tanimoto Similarity
T (w1, w2) =
...
Closest senses to two senses of two ambiguous nouns:
{ bank, number }
SensEmbed: Learning Sense Embeddings for Word and Re...
Leveraging BabelNet for SensEmbed
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Improving Coverage
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Improving Coverage
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Expansion
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Less Frequent Words
SimLex-999, Hill et al. 2014
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity A...
Handling Less Frequent Words
Sense frequency
Orthodontist occur only 70 times in our training corpus.
Rationalize occur on...
Graph Vicinity Factor
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed for Word and Relational Similarity
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-I...
Word Similarity
Measures degree of semantic similarity and relatedness
between terms.
SensEmbed: Learning Sense Embeddings...
Strategies - Closest
Measuring Similarity
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCN...
Strategies - Closest
Measuring Similarity
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCN...
Strategies - Closest
Measuring Similarity
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCN...
Strategies - Closest
Measuring Similarity
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCN...
Strategies - Closest
Measuring Similarity
Closest Similarity
Simclosest (w1, w2) = max
s1∈Sw1
s2∈Sw2
T (s1, s2) (3)
SensEm...
Strategies - Closest
WordSim-353, Finkelstein et al., 2002
SensEmbed: Learning Sense Embeddings for Word and Relational Si...
Strategies - Weighted
Measuring Similarity
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJC...
Strategies - Weighted
Measuring Similarity
Weighted Similarity
Simweighted (w1, w2) =
s1∈Sw1 s2∈Sw2
d(s1) d(s2) T (s1, s2)...
Relational Similarity
Relational similarity evaluates the correspondence
between relations.
SensEmbed: Learning Sense Embe...
Relational Similarity
Relational Similarity
Paris − France + Italy ≈ Rome (6)
Mikolov et al. (2013)
SensEmbed: Learning Se...
Relational Similarity
Analogy
Analogy(wa, wb, wc, wd ) = T (wb − wa, wd − wc) (7)
SensEmbed: Learning Sense Embeddings for...
Experiments
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Word Similarity
Datasets
RG-65 (Rubenstein and Goodenough, 1965) for measuring word
similarity, made only with nouns.
Word...
Word Similarity - From Words to Senses
Spearman Correlation across Datasets
SensEmbed: Learning Sense Embeddings for Word ...
Word Similarity
Spearman Correlation across Datasets
SensEmbed: Learning Sense Embeddings for Word and Relational Similari...
Word Similarity
Similar approaches
WordSim-353 Spearman Correlation
SensEmbed: Learning Sense Embeddings for Word and Rela...
Relational Similarity
Semeval 2012 - Relational Similarity Task
SensEmbed: Learning Sense Embeddings for Word and Relation...
Conclusion
A novel approach for obtaining continuous representations of
individual word senses.
SensEmbed: Learning Sense ...
Conclusion
A novel approach for obtaining continuous representations of
individual word senses.
SensEmbed, effective simila...
Conclusion
A novel approach for obtaining continuous representations of
individual word senses.
We put forward SensEmbed, ...
Conclusion
A novel approach for obtaining continuous representations of
individual word senses.
We put forward SensEmbed, ...
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Backup Slides
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
Word Similarity
Similarity
Spearman Correlation RG-65
SensEmbed: Learning Sense Embeddings for Word and Relational Similar...
Word Similarity
Similarity
Spearman Correlation WS-Sim
SensEmbed: Learning Sense Embeddings for Word and Relational Simila...
Word Similarity
Relatedness
Spearman Correlation WS-Rel
SensEmbed: Learning Sense Embeddings for Word and Relational Simil...
Word Similarity
Only Verbs
Spearman Correlation YP-130
SensEmbed: Learning Sense Embeddings for Word and Relational Simila...
Word Similarity
Spearman Correlation MEN
SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNL...
Relational Similarity
Definition
Definition
Relational similarity evaluates the correspondence between relations.
Medin, D. ...
Word Similarity - From Words to Senses
Spearman Correlation across Datasets
SensEmbed: Learning Sense Embeddings for Word ...
Upcoming SlideShare
Loading in …5
×

SensEmbed Oral Presentation ACL 2015

127 views

Published on

SensEmbed Oral Presentation

Published in: Science
  • Be the first to comment

  • Be the first to like this

SensEmbed Oral Presentation ACL 2015

  1. 1. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity Ignacio Iacobacci, Mohammad Taher Pilehvar, Roberto Navigli {iacobacci,pilehvar,navigli}@di.uniroma1.it http://lcl.uniroma1.it ERC Starting Grant n. 259234 ACL-IJCNLP 2015 - 25-31 July 2015 Beijing, China
  2. 2. Vector Space Model SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  3. 3. Word Embeddings SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  4. 4. Bengio et al. (2003) Mikolov et al. (2013) Collobert & Weston (2008) Pennington et al. (2014) SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  5. 5. Problem I Word Embeddings cannot capture polysemy SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  6. 6. Bankn 1: Sloping land (especially the slope beside a body of water). Bankn 2: A financial institution that accepts deposits and channels the money into lending activities. Bankn 11: is a part of the sea which is shallow compared to its surrounding area, such as a shoal or the top of an underwater hill. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  7. 7. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  8. 8. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  9. 9. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  10. 10. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  11. 11. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  12. 12. Problem II Word Embeddings do not take advantage of existing semantic resources SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  13. 13. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  14. 14. Our Solution Sense Embeddings + Semantic Knowledge = SensEmbed SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  15. 15. What are Sense Embeddings? SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  16. 16. Building SensEmbed SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  17. 17. Software: Word2vec - CBOW architecture SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  18. 18. SensEmbed construction SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  19. 19. SensEmbed construction SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  20. 20. SensEmbed construction SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  21. 21. SensEmbed construction SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  22. 22. SensEmbed construction SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  23. 23. Associating words with senses SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  24. 24. Associating words with senses SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  25. 25. Associating words with senses SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  26. 26. Associating words with senses SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  27. 27. Measuring Similarity SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  28. 28. Measuring Similarity Vector Similarity Cosine Similarity S(w1, w2) = w1 · w2 w1 · w2 (1) Tanimoto Similarity T (w1, w2) = w1 · w2 w1 2 + w2 2 − w1 · w2 (2) SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  29. 29. Closest senses to two senses of two ambiguous nouns: { bank, number } SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  30. 30. Leveraging BabelNet for SensEmbed SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  31. 31. Improving Coverage SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  32. 32. Improving Coverage SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  33. 33. Expansion SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  34. 34. Less Frequent Words SimLex-999, Hill et al. 2014 SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  35. 35. Handling Less Frequent Words Sense frequency Orthodontist occur only 70 times in our training corpus. Rationalize occur only 187 times in our training corpus. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  36. 36. Graph Vicinity Factor SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  37. 37. SensEmbed for Word and Relational Similarity SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  38. 38. Word Similarity Measures degree of semantic similarity and relatedness between terms. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  39. 39. Strategies - Closest Measuring Similarity SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  40. 40. Strategies - Closest Measuring Similarity SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  41. 41. Strategies - Closest Measuring Similarity SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  42. 42. Strategies - Closest Measuring Similarity SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  43. 43. Strategies - Closest Measuring Similarity Closest Similarity Simclosest (w1, w2) = max s1∈Sw1 s2∈Sw2 T (s1, s2) (3) SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  44. 44. Strategies - Closest WordSim-353, Finkelstein et al., 2002 SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  45. 45. Strategies - Weighted Measuring Similarity SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  46. 46. Strategies - Weighted Measuring Similarity Weighted Similarity Simweighted (w1, w2) = s1∈Sw1 s2∈Sw2 d(s1) d(s2) T (s1, s2)α (4) Normalized Frequency d(s) = freq(s) s ∈Sw freq(s ) (5) SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  47. 47. Relational Similarity Relational similarity evaluates the correspondence between relations. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  48. 48. Relational Similarity Relational Similarity Paris − France + Italy ≈ Rome (6) Mikolov et al. (2013) SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  49. 49. Relational Similarity Analogy Analogy(wa, wb, wc, wd ) = T (wb − wa, wd − wc) (7) SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  50. 50. Experiments SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  51. 51. Word Similarity Datasets RG-65 (Rubenstein and Goodenough, 1965) for measuring word similarity, made only with nouns. WordSim-353 (Finkelstein et al., 2002) dataset. WS-Sim (Agirre et al. 2009) subset of WS-353 on word similarity. WS-Rel (Agirre et al. 2009) subset of WS-353 on word relatedness. YP-130 dataset (Yang and Powers 2005) for measuring verb similarity. MEN (Bruni et al., 2014), a general word-similarity dataset obtained by crowdsourcing. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  52. 52. Word Similarity - From Words to Senses Spearman Correlation across Datasets SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  53. 53. Word Similarity Spearman Correlation across Datasets SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  54. 54. Word Similarity Similar approaches WordSim-353 Spearman Correlation SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  55. 55. Relational Similarity Semeval 2012 - Relational Similarity Task SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  56. 56. Conclusion A novel approach for obtaining continuous representations of individual word senses. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  57. 57. Conclusion A novel approach for obtaining continuous representations of individual word senses. SensEmbed, effective similarity measurement: sense-specific representations + lexical-semantic knowledge. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  58. 58. Conclusion A novel approach for obtaining continuous representations of individual word senses. We put forward SensEmbed, effective similarity measurement: sense-specific representations + lexical-semantic knowledge. Consistent improvements upon moving from word to sense representation. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  59. 59. Conclusion A novel approach for obtaining continuous representations of individual word senses. We put forward SensEmbed, effective similarity measurement: sense-specific representations + lexical-semantic knowledge. Consistent improvements upon moving from word to sense representation. State-of-the-art performance on multiple tasks: word and relational similarity. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  60. 60. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  61. 61. SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  62. 62. Backup Slides SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  63. 63. Word Similarity Similarity Spearman Correlation RG-65 SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  64. 64. Word Similarity Similarity Spearman Correlation WS-Sim SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  65. 65. Word Similarity Relatedness Spearman Correlation WS-Rel SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  66. 66. Word Similarity Only Verbs Spearman Correlation YP-130 SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  67. 67. Word Similarity Spearman Correlation MEN SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  68. 68. Relational Similarity Definition Definition Relational similarity evaluates the correspondence between relations. Medin, D. et al. (1990) Definition When two pairs of words have a high degree of relational similarity, we say that their relations are analogous. Turney, P. (2006) SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015
  69. 69. Word Similarity - From Words to Senses Spearman Correlation across Datasets SensEmbed: Learning Sense Embeddings for Word and Relational Similarity ACL-IJCNLP 2015

×