This document discusses methods for creating word embeddings from text using distributional semantics and dependency structures. It presents experiments comparing word embeddings generated from various dependency structures to traditional Word2Vec models. The embeddings are evaluated using WordNet and VerbNet lexical databases, as well as in a sentiment analysis task. The results show that embeddings incorporating dependency structures outperform topical models on these evaluations, and that certain verb classes and thematic roles yield better embeddings than others. The document concludes that structure-based embeddings capture semantic information better than topical models.

1. Advances in Methods and
Evaluations for Distributional
Semantic Models using
Computational Lexicons
By Meera Hahn
Advisor: Jinho Choi

2. Distributional Semantics
Words used in similar contexts have similar semantic
and functional meaning - Harris

3. Word Embeddings
• From words to dense vectors
• Capturing semantics in a quantifiable vector
• Uses of these embeddings
• Current embedding methods
• Changing the game of natural language processing

4. Word Embeddings
KING – MAN + WOMEN = QUEEN

5. Word2Vec
I threw a green ball across the yard to him
vs.
I threw a green Frisbee across the yard to him

6. Word2Vec
Skip-Gram Architecture
Continuous Bag of Words
Architecture
I threw a green ball across the yard to him
3 4 5 6210 7-1 8
threw
I
a
green
threw
I
a
green

7. Dependency Structure

8. Dependency Structure Word
Embeddings
• Levy & Goldberg, 2014.
• Used only the head word and word and dependents
• Evaluated on differences of similarity (function) and
relatedness (topical)
• Evaluated on a dataset called WordSim, which is
mostly nouns
• Did not look at if the embeddings created a better
overall model nor did they try varying structures

9. Dependency Structure

10. Predicate Structure
• ARG0: agent
• ARG1: patient, theme
• ARG2: instrument, benefactive, attribute
• ARG3: starting point
• ARG4: ending point
• ARG5: external causer

11. Experiments
• DEP1: the first order dependencies of W
• DEP1H: the first order dependencies of W and the dependency head of W
• DEP12H: the first and second order dependencies of W and the dependency head of W
• DEP1SIB1: the first order dependencies of W, the rightmost sibling of W and the
leftmost sibling of W
• DEP1ALLSIB: the first order dependencies of W, all siblings of W
• DEP1SRLH: the first order dependencies of W, the semantic head of W
• DEP1SRLARG: the first order dependencies of W, all the semantic arguments of W

12. System Overview

13. Evaluations
• Used 2 lexical databases for experiments
• WordNet
• Synsets
• WordNet similarity measurements: LIN, LCH, WUP
• VerbNet
• Categorizes all verbs into verb classes
Members assume, adopt, take
Thematic Roles agent, theme
Semantic Restriction animate, organization
Ex. of verb class Adopt-93

14. WordNet
• Creation of similarity matrices
• WordNet Similarity Matrix
• Word Embedding Similarity Matrix
• Comparison of matrices
• ranking correlation: Spearman’s and Kendall’s

15. Example Similarity Matrix
w1 w2 … wn
w1 WS(w1,w1) WS(w1,w2) … WS(w1,wn)
w2 WS(w2,w1) WS(w2,w2) … WS(w2,wn)
… … … … …
wn WS(wn,w1) WS(wn,w2) … WS(wn,wn)

16. Kendall’s Ranking Correlation for Nouns

17. Kendall’s Ranking Correlation for Adjectives

18. Kendall’s Ranking Correlation for Adverbs

19. Kendall’s Ranking Correlation for Verbs

21. VerbNet
• Finding the best and worst verb classes
• Finding patterns in the best and worst verb classes
• Thematic role labels
• Semantic restrictions
• Note*: for each verb class VerbNet labels both of the above
• Plotting patterns
Ex. of verb class Adopt-93
Members assume, adopt, take
Thematic Roles agent, theme
Semantic Restriction animate, organization

22. Top and Bottom Verb Classes
by Average Rank Correlation
• Sorted all verb classes by the average rank correlation of
verbs in that class
• some verb classes did better than others but no outliers
• many of the top/bottom verb classes were the same for
Word2Vec embeddings and DEP1 embeddings
• What attributes cause certain verb classes to do consistently better?
Word2Vec DEP1
Top Class consider-29.9 cooperate-73
Rank of Top Class 0.1642 0.299
Bottom Class light_emission-43.1 exhale-40.1.3
Rank of Bottom Class -0.0151 -0.1473

23. Thematic Role Labels

24. Semantic Restrictions

25. Extrinsic Evaluation:
Sentiment Analysis
• Task is to categorizes sentences according to their positive
or negative sentiment
• “I hate this movie” VS “This movie is ridiculously good”
• Using Kaggle Challenge data from Rotten Tomatoes Movie
Reviews
• System is composed of a convolutional neural network that
is feed the word vectors of the words in the sentence

26. System Overview

27. Results of Sentiment Analysis Task
Accuracy
Emory W2V 0.8624373
DEP1 0.8814250
DEP1H 0.8867580
DEP12 0.8927635
DEP12H 0.8969352
DEP1SIB1 0.8910444
DEP1ALLSIB 0.8896382
DEP1SRLARG 0.8925598
DEP1SRLH 0.8997660

28. Conclusions
• Structure based embeddings are better then topical based
embeddings
• Shown by the evaluations on WordNet, VerbNet and
sentiment analysis task case study
• Different POS capture semantic information from
different sentence structures
• Within verbs embeddings: certain subclasses of verbs do
better than others
• Verbs with certain semantic restricts far outperform other
verb classes

29. Questions
and
Discussion