E. Shutova, D. Kiela, and J. Maillard. Black holes and white rabbits: Metaphor identification with visual features. In Proc. of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 160–170, San Diego, California, June 2016. Association for Computational Linguistics Abstract: Sarcasm is a peculiar form of sentiment expression, where the surface sentiment differs from the implied sentiment. The detection of sarcasm in social media platforms has been applied in the past mainly to textual utterances where lex- ical indicators (such as interjections and intensifiers), lin- guistic markers, and contextual information (such as user profiles, or past conversations) were used to detect the sar- castic tone. However, modern social media platforms allow to create multimodal messages where audiovisual content is integrated with the text, making the analysis of a mode in isolation partial. In our work, we first study the relation- ship between the textual and visual aspects in multimodal posts from three major social media platforms, i.e., Insta- gram, Tumblr and Twitter, and we run a crowdsourcing task to quantify the extent to which images are perceived as necessary by human annotators. Moreover, we propose two different computational frameworks to detect sarcasm that integrate the textual and visual modalities. The first approach exploits visual semantics trained on an external dataset, and concatenates the semantics features with state- of-the-art textual features. The second method adapts a vi- sual neural network initialized with parameters trained on ImageNet to multimodal sarcastic posts. Results show the positive effect of combining modalities for the detection of sarcasm across platforms and methods.

1. Black Holes and White Rabbits:
Metaphor Identification with Visual
Features
E. Shutova, D. Kiela, and J. Maillard
NAACL 2016
Presented by:
Sumit Maharjan
Inui Okazaki Lab.
Tohoku University

2. A political machine
Black Holes and White Rabbits: Metaphor Identification with Visual Features
A political machine
The turning wheels of a political regime
Rebuilding the campaign machinery
mending foreign policy
Political system as Mechanisms
Mechanisms(Machine) can
function, break, be
mended
Lakoff and Johnson 1980
• Metaphor arises due to systematic association between
distinct and seemingly unrelated concepts.
• Reasoning about the target domain in terms of the
properties of the source

3. Task
Given a phrase (verb-noun or adjective-noun pair)
identify whether it is a metaphor or not
Black Holes and White Rabbits: Metaphor Identification with Visual Features
Mohammad et al. 2016 Tsvetkov et al. 2016

4. Previous research
• Previous researches used hand-crafted features likes
semantic roles, ontology, information about domains,
concreteness, animateness, named entity, pos tag, wordnet
supersense, concreteness
[Gedigian 2006, Dunn 2013, Tsvetkov 2013, Hovy 2013,
Mohler 2013, Turney 2011, Neuman 2013, Gandy 2013]
• First model to uses word and phrase embedding.
• First model to integrate linguistic and visual information.
Black Holes and White Rabbits: Metaphor Identification with Visual Features

5. Contributions
• Investigate compositional properties of metaphorical
language
– Make word and phrase vectors
– Compare the phrase vector and those of component words
• Investigate role of visual information
– Learn visual representation of words and phrases
– Experiment with different multimodal fusion strategies
– Investigate whether visual features improve performance
Black Holes and White Rabbits: Metaphor Identification with Visual Features

6. Linguistic representations
• Skip-gram (Mikolov et al, 2013)
• 100 dimensional word and phrase embedding
from Wikipedia
– First learn word embedding [first pass]
– Identify verb-object, subject-verb and adjective-
noun phrases on corpus
– Rerun skip-gram to to learn phrase embedding
[second pass with context vector from before]
Black Holes and White Rabbits: Metaphor Identification with Visual Features

7. Visual representations
• Find 10 images for words or phrases using
google image search
• Extract features for each image
from fc7 layer of CNN (Krizhevsky et al., 2012)
pretrained on ImageNet classification task
• Take average
Black Holes and White Rabbits: Metaphor Identification with Visual Features

8. Multimodal fusion
After leaning uni-modal embeddings independently
combine them in the following ways:
• Middle fusion [Combine first and then score]
– Normalize and concatenate the vectors to obtain
multimodal representations
• Late fusion [Score independently and then
combine]
– First compute linguistic and visual metaphoricity
scores independently
– then combine the scores by taking average
Black Holes and White Rabbits: Metaphor Identification with Visual Features

9. Measuring metaphoricity
• Word-based:
WordCos: cos(word1, word2)
• Phrase-based:
PhrasCos1: cos(phrase - word1, word2)
Classification
– Use training set to learn threshold by optimizing
accuracy for the above scoring methods
Black Holes and White Rabbits: Metaphor Identification with Visual Features

10. Experiments: Verb-noun phrases
Dataset: Mohammed et al.(2016)
Training set: 80 verb-noun pairs (for classification threshold)
Test set: 567 verb-noun pairs
Black Holes and White Rabbits: Metaphor Identification with Visual Features
WordCos: cos(word1, word2)
PhrasCos1: cos(phrase - word1,
word2)
MID: normalize and concatenate
the vectors, then score
LATE: Score independently and
then average
MIXLATE: late fusion with linguistic
WordCos and visual PhrasCos

11. Experiments: Adjective-noun phrases
Dataset: Tsvetkov et al 2014
Training set: 80 adjective-noun pairs (for classification threshold)
Test set: 222 adjective-noun pairs
Black Holes and White Rabbits: Metaphor Identification with Visual Features
WordCos: cos(word1, word2)
PhrasCos1: cos(phrase - word1,
word2)
MID: normalize and concatenate
the vectors, then score
LATE: Score independently and
then average
MIXLATE: late fusion with linguistic
WordCos and visual PhrasCos

12. Comparison with other methods
Models F-Score
[Verb-noun]
F-Score
[Adjective-noun]
MIXLATE 0.75 0.79
Tsvetkov et al.
(2014)
0.85 Concreteness
features and hand
coded domain
information
Turney (2011) 0.68 0.79
[Accuracy,
evaluated only on
10 adjectives]
Hand annotated
abstractness scores
for words
Black Holes and White Rabbits: Metaphor Identification with Visual Features
Unlike supervised methods do not need large training data set to learn
the threshold

13. Conclusion
• Visual features help in metaphor identification
• Visual features are useful for modelling
compositionality
• Late fusion combining different scores perform
best.
• High performance with little annotated training
data.
• Visual features perform better for adjectives than
verbs: use video
Black Holes and White Rabbits: Metaphor Identification with Visual Features