The original paper is written by Saima Aman and Stan Szpakowicz. I presented it to my batch. Sentiment Analysis has typically focused on recognizing positive and negative words.The authors goal was to investigate the expression of emotion in the language through a corpus annotation study.They achieve 73.89% accuracy according to the preliminary results of their emotion classification experiments.1

1. SENTENCE LEVEL
SENTIMENT ANALYSIS
IDENTIFYING EXPRESSIONS OF EMOTIONS IN TEXT
Paper published by Saima Aman and Stan
Szpakowicz
Presented by Vipul Munot
Indiana University Bloomington
MS Data Science

2. AGENDA
 Overview
 Ekman Basic Emotions
 Datasets
 Emotion Annotations
 Measuring Annotations
 Automatic Emotion Classification
 Experiments and results
 Future Work
 Related work
10/5/2015Presented by Vipul Munot
2

3. OVERVIEW
 Sentiment Analysis has typically focused on recognizing positive
and negative words.
 The authors goal was to investigate the expression of emotion in
language through a corpus annotation study.
 They achieve 73.89% accuracy according to the preliminary
results of their emotion classification experiments.
10/5/2015Presented by Vipul Munot
3

4. EKMAN BASIC EMOTIONS
 Paul Ekman classified emotions in six different basic categories :
1. Happiness
2. Sadness
3. Surprise
4. Fear
5. Disgust
6. Anger
10/5/2015Presented by Vipul Munot
4

5. DATASETS (1/2)
 This paper developed their own datasets using blog posts
collected directly from the web.
 Datasets were classified into Ekman’s six emotions.
 They took words commonly used in the context of a particular
emotions, referred as seed words.
 “happiness” for words – happy, enjoy, pleased.
 “fear” for words – afraid, scared, panic and so on.
10/5/2015Presented by Vipul Munot
5

6. DATASETS (2/2)
Dataset # posts # sentences Collected using
seed words for
Ec-hp 34 848 Happiness
Ec-sd 30 884 Sadness
Ec-ag 26 883 Anger
Ec-dg 21 882 Disgust
Ec-sp 31 847 Surprise
Ec-fr 31 861 Fear
Total 173 5205
10/5/2015Presented by Vipul Munot
6

7. EMOTION ANNOTATIONS (1/4)
 Objectives –
 Emotion labelling.
 To identify spans of text (individual words or strings of consecutive
words) that convey emotional content in a sentence.
10/5/2015Presented by Vipul Munot
7

8. EMOTION ANNOTATIONS (2/4)
 Emotion labeling had to be done manually.
 To achieve this they used four judges to annotate the corpus.
 Each sentence was subject to two judgements.
 No training was provided to judges.
 They were given a set of examples to annotate data.
 Time spent to annotate data – 3 months.
 The authors added two more emotions to original Ekman’s six emotions – Mixed
emotions and no emotion.
 Reason – sometimes you cannot differentiate the emotions in the sentences.
 For examples –
 It's like everything everywhere is going crazy, so we don't go out any more. - No Emotion.
 I felt bored and wanted to leave at intermission, but my wife was really enjoying it, so we stayed.
– Mixed emotion .
10/5/2015Presented by Vipul Munot
8

9. EMOTION ANNOTATIONS(3/4)
 Now, their goal was to label the emotions with its intensity. For
example,
 I have to look at life in her perspective, and it would break anyone’s
heart. (sadness, high)
 I hate it when certain people always seem to be better at me in
everything they do. (disgust, low)
 But the rest of it came across as a really angry, drunken rant. (anger,
high)
10/5/2015Presented by Vipul Munot
9

10. EMOTION ANNOTATIONS (4/4)
 Spans of text
 Emotion is often conveyed by longer units of text or by phrases, for
example
 I can't believe this went on for so long, and we were blissfully unaware of
it.
10/5/2015Presented by Vipul Munot
10

11. MEASURING ANNOTATIONS(1/7)
 Objectives –
 Achieving consensus between judges for particular emotion and it’s
intensity for a particular sentence.
10/5/2015Presented by Vipul Munot
11

12. MEASURING ANNOTATIONS(2/7)
 The interpretation of sentiment information in text is highly
subjective, which leads to disparity in the annotations by different
judges.
 Difference in skills and focus of the judges, and ambiguity in the
annotation guidelines and in the annotation task itself also
contribute to disagreement between the judges
10/5/2015Presented by Vipul Munot
12

13. MEASURING ANNOTATIONS(3/7)
 Cohen's kappa(https://en.wikipedia.org/wiki/Cohen%27s_kappa)
 measures the agreements between two raters who each classify N items
into C mutually exclusive categories.
A   B A   C A   D Average
Kappa 0.73 0.84 0.71 0.76
Pair-wise agreement in emotion/non-emotion labeling
10/5/2015Presented by Vipul Munot
13

14. MEASURING ANNOTATIONS(4/7)
Category A   B A   C A   D Average
Happiness 0.76 0.84 0.71 0.77
Sadness 0.68 0.79 0.56 0.68
Anger 0.62 0.76 0.59 0.66
Disgust 0.64 0.62 0.74 0.67
Surprise 0.61 0.72 0.48 0.6
Fear 0.78 0.8 0.78 0.79
Mixed
Emotions
0.24 0.61 0.44 0.43
Pair-wise agreement in emotion categories
10/5/2015Presented by Vipul Munot
14

15. MEASURING ANNOTATIONS(5/7)
Intensity A   B A   C A   D Average
High 0.69 0.82 0.65 0.72
Medium 0.39 0.61 0.38 0.46
Low 0.31 0.5 0.29 0.37
Pair-wise agreement in emotion intensities
 The judges agreed more when the
emotion intensity was high;
agreement declined with decrease
in the intensity of emotion.
 It is a major factor in disagreement
that where one judge perceives a
low-intensity, another judge may
find no emotion.
10/5/2015Presented by Vipul Munot
15

16. MEASURING ANNOTATIONS(6/7)
 For this paper authors have used measure of agreement on set-
valued items (MASI)
 MASI is a distance between sets whose value is 1 for identical sets, and 0
fordisjoint sets.
 MASI = J * M, where the Jaccard metric is
J = |A∩B| / |A∪B|
 If one set is monotonic with respect to another, one set's elements
always match those of the other set.
 The presence of monotonicity factor in MASI therefore ensures that
the latter cases are penalized more heavily than the former.
10/5/2015Presented by Vipul Munot
16

17. MEASURING ANNOTATIONS(7/7)
 They used another method called IOB encoding(variant of the
method) of measuring agreement between emotion indicators.
 Use IO encoding, in which each word in the sentence is labeled
as being either In or Outside an emotion indicator text span
 Sorry/I for/O the/O ranting/I post/O, but/O I/O am/O just/O really/I
annoyed/
Metric A   B A   C A   D Average
MASI 0.59 0.66 0.59 0.61
Kappa 0.61 0.73 0.65 0.66
Pair-wise agreement in emotion indicators
10/5/2015Presented by Vipul Munot
17

18. AUTOMATIC EMOTION
CLASSIFICATION(1/3)
 Focus is on recognizing emotional sentences in text, regardless of
their emotion category.
 Extracted all those sentences from the corpus for which there
was consensus among the judges on their emotion category.
 Assigned all emotion category sentences to the class “EM”, while
all no emotion sentences were assigned to the class “NE”.
 Resulting dataset
 1466 sentences belonged EM class.
 and 2800 sentences belonged to the NE class.
10/5/2015Presented by Vipul Munot
18

19. AUTOMATIC EMOTION
CLASSIFICATION(2/3)
 In order to differentiate between EM and NE authors used
publicly available lexical resources -
 General Inquirer - consists of words drawn from several dictionaries
and grouped into various semantic categories.
 WordNet-Affect - assigns a variety of affect labels to a subset of
synsets in WordNet.
 Even symbols such as emoticons and punctuations are used for
expressing emotions.
10/5/2015Presented by Vipul Munot
19

20. AUTOMATIC EMOTION
CLASSIFICATION(3/3)
GI Features WN-Affect Features Other Features
Emotion Words
Positive Words
Negative Words
Interjection Words
Pleasure Words
Pain Words
Happiness Words
Sadness Words
Anger Words
Disgust Words
Surprise Words
Fear Words
Emoticons
Exclamation(“!”)
And question (“?”)
marks
Features used in emotion classification
10/5/2015Presented by Vipul Munot
20

21. FINDINGS
 No emotion category was most frequent.
 No intensity label was assigned to the no emotion sentences.
 Individual words are not sufficient for labelling the sentences.
 A sentence was found to exhibit more than one emotion.
 Emotion conveyed in some sentences could not be attributed to
any basic category.
10/5/2015Presented by Vipul Munot
21

22. EXPERIMENTS
 Navie Bayes and Suppor Vector Machines (SVM) were used for
binary classification experiments.
10/5/2015Presented by Vipul Munot
22
Features Naïve Bayes SVM
GI 71.45% 71.33%
WN-Affect 70.16% 70.58%
GI+WN-Affect 71.7% 73.89%
ALL 72.08% 73.89%
Emotion Classification accuracy

23. RESULTS
 The best results in were achieved when all features were
combined.
 Non-lexical features increased the accuracy of Naïve Bayes
classifier, but didn’t affect the SVM results.
 Automatic Emotion Classification – shows how external
knowledge resources can be leveraged in identifying emotion-
related words in text.
10/5/2015Presented by Vipul Munot
23

24. FUTURE WORK
 Acronym such as “lol” need to be incorporated in future work.
 To address typographical errors and orthographic words (for eg.
“soo sweet”).
 To incorporate contextual and semantic analysis as well.
10/5/2015Presented by Vipul Munot
24

25. RELATED WORK
 Appraisal framework (http://grammatics.com/appraisal/)
 Ekman Basic Emotions (https://www.paulekman.com/wp-content/uploads/2013/07/An-Argument-For-Basic-Emotions.pdf)
 Mihalcea, R., Liu, H.: A corpus-based approach to finding happiness. In: The
AAAI Spring Symposium on Computational Approaches to Weblogs, Stanford,
CA (2006)
10/5/2015Presented by Vipul Munot
25

26. REFERENCES
 http://www-scf.usc.edu/~saman/pubs/2007-TSD-paper.pdf
10/5/2015Presented by Vipul Munot
26

27. QUESTIONS?
10/5/2015Presented by Vipul Munot
27

28. THANK YOU.
10/5/2015Presented by Vipul Munot
28