This document discusses the current limitations and challenges of sentiment analysis. It summarizes that while sentiment analysis systems can detect the polarity of sentiment at a basic level, they struggle with more complex linguistic expressions that change sentiment. Specifically: - Systems are largely binary (positive vs negative) and cannot capture varying degrees of sentiment. - Common linguistic constructions like negation, conjunct reversal, and context dependence still break many systems. - Averaging sentiment scores obscures meaningful differences in sentiment within text. - Data used to train systems contains inherent cultural biases that are learned and reproduced.

1. What lies beneath sentiment analysis
Marie-Catherine de Marneffe
The Ohio State University
Linguistics

2. ?

8. Sentiment analysis is implicit meaning
and context-dependent
unpredictable plot

9. Sentiment analysis is implicit meaning
and context-dependent
unpredictable plot
unpredictable boss

10. How does it work?
Machine learning
training
Annotated corpus Classifier

11. How does it work?
Machine learning
training
Annotated corpus
OR
Classification result
New text
Classifier

12. Stanford Sentiment Treebank
[Socher et al. 2013]
9
9
13
13
11
.
This
13
8½
13 11 14
1318
new idea
is n’t a
16

13. Sentiment score: average of 3 scores
00.
1 25
Overall sentiment score: 0.4444
Raw scores: 13,13, 9

14. 5-way labeling
(0.8, 1.0]
(0.6, 0.8]
(0.4, 0.6]
(0.2, 0.4]
[0, 0.2]

15. Binary labeling
(0.8, 1.0]
(0.6, 0.8]
(0.4, 0.6]
(0.2, 0.4]
[0, 0.2]

16. RNTN
All phrases Sentences
binary 87.6 85.4
5-class 80.7 45.7
Results on both categorizations

17. Stanford Sentiment System
http://nlp.stanford.edu:8080/sentiment/rntnDemo.html

19. ✓
✓

20. Far from being solved
Minimally changing sentences breaks current
sentiment analysis systems
Systems can only handle the ends of the
sentiment spectrum (+/-)
Inherent bias in the data

21. https://bibinlp.umiacs.umd.edu/sharedtask.html
EMNLP 2017

22. Systems fail to capture the difference
If not for its show-offy back-and- forthing of time,
the movie would be a banal, pointlessly
depressing exercise.
Despite its show-offy back-and- forthing of time,
the movie is a banal, pointlessly depressing
exercise.
+
-

23. Using mostly unknown and first-time actors, Loach
spins a passable coming-of-age tale, which
should please his fans and provides a diversion
for the rest of us.
Using mostly unknown and first-time actors, Loach
spins a passable coming-of-age tale, which ought
to have pleased his fans and provided a diversion
for the rest of us.
+
-
Systems fail to capture the difference

24. We are far from robust sentiment
analysis
[Mahler et al. 2017]

25. We are far from robust sentiment
analysis

26. “Standard” negation
This is one of the worst movies of the year.
This is not one of the worst movies of the year.
Director Jordan fails to deliver a film
worthy of such mythical figure.
Director Jordan did not fail to deliver a film
worthy of such mythical figure.
-
58% break, n=6
-
+
0%
-
83%

27. Post-fixed negation
Exciting heist film for teens and their families.
Exciting heist film for teens and their families – not!
A fun bit of action comedy with a bang up cast.
A fun bit of action comedy with a bang up cast –
not!
+
-
100% break, n=2
-
+

28. Conjunct reversal
Munich ricochets all over the place, but it hits its
target dead-on.
Munich hits its target dead-on, but ricochets all over
the place.
71% break, n=7
-
100%
+

29. Conjunct reversal
Munich ricochets all over the place, but it hits its
target dead-on.
Munich hits its target dead-on, but ricochets all over
the place.
The sentiments are right on the money, but the
execution never quite filled me with holiday cheer.
The execution never quite filled me with holiday
cheer, but the sentiments are right on the money.
-
71% break, n=7
-
+
100%
67%
+

30. Do systems understand the
linguistic structure?
Systems seem to understand the individual
words to detect sentiment
But they cannot yet handle more complex
linguistic expressions

31. Currently sentiment analysis is binary

32. POSITIVE
NEGATIVE
Currently sentiment analysis is binary

33. Phrases with sentiment score
0.55556
a film as Byatt fans could hope for
A sly dissection of the inanities of the
contemporary music business and a
rather sad story of the difficulties
13,15,15
13,15,15

34. a film as Byatt fans could hope for
A sly dissection of the inanities of the
contemporary music business and a
rather sad story of the difficulties
a sad, superior human
a joke
Angel presents events partly from the
perspective of Aurelie and Christelle and
infuses the film with the sensibility of a
particularly nightmarish fairytale
13,15,15
13,15,15
9,13, 21
8,15, 20
9,17 17
Phrases with sentiment score
0.55556

35. Averages obscure differences
In most datasets sentiment scores are averaged
But this conflates the data in a way that
obscures differences
Another common measure: majority voting
But data with variance is informative too!

36. Acknowledging variability
Precisio
n
Recall F1
Complicate
d
19.9 13.8 16.3
Positive 59.9 60.5 60.2
Neutral 76.6 80.6 78.6
Negative 51.0 48.7 49.8
Total 51.8 50.9 51.2
[Kenyon-Dean et al. 2018]

37. Inherent bias
[Blodgett & O’Connor 2017]

38. Inherent bias
[Blodgett & O’Connor 2017]

39. Data representation: word
embeddings
Based on distributional semantics
With long, wispy eyelashes and a vibrant blue face,
Bernie the southern cassowary has a look that
rivals even the fanciest of peacocks.
Cassowaries stand between 1.5-2 meters in height
with both sexes similar in appearance.
How Dangerous Are Cassowaries, Really? -
Scientific American Blog ...
Scientists currently recognize three living species of
cassowary—all of which are restricted to New
Guinea, northeastern Australia, and nearby islands.

40. Image from Wikipedia

41. Data representation
occurs near
Australia
occurs near
dangerous
cassowary
never often
often

42. What other things are dangerous?
What other things live in Australia?
Most Dangerous Cancers in Men and Women Infographic
Why are cancer cells dangerous?
The koala is an arboreal herbivorous marsupial native
to Australia.
The AKF estimates that there are likely to be less than
80,000 Koalas remaining in Australia today and it could
be as low as 43,000.

43. Data representation
occurs near
Australia
occurs near
dangerous
koala cassowary
cancer
often
never often

44. Data representation
occurs near
Australia
occurs near
dangerous
koala cassowary
cancer
often
never often

45. There are lots of dimensions
[Kim & de Marneffe 2013]

46. Word2Vec/Elmo/Bert
https://rare-technologies.com/word2vec-tutorial/

47. Word2Vec learns the bias in the
data
https://rare-technologies.com/word2vec-tutorial/

48. Word2Vec learns the bias in the
data

49. Wrapping up
Know the kind of data on which systems you use
are trained.

50. Thank you!