Presentation discussing the potential of Twitter as a source of insight about customer sentiment towards the brand, but also highlighting the challenges of doing so via automated tools. For more information, or to join the discussion, check my blog www.anacanhoto.com

1. Studying sentiment on social media
Ana Isabel Canhoto - Oxford Brookes University
www.anacanhoto.com
Canhoto 2015 1

2. 2
Emotions impact on:
•Information retrieval
•Information processing
•Information retention
•Decision-making
•Behaviour
•Assessment of
consumption experiences
Why study sentiment?
Canhoto 2015
Image source:
http://images.flatworldknowledge.com/sirgy/
sirgy-fig06_x003.jpg

3. 3
What are we talking about when we talk
about sentiment analysis?
More:
http://www.mxmindia.com/2012/03/tweets-
take-wing-in-airline-social-media-study/
Canhoto 2015

4. 4
Traditional approaches - Experiments
More:
http://www.psych.nyu.edu/amodiolab/Publi
cations_files/Social_Psychological_Methods_
of_Emotion_Elicitation.pdf
Canhoto 2015

5. 5
Traditional approaches – Interviews
Canhoto 2015

6.  Real time
 Unprompted
 No need to recall past
behaviour
 Non-intrusive
 Cost-effective
…
6
The Social Media Promise
Canhoto 2015

7. 7Canhoto 2015
Source: http://cs-wordpress.s3.amazonaws.com/crowdsource-v4/uploads/2013/11/sentiment-
analysis-ui.png

8. Canhoto 2015 8
Pratik Thakar, Head of creative content
for Coca-Cola Asia-Pacific:
“Every office has a listening centre
listening to what people are saying about
our brands, good and bad, 24 hours a
day. We look at what’s trending and how
we can respond [to discussions about
Coca-Cola] and to anything happening in
the world. (…) I believe that social media
is a big focus group. It’s a good way to
identify trends and what people are
talking about”
Source:
http://www.campaignasia.com/Article/402239,Dont+believe+
everything+you+hear+Cokes+Pratik+Thakar.aspx

9. 9
Many turning to third parties for automated
tracking and analysis of SM conversations…
Canhoto 2015
44% of businesses
engaged in sentiment
analysis
Hilpern, K. 'In it to win it?' The Marketer,
July-August 2012, pp.34-37
Estimated cumulative
revenues cc $2bn in
2014
Source:
http://breakthroughanalysis.com/2013/12/30/a
w-re-aw-text-analytics-industry-study_start-ups-
and-aquisition-activities_max-breitsprecher/
How accurate are
these tools?

10. Promotional literature: accuracy rates of
70% - 80% (Davis & O’Flaherty, 2012)
– Not clear how the coefficients were
calculated
– Not possible to independently verify these
claims 10Canhoto 2015

11. 11
Click for updates
Journal of MarketingManagement
Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/rjmm20
‘We (don’t)knowhowyou feel’ –
a comparative study of automated
vs. manual analysisof social media
conversations
Ana Isabel Canhoto
a
& Yuvraj Padmanabhan
b
a
Faculty of Business, Oxford Brookes University, UK
b
Mindgraph, UK
Published online: 18 Jun 2015.
To cite thisarticle: Ana Isabel Canhoto & Yuvraj Padmanabhan (2015) ‘We (don’t) know how you
feel’ – a comparative study of automated vs. manual analysis of social media conversations, Journal
of Marketing Management, 31:9-10, 1141-1157, DOI: 10.1080/0267257X.2015.1047466
To link to this article: http://dx.doi.org/10.1080/0267257X.2015.1047466
PLEASE SCROLL DOWN FOR ARTICLE
Taylor & Francis makes every effort to ensure the accuracy of all the information (the
“Content”) contained in the publications on our platform. However, Taylor & Francis,
our agents, and our licensors make no representations or warranties whatsoever as to
the accuracy, completeness, or suitability for any purpose of the Content. Any opinions
Canhoto 2015
Open access

12. 12
Sources of vulnerability
Canhoto 2015

13. • Accuracy: extent to which different researchers
agree on the classification of a particular data
object (Gwet, 2012)
– System vs human coders
– System A vs System B…
13Canhoto 2015

14. Conversations about coffee
•Food and beverages = most widely discussed
topic on social media (Forsyth, 2011)
•‘Charged with a wide range of cultural
meanings’ (Grinshpun, 2014)
•Subject of many (netnographic) studies - e.g.,
Kozinets, 2002
14Canhoto 2015

15. • Sample of 200 tweets
• Search terms: ‘coffee’ + variants ‘latte’,
‘mocha’, ‘cappuccino’, ‘espresso’ and
‘Americano’, as well as the terms ‘flavour’,
‘aroma’ and ‘caffeine’.
• Multiple users
– Exclude manufacturers and retailers.
15Canhoto 2015

16. Analysis - Stage 1: Polarity of emotion
•Positive vs. negative
– As per Koppel & Schler (2006): comments that did
not express an emotion, were given the code
‘neutral’.
•Manual + 2 automated tools
16Canhoto 2015

17. 17Canhoto 2015

18. Analysis - Stage 2: Type of emotion
• As per Plutchik (2001)
•Manual + 3 Automated tools
18Canhoto 2015

19. 19Canhoto 2015

20. 20Canhoto 2015

21. Messages where all types of coders agreed
Examples:
“Found a euro cent on my walk and have a great
cup of coffee in hand. Monday is already off to a
good start”
“Feeling much more alive this morning now that
I’ve had my coffee. Thank you #Nespresso”.
Clearly positive! 21Canhoto 2015

22. Messages where automated tools agreed (but
different from manual coding)
Example:
“In uni. I think without this cup of coffee I would
hulk out”
Very short segments
22Canhoto 2015

23. The rest
Example:
“The early shift sucks. Oh well at least my latte is
yummy :) “
23
Multiple
objects
Multiple
emotions
Canhoto 2015

24. Example:
“100 copies of Ghosts sold overnight means a
definite Starbucks run this morning. Possibly
coffee out twice this week! Maybe even sushi!!”
Lack of emotionally charged words
24Canhoto 2015

25. Example:
“How the heck am I supposed to be able to sleep
well without coffee in my system? fucking snow”
Subtlety - Negative sentiment due to absence of
product
25Canhoto 2015

26. Example:
“Having coffee with my grandma before work
right now. QT”
Syntax and style, specially abbreviations and
slang
26Canhoto 2015

27. Example:
“This coffee shop needs to change there music
up every once and a while. Or maybe I should go
home”
Target of emotion is not coffee!
27Canhoto 2015

28. 28Canhoto 2015

29. 29Canhoto 2015

30. 30
Compounded by:
• Very short segments of text
• Rich in abbreviations and slang
• Typos or grammatical errors
• Specific culture and netiquette of the media
• Skills of data analystCanhoto 2015

31. As a consequence:
•Inaccurate representation of the overall sentiment
[towards coffee]
– Both sentiment polarity and emotional state
•Segments that should have been excluded from the
analysis were retained in the corpus of data
– Might skew results
•Concerns with the quality of the insights and
subsequent decisions
31Canhoto 2015

32. To improve accuracy [1/2]:
•Take into consideration the social context of the
conversation
– E.g., Tweets before or after the one being coded; wide
patterns (e.g., Mondays); cultural connotations (e.g., Japan
vs. UK)
– But what about sarcasm and highly contextualised uses of
language? (e.g., Sick)
32Canhoto 2015
Pratik Thakar:
“When people say good things, you don’t just take it as
it is. Someone might be asking them to say it; there
might be some design mechanism working. But when
people are unhappy, they go super-loud, and they are
genuine at that time. ”
Source:
http://www.campaignasia.com/Article/402239,Dont+believe+everything+you+hea
r+Cokes+Pratik+Thakar.aspx

33. To improve accuracy [2/2]:
•Develop dictionaries that reflect the specific syntax
and style
•Software solutions that “translate” commonly used
abbreviations and typos
– E.g., BRB – be right back
– Changing norms – e.g., LOL
•Familiarise with software
33Canhoto 2015

34. Studying sentiment on social media
Ana Isabel Canhoto - Oxford Brookes University
www.anacanhoto.com
Canhoto 2015 34