Mat Morrison                        @mediaczarTuesday, 5 July 2011
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Elephant in the roomTuesday, 5 July 2011
or Emperor’s ClothesTuesday, 5 July 2011
Tuesday, 5 July 2011
VolumeTuesday, 5 July 2011
qualified byVolume                           Relevance                           AUTHORITY                           Senti...
quick Question:             How Accurate                       is Sentiment analysis?Tuesday, 5 July 2011
The                       stinksTuesday, 5 July 2011
supervised                        Learning                         Machine      Robot                         HumanTuesday...
Robots first...Tuesday, 5 July 2011
I’m unqualified to              make judgements   about the science...Tuesday, 5 July 2011
I’m unqualified to              make judgements   about the science...                       ...and So are you.Tuesday, 5 ...
Because the Science      Is hidden in a boxTuesday, 5 July 2011
Because the Science      Is hidden in a box                       But scientific Results are                            Re...
What’s the buzz around                         “Green Lantern”?Tuesday, 5 July 2011
Positive    Negative     Sysomos (Blogs)               40             17           TweetSentiment          29             ...
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Turk THis!               RT @BangsandaBun: According to               some, I wasnt deserving/grateful/               cool...
Turk THis!               RT @BangsandaBun: According to               some, I wasnt deserving/grateful/               cool...
Turk THis!               I know the bands for Glastonbury               were great this year, but I still               do...
Turk THis!               I know the bands for Glastonbury               were great this year, but I still               do...
SENTIMENT scoring using       Amazon Mechanical Turk       disagreed with 16/40 (40%)       robot judgments.       Only 13...
But Are Humans    Reliable?               ???     Emotional State/Mood                       Priming                      ...
It Gets Worse...             The Same Human Scorer,             when presented with the             same statement on     ...
Tuesday, 5 July 2011
The                       stinksTuesday, 5 July 2011
Academic studies                       generally focus on                 Movies &            the STOCK Market            ...
Social Media    data are full of    irrelevant...Tuesday, 5 July 2011
Social Media    data are full of    irrelevant...Tuesday, 5 July 2011
Tuesday, 5 July 2011
Volume                       Relevance                       Authority                       SentimentTuesday, 5 July 2011
All four are needed to make               sense of dataVolume                                      Relevance              ...
The                       stinksTuesday, 5 July 2011
Why might marketers     want to perform     Sentiment analysis?                   To guide customer service               ...
Is sentiment         actionable?                         Product Query            Is it a...                         Produ...
THE KPI Problem                                                                        !"#"$%&#                       ##$ ...
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Tuesday, 5 July 2011
Upcoming SlideShare
Loading in...5
×

Negative Sentiment (or "Sentiment Analysis is Sh*te")

6,151

Published on

I used to believe that sentiment analysis was one of the most important tools in a smart marketer's box.

The rise of social media offered (I thought) a huge, free, and above all, reliable source of both qualitative and quantitative data about how real people really feel about brands, and how our marketing contributed to those feelings.

Over the years, though, I've become increasingly disappointed in both the reality and the promise; to the point at which I'm actively recommending that our clients avoid using social media sentiment in their models or as a performance metric.

It's time for us all to stop kidding ourselves, 'fess up, and end the conspiracy. In this short talk, I hope to explain why sentiment analysis doesn't function as a useful social media metric, now or EVER.

Published in: Business, Economy & Finance
2 Comments
10 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,151
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
81
Comments
2
Likes
10
Embeds 0
No embeds

No notes for slide

Transcript of "Negative Sentiment (or "Sentiment Analysis is Sh*te")"

  1. 1. Mat Morrison @mediaczarTuesday, 5 July 2011
  2. 2. Tuesday, 5 July 2011
  3. 3. Tuesday, 5 July 2011
  4. 4. Elephant in the roomTuesday, 5 July 2011
  5. 5. or Emperor’s ClothesTuesday, 5 July 2011
  6. 6. Tuesday, 5 July 2011
  7. 7. VolumeTuesday, 5 July 2011
  8. 8. qualified byVolume Relevance AUTHORITY SentimentTuesday, 5 July 2011
  9. 9. quick Question: How Accurate is Sentiment analysis?Tuesday, 5 July 2011
  10. 10. The stinksTuesday, 5 July 2011
  11. 11. supervised Learning Machine Robot HumanTuesday, 5 July 2011
  12. 12. Robots first...Tuesday, 5 July 2011
  13. 13. I’m unqualified to make judgements about the science...Tuesday, 5 July 2011
  14. 14. I’m unqualified to make judgements about the science... ...and So are you.Tuesday, 5 July 2011
  15. 15. Because the Science Is hidden in a boxTuesday, 5 July 2011
  16. 16. Because the Science Is hidden in a box But scientific Results are Reproducable!!!Tuesday, 5 July 2011
  17. 17. What’s the buzz around “Green Lantern”?Tuesday, 5 July 2011
  18. 18. Positive Negative Sysomos (Blogs) 40 17 TweetSentiment 29 11 WDTIT? 56.8 42 SocialMention 18 5 SM2 20 16 Sysomos (Tweets) 6.4 7.6 Rotten Tomatoes 27 73 A quick poll of a few sentiment engines’ analysis of the buzz surrounding the film “Green Lantern” demonstrates no correspondence.Tuesday, 5 July 2011
  19. 19. Tuesday, 5 July 2011
  20. 20. Tuesday, 5 July 2011
  21. 21. Turk THis! RT @BangsandaBun: According to some, I wasnt deserving/grateful/ cool enough to be there, but I blogged about my Glastonbury experience http://t.co/eEInrnuTuesday, 5 July 2011
  22. 22. Turk THis! RT @BangsandaBun: According to some, I wasnt deserving/grateful/ cool enough to be there, but I blogged about my Glastonbury experience http://t.co/eEInrnu Robot says NegativeTuesday, 5 July 2011
  23. 23. Turk THis! I know the bands for Glastonbury were great this year, but I still dont think you could have paid me to be part of that mental crowd! You?Tuesday, 5 July 2011
  24. 24. Turk THis! I know the bands for Glastonbury were great this year, but I still dont think you could have paid me to be part of that mental crowd! You? Robot says positiveTuesday, 5 July 2011
  25. 25. SENTIMENT scoring using Amazon Mechanical Turk disagreed with 16/40 (40%) robot judgments. Only 13/40 (32.5%) of robotic judgments were passed by all three human Judges. Raw Data available at http://mczar.me/mkue1pTuesday, 5 July 2011
  26. 26. But Are Humans Reliable? ??? Emotional State/Mood Priming Cultural bias Cognitive Bias Famously not.Tuesday, 5 July 2011
  27. 27. It Gets Worse... The Same Human Scorer, when presented with the same statement on multiple occasions, may score it differently each time. cf. A Guerilla Experiment in Sentiment Analysis http://mczar.me/lrp7YiTuesday, 5 July 2011
  28. 28. Tuesday, 5 July 2011
  29. 29. The stinksTuesday, 5 July 2011
  30. 30. Academic studies generally focus on Movies & the STOCK Market Why is this?Tuesday, 5 July 2011
  31. 31. Social Media data are full of irrelevant...Tuesday, 5 July 2011
  32. 32. Social Media data are full of irrelevant...Tuesday, 5 July 2011
  33. 33. Tuesday, 5 July 2011
  34. 34. Volume Relevance Authority SentimentTuesday, 5 July 2011
  35. 35. All four are needed to make sense of dataVolume Relevance Authority Sentiment But I’m only talking sentimentTuesday, 5 July 2011
  36. 36. The stinksTuesday, 5 July 2011
  37. 37. Why might marketers want to perform Sentiment analysis? To guide customer service interactions As a Performance indicator To qualify social media buzz To identify fansTuesday, 5 July 2011
  38. 38. Is sentiment actionable? Product Query Is it a... Product Complaint ??? Product endorsement Product review Product COmparison ...I need to know moreTuesday, 5 July 2011
  39. 39. THE KPI Problem !"#"$%&# ##$ &"#$ &##$ %"#$ =)*;>$?)@2<$ A2.-4.@)3>$?)@2<$ %##$ B;3C>,;<<$?)@2<$ "#$ #$ ()*+,-+.$ /+0+,-+.$ 12342.5$ 6+-.42.5$ 72.08$ 9:.;<$ 725$ !"#$ Massive (and unrepresentative) spike in Net Positive Sentiment caused by specific campaign activity Should we be allowed to mark our own homework?Tuesday, 5 July 2011
  40. 40. Tuesday, 5 July 2011
  41. 41. Tuesday, 5 July 2011
  42. 42. Tuesday, 5 July 2011
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×