Negative Sentiment (or "Sentiment Analysis is Sh*te")

6,314
-1

Published on

I used to believe that sentiment analysis was one of the most important tools in a smart marketer's box.

The rise of social media offered (I thought) a huge, free, and above all, reliable source of both qualitative and quantitative data about how real people really feel about brands, and how our marketing contributed to those feelings.

Over the years, though, I've become increasingly disappointed in both the reality and the promise; to the point at which I'm actively recommending that our clients avoid using social media sentiment in their models or as a performance metric.

It's time for us all to stop kidding ourselves, 'fess up, and end the conspiracy. In this short talk, I hope to explain why sentiment analysis doesn't function as a useful social media metric, now or EVER.

Published in: Business, Economy & Finance
2 Comments
10 Likes
Statistics
Notes
No Downloads
Views
Total Views
6,314
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
82
Comments
2
Likes
10
Embeds 0
No embeds

No notes for slide

Negative Sentiment (or "Sentiment Analysis is Sh*te")

  1. 1. Mat Morrison @mediaczarTuesday, 5 July 2011
  2. 2. Tuesday, 5 July 2011
  3. 3. Tuesday, 5 July 2011
  4. 4. Elephant in the roomTuesday, 5 July 2011
  5. 5. or Emperor’s ClothesTuesday, 5 July 2011
  6. 6. Tuesday, 5 July 2011
  7. 7. VolumeTuesday, 5 July 2011
  8. 8. qualified byVolume Relevance AUTHORITY SentimentTuesday, 5 July 2011
  9. 9. quick Question: How Accurate is Sentiment analysis?Tuesday, 5 July 2011
  10. 10. The stinksTuesday, 5 July 2011
  11. 11. supervised Learning Machine Robot HumanTuesday, 5 July 2011
  12. 12. Robots first...Tuesday, 5 July 2011
  13. 13. I’m unqualified to make judgements about the science...Tuesday, 5 July 2011
  14. 14. I’m unqualified to make judgements about the science... ...and So are you.Tuesday, 5 July 2011
  15. 15. Because the Science Is hidden in a boxTuesday, 5 July 2011
  16. 16. Because the Science Is hidden in a box But scientific Results are Reproducable!!!Tuesday, 5 July 2011
  17. 17. What’s the buzz around “Green Lantern”?Tuesday, 5 July 2011
  18. 18. Positive Negative Sysomos (Blogs) 40 17 TweetSentiment 29 11 WDTIT? 56.8 42 SocialMention 18 5 SM2 20 16 Sysomos (Tweets) 6.4 7.6 Rotten Tomatoes 27 73 A quick poll of a few sentiment engines’ analysis of the buzz surrounding the film “Green Lantern” demonstrates no correspondence.Tuesday, 5 July 2011
  19. 19. Tuesday, 5 July 2011
  20. 20. Tuesday, 5 July 2011
  21. 21. Turk THis! RT @BangsandaBun: According to some, I wasnt deserving/grateful/ cool enough to be there, but I blogged about my Glastonbury experience http://t.co/eEInrnuTuesday, 5 July 2011
  22. 22. Turk THis! RT @BangsandaBun: According to some, I wasnt deserving/grateful/ cool enough to be there, but I blogged about my Glastonbury experience http://t.co/eEInrnu Robot says NegativeTuesday, 5 July 2011
  23. 23. Turk THis! I know the bands for Glastonbury were great this year, but I still dont think you could have paid me to be part of that mental crowd! You?Tuesday, 5 July 2011
  24. 24. Turk THis! I know the bands for Glastonbury were great this year, but I still dont think you could have paid me to be part of that mental crowd! You? Robot says positiveTuesday, 5 July 2011
  25. 25. SENTIMENT scoring using Amazon Mechanical Turk disagreed with 16/40 (40%) robot judgments. Only 13/40 (32.5%) of robotic judgments were passed by all three human Judges. Raw Data available at http://mczar.me/mkue1pTuesday, 5 July 2011
  26. 26. But Are Humans Reliable? ??? Emotional State/Mood Priming Cultural bias Cognitive Bias Famously not.Tuesday, 5 July 2011
  27. 27. It Gets Worse... The Same Human Scorer, when presented with the same statement on multiple occasions, may score it differently each time. cf. A Guerilla Experiment in Sentiment Analysis http://mczar.me/lrp7YiTuesday, 5 July 2011
  28. 28. Tuesday, 5 July 2011
  29. 29. The stinksTuesday, 5 July 2011
  30. 30. Academic studies generally focus on Movies & the STOCK Market Why is this?Tuesday, 5 July 2011
  31. 31. Social Media data are full of irrelevant...Tuesday, 5 July 2011
  32. 32. Social Media data are full of irrelevant...Tuesday, 5 July 2011
  33. 33. Tuesday, 5 July 2011
  34. 34. Volume Relevance Authority SentimentTuesday, 5 July 2011
  35. 35. All four are needed to make sense of dataVolume Relevance Authority Sentiment But I’m only talking sentimentTuesday, 5 July 2011
  36. 36. The stinksTuesday, 5 July 2011
  37. 37. Why might marketers want to perform Sentiment analysis? To guide customer service interactions As a Performance indicator To qualify social media buzz To identify fansTuesday, 5 July 2011
  38. 38. Is sentiment actionable? Product Query Is it a... Product Complaint ??? Product endorsement Product review Product COmparison ...I need to know moreTuesday, 5 July 2011
  39. 39. THE KPI Problem !"#"$%&# ##$ &"#$ &##$ %"#$ =)*;>$?)@2<$ A2.-4.@)3>$?)@2<$ %##$ B;3C>,;<<$?)@2<$ "#$ #$ ()*+,-+.$ /+0+,-+.$ 12342.5$ 6+-.42.5$ 72.08$ 9:.;<$ 725$ !"#$ Massive (and unrepresentative) spike in Net Positive Sentiment caused by specific campaign activity Should we be allowed to mark our own homework?Tuesday, 5 July 2011
  40. 40. Tuesday, 5 July 2011
  41. 41. Tuesday, 5 July 2011
  42. 42. Tuesday, 5 July 2011

×