Sentiment is Shite (2014)
•
13 likes•4,283 views
Slide deck from a talk given at the BreakingNews.ie Measurement Conference on 10th September 2014. It's an updated version of a talk I gave (only once) in 2011. My understanding (and the technology) have developed since then.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (13)
Similar to Sentiment is Shite (2014)
Similar to Sentiment is Shite (2014) (20)
More from Mat Morrison
More from Mat Morrison (20)
Recently uploaded
Recently uploaded (20)
Sentiment is Shite (2014)
- 1. NEGATIVE SENTIMENT another grumbly presentation from @mediaczar
- 2. TWITTER VOLUME 15 10 5 0 1 Jun 2014 1 Jul 2014 Thousands “facebook” AND (“newsfeed” OR “news feed”) June 1 – July 31 2014 Data: Netbase
- 3. WHAT HAPPENED HERE? aND HOW DOES THAT MAKE YOU FEEL? Topic Analysis Emotions Data: Brandwatch, Netbase
- 4. DELOITTE: BALANCE OF SENTIMENT AFFECTS SALES The balance of sentiment in Tweets is a more powerful driver of sales than reach (or volume) alone, with positive Tweets having generally a higher impact than negative Tweets. Therefore, to gain the most out of the online word-of- mouth embodied by Tweets, companies would be best served by addressing the balance of sentiment about their games through increasing the number of positive Tweets. 6.10% 3.30% 1.60% 30% more positive Tweets 30% fewer negative Tweets 30% more non-Twitter advertising Deloitte, “Tweets for Sales, Gaming” (2013)
- 5. “SENTIMENT” HAS BECOME A MARKETING OBJECTIVE We want to increase the positive buzz around Your brand
- 6. “SENTIMENT” HAS BECOME A MARKETING OBJECTIVE Real case study. Name obscured to protect the innocent.
- 7. THIS IS HONEST ABE Public sentiment is everything. With public sentiment, nothing can fail. Without it, nothing can succeed.
- 8. I’M GOING TO BE EVEN MORE HONEST SENTIMENT ANALYSIS IS SHITE! DTEOLNL'T U HSO WLDH ABTA CYKO.U. .REALLY THINK!
- 9. THE PROMISE OF SOCIAL INTELLIGENCE AN ALMOST INFINITE SOURCE OF QUAL QUANT DATA! FINALLY THEY WILL REVEAL WHAT THEY REALLY THINK!
- 10. RELEVANCE VOLUME AUTHORITY TOPICS SENTIMENT
- 11. WEIGHTED SENTIMENT, MRS BROWN’S BOYS, JULY 2014 14.8% 12.2% 11.4% 7.3% -4.1% -5.2% -6.3% -8.5% POSITIVE NEGATIVE 4 DIFFERENT TOOLS GIVE 4 DIFFERENT MEASURES OF SAME DATA
- 12. METHODOLOGICAL PROBLEMS LEXICAL ANALYSIS CLASSIFICATION MANUAL INPUT
- 13. LEXICAL ANALYSIS visit: http://www.wjh.harvard.edu/~inquirer/spreadsheet_guide.htm e.g. Harvard General Enquirer: 11.8K categorised, tagged words
- 14. QUICK DIRTY DIY SENTIMENT ANALYSIS TOOL
- 16. CAN’T HANDLE IDIOMS, SYNONYMS (OR IRONY) WELL, THAT'S JUST GREAT
- 17. FACEBOOK USED LEXICAL APPROACH “Posts were determined to be positive or negative if they contained at least one positive or negative word” Visit: http://www.pnas.org/content/111/24/8788.full LIWC: http://www.liwc.net/ 15 10 5 0 Thousands
- 18. CLASSIFIERS SUPERVISED LEARNING… TRAIN MODEL TAGGED TEST MODEL DATA TRAINING DATA TEST DATA POSITIVE NEUTRAL NEGATIVE
- 19. THIS PRESENTATION IS GOOD visit: text-processing.com/demo/sentiment CLASSIFIER SAYS “POSITIVE”
- 20. THIS PRESENTATION IS BAD CLASSIFIER SAYS “NEGATIVE” visit: text-processing.com/demo/sentiment
- 21. THIS PRESENTATION IS NOT GOOD CLASSIFIER SAYS “NEGATIVE” visit: text-processing.com/demo/sentiment
- 22. THESE ARE BOTH POSITIVE analysed analysed
- 23. THIS IS NEUTRAL analysed
- 24. WHAT ABOUT THIS?
- 25. DOES IT UNDERSTAND WHAT IT’S READING? 2. 1. Take the original text Randomise the word order 3. Re-test recipe: http://stackoverflow.com/questions/17825945/generating-a-list-of-random-words-in-excel-but-no-duplicates
- 26. WORD ORDER MAKES NO DIFFERENCE
- 27. WORD ORDER MAKES NO DIFFERENCE
- 28. WORD ORDER MAKES NO DIFFERENCE
- 29. DOMAIN SPECIFIC Models trained on one set of data may not work well on other sets
- 30. WHAT ABOUT HUMAN MARKERS?
- 31. FAIRLY EASY TO ASSESS ENTERTAINMENT CATEGORY
- 32. EXPERIMENT: IS GUINNESS GOOD FOR YOU? Selected 50 positive and 50 negative tweets as scored by classifier. Passed these tweets to human markers. Each tweet scored 3 times (5 point scale) Average score compared to classifier.
- 33. MECHANICAL TURKS! Visit: http://www.crowdflower.com/
- 34. RESULTS: IS GUINNESS GOOD FOR YOU? NEG NEUT POS CLASSIFIER 50 0 50 MANUAL 23 30 47 AGREEMENT 40% 0% 72% (Agreement based on #tweets with matched judgments)
- 35. HUMANS DON’T ALWAYS AGREE… Advert used to say #Guinness is good for you but I think it is not acceptable to say that these days, but in moderation I thrive on it at 70 It's called a rotten apple #twobeersonecup #Guinness #angryorchard #delicious http://t.co/GUFxrYoF6i
- 36. CLASSIFY THIS… THERE ARE HUGE LINES AT THE APPLE STORE TODAY OR
- 37. BIG ENOUGH NUMBERS If the sample is large enough, won’t these problems get ironed out?
- 38. SAMPLE BIAS (1936 US PRESIDENTIAL ELECTION) SURVEY SIZE ROOSEVELT LITERARY DIGEST 2,400,000 43% GALLUP 50,000 54% ACTUAL 61% See: Tim Harford, “Big Data: are we making a big mistake?” (FT Magazine, 28 March 2008) I THINK YOU'll find iT'S 48 times bigger ALF LANDON
- 39. WIN 30 MINS OF FREE CONSULTANCY (VALUE £750) I'm loving #breakingnewsconf, @mediaczar. IT’S FAR TOO EASY TO GAME “POSITIVE SENTIMENT” METRICS
- 40. RECOMMENDATIONS LIFE ISN’T A POPULARITY CONTEST DON’T LET SENTIMENT BECOME A KPI MAKE SENTIMENT A TOOL FOR MORE COMPLEX RESEARCH PUT GREAT TECH TO MORE MEANINGFUL USE
- 41. THANK YOU! PLEASE DON’T ASK ME ANY TRICKY QUESTIONS THAT WILL MAKE ME LOOK STUPID I’M @MEDIACZAR ON TWITTER FEEL FREE TO COME AND TALK TO ME AFTERWARDS
Editor's Notes
- "respondents who returned their questionnaires represented only that subset of the population with a relatively intense interest in the subject at hand, and as such constitute in no sense a random sample... it seems clear that the minority of anti-Roosevelt voters felt more strongly about the election than did the pro-Roosevelt majority.” 50x bigger sample. Wrong answer.