Why your Sentiment is Wrong             A Statistical Analysis of the Sentiment TaskT.R. Fitz-Gibbon — Chief ScientistText...
Subjectivity    “There is no true interpretation of    anything; interpretation is a vehicle in    the service of human co...
Subjectivity    Subject: iPhone 4 battery life, not so good?      “I would never get 7              “Anyway 5 hours is    ...
Sentiment Analysis vs. Semantic Analysis    Sentiment Analysis      Semantic Analysis    Ignores most Data       Analyzes ...
The Sentiment Task    Natural Language Processing                 Manual Analysis by Experts                              ...
The Sentiment Task    Natural Language Processing                 Manual Analysis by Experts                              ...
The Sentiment Task    Natural Language Processing                     Manual Analysis by Experts                          ...
Why Sentiment FailsCauses                          Effects1. Narrow view of meaning       - Ignores data2. Low statistical...
1. Why Sentiment Fails - Narrow View of Meaning    Percentage of posts that contain sentiment                             ...
2. Why Sentiment Fails - Statistical Confidence10                                        © 2011 Networked Insights
2. Why Sentiment Fails - Statistical Confidence11                                        © 2011 Networked Insights
2. Why Sentiment Fails - Statistical Confidence12                                        © 2011 Networked Insights
2. Why Sentiment Fails - Statistical Confidence13                                        © 2011 Networked Insights
2. Why Sentiment Fails - Statistical Confidence14                                        © 2011 Networked Insights
2. Why Sentiment Fails - Statistical Confidence15                                        © 2011 Networked Insights
2. Why Sentiment Fails - Statistical ConfidenceConfidence intervals for a sample of four readers                          ...
2. Why Sentiment Fails - Statistical Confidence     Inter-Reader Agreement   Sample Size 95% Confidence                   ...
Why Sentiment Fails - Putting it all Together     x 10^4                                      Expected Outcome      Correc...
What is the Alternative               Semantic Analysis                Topic Discovery                 Topic Trending19   ...
Topic Discovery - Apple Topic Tree20                                   © 2011 Networked Insights
Topic Discovery - Apple Topic Tree21                                   © 2011 Networked Insights
Topic Trending - Apple                  ”iPhone Problems" and “Support" are                  both high before “Back to the...
Topic Trending - Moms and CPG     “baby, baby clothes”     Moms discussing fabric care   “saving money” drives            ...
Sentiment Analysis vs. Semantic AnalysisSentiment Analysis      Semantic Analysis    Semantic ValueIgnores most Data      ...
What if I need Sentiment Analysis     3 Questions to Ask your Provider:       •   What is the inter-reader agreement of yo...
We fuel insights, helping brands and their       agencies make better marketing decisions.T.R. Fitz-Gibbon, Chief Scientis...
Upcoming SlideShare
Loading in …5
×

Why Your Sentiment Is Wrong by Networked Insights

2,359 views

Published on

Published in: Business, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,359
On SlideShare
0
From Embeds
0
Number of Embeds
121
Actions
Shares
0
Downloads
12
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Why Your Sentiment Is Wrong by Networked Insights

  1. 1. Why your Sentiment is Wrong A Statistical Analysis of the Sentiment TaskT.R. Fitz-Gibbon — Chief ScientistText Analytics Summit May, 2011
  2. 2. Subjectivity “There is no true interpretation of anything; interpretation is a vehicle in the service of human comprehension. The value of interpretation is in enabling others to fruitfully think about an idea.” - Andreas Buja2 © 2011 Networked Insights
  3. 3. Subjectivity Subject: iPhone 4 battery life, not so good? “I would never get 7 “Anyway 5 hours is hours just browsing enough for me. It was with Wi-Fi. Not even 6 enough with 3GS. hours. I think my Maybe with next iPhone record has been 5 I will have better luck hours 30 minutes with the battery.” something.”3 © 2011 Networked Insights
  4. 4. Sentiment Analysis vs. Semantic Analysis Sentiment Analysis Semantic Analysis Ignores most Data Analyzes all Data Results Determined Results Determined by Chance by Data High Subjective Error Communicates Subjectivity4 © 2011 Networked Insights
  5. 5. The Sentiment Task Natural Language Processing Manual Analysis by Experts Sentiment Analysis Machine Learning Manual Analysis by Crowdsourcing Positive Negative Neutral5 © 2011 Networked Insights
  6. 6. The Sentiment Task Natural Language Processing Manual Analysis by Experts Sentiment Analysis Machine Learning Manual Analysis by Crowdsourcing Happy Sad Indifferent6 © 2011 Networked Insights
  7. 7. The Sentiment Task Natural Language Processing Manual Analysis by Experts Sentiment Analysis Machine Learning Manual Analysis by Crowdsourcing Intends to Purchase Intends to Renew Intends to Cancel7 © 2011 Networked Insights
  8. 8. Why Sentiment FailsCauses Effects1. Narrow view of meaning - Ignores data2. Low statistical confidence - Results left up to chance8 © 2011 Networked Insights
  9. 9. 1. Why Sentiment Fails - Narrow View of Meaning Percentage of posts that contain sentiment Data is based on a 500-post sentiment study we conducted around smartphones. The posts were classified by 20 people each. Posts were assigned to a sentiment category based on a majority vote. Only about 10% of posts were found to contain sentiment.9 © 2011 Networked Insights
  10. 10. 2. Why Sentiment Fails - Statistical Confidence10 © 2011 Networked Insights
  11. 11. 2. Why Sentiment Fails - Statistical Confidence11 © 2011 Networked Insights
  12. 12. 2. Why Sentiment Fails - Statistical Confidence12 © 2011 Networked Insights
  13. 13. 2. Why Sentiment Fails - Statistical Confidence13 © 2011 Networked Insights
  14. 14. 2. Why Sentiment Fails - Statistical Confidence14 © 2011 Networked Insights
  15. 15. 2. Why Sentiment Fails - Statistical Confidence15 © 2011 Networked Insights
  16. 16. 2. Why Sentiment Fails - Statistical ConfidenceConfidence intervals for a sample of four readers When three out of four readers agree on the sentiment of a post, we can only be 35% confident that a majority of all readers would agree. Normally, statistical significance at the 95% level is desired (for research and opinion polls). This example is only statistically significant at the 35% level. Thus, in this case we have not yet sampled enough readers to concluded that we know the sentiment of this post.16 © 2011 Networked Insights
  17. 17. 2. Why Sentiment Fails - Statistical Confidence Inter-Reader Agreement Sample Size 95% Confidence 90% < 10 75% ~ 20 65% ~ 50 55% ~ 50017 © 2011 Networked Insights
  18. 18. Why Sentiment Fails - Putting it all Together x 10^4 Expected Outcome Correct of Value Sentiment Analysis (Positive) Sentiment as a Percent18 © 2011 Networked Insights
  19. 19. What is the Alternative Semantic Analysis Topic Discovery Topic Trending19 © 2011 Networked Insights
  20. 20. Topic Discovery - Apple Topic Tree20 © 2011 Networked Insights
  21. 21. Topic Discovery - Apple Topic Tree21 © 2011 Networked Insights
  22. 22. Topic Trending - Apple ”iPhone Problems" and “Support" are both high before “Back to the Mac" and quite low after Support, Apple Products Conversation becomes all about buying iPhones and Mac Problem iPhone Buy iPhone Volume Mac OS, Mac Pro Apples big "Back to the Mac" event The tail end of antenna-gate, signal issues,22 glass cracking, daylight savings bug © 2011 Networked Insights
  23. 23. Topic Trending - Moms and CPG “baby, baby clothes” Moms discussing fabric care “saving money” drives this acceleration around experiences a lone “cloth diapers” spike around Christmas23 © 2011 Networked Insights
  24. 24. Sentiment Analysis vs. Semantic AnalysisSentiment Analysis Semantic Analysis Semantic ValueIgnores most Data Analyzes all Data Understand the Entire ConversationResults Determined Results Determined Understand the Actualby Chance by Data Conversation Communicates Actually UnderstandHigh Subjective Error the Conversation Subjectivity 24 © 2011 Networked Insights
  25. 25. What if I need Sentiment Analysis 3 Questions to Ask your Provider: • What is the inter-reader agreement of your manually scored sentiment data? • When you manually score/label posts with sentiment, to how many readers do you give each post? • For what type of posts was your solution designed and tested?25 © 2011 Networked Insights
  26. 26. We fuel insights, helping brands and their agencies make better marketing decisions.T.R. Fitz-Gibbon, Chief Scientist Founded in 2006 by industry leaders and608-237-1867 seasoned entrepreneurs in the fields of socialtr.fitzgibbon@networkedinsights.com media and customer intelligence. Headquartered in Madison, WI, with officesnetworkedinsights.com in New York and Chicago.

×