Why Your Sentiment Is Wrong by Networked Insights

Why your Sentiment is Wrong A Statistical Analysis of the Sentiment TaskT.R. Fitz-Gibbon — Chief ScientistText Analytics Summit May, 2011

Subjectivity “There is no true interpretation of anything; interpretation is a vehicle in the service of human comprehension. The value of interpretation is in enabling others to fruitfully think about an idea.” - Andreas Buja2 © 2011 Networked Insights

Subjectivity Subject: iPhone 4 battery life, not so good? “I would never get 7 “Anyway 5 hours is hours just browsing enough for me. It was with Wi-Fi. Not even 6 enough with 3GS. hours. I think my Maybe with next iPhone record has been 5 I will have better luck hours 30 minutes with the battery.” something.”3 © 2011 Networked Insights

Sentiment Analysis vs. Semantic Analysis Sentiment Analysis Semantic Analysis Ignores most Data Analyzes all Data Results Determined Results Determined by Chance by Data High Subjective Error Communicates Subjectivity4 © 2011 Networked Insights

The Sentiment Task Natural Language Processing Manual Analysis by Experts Sentiment Analysis Machine Learning Manual Analysis by Crowdsourcing Intends to Purchase Intends to Renew Intends to Cancel7 © 2011 Networked Insights

1. Why Sentiment Fails - Narrow View of Meaning Percentage of posts that contain sentiment Data is based on a 500-post sentiment study we conducted around smartphones. The posts were classified by 20 people each. Posts were assigned to a sentiment category based on a majority vote. Only about 10% of posts were found to contain sentiment.9 © 2011 Networked Insights

2. Why Sentiment Fails - Statistical ConfidenceConfidence intervals for a sample of four readers When three out of four readers agree on the sentiment of a post, we can only be 35% confident that a majority of all readers would agree. Normally, statistical significance at the 95% level is desired (for research and opinion polls). This example is only statistically significant at the 35% level. Thus, in this case we have not yet sampled enough readers to concluded that we know the sentiment of this post.16 © 2011 Networked Insights

Topic Trending - Apple ”iPhone Problems" and “Support" are both high before “Back to the Mac" and quite low after Support, Apple Products Conversation becomes all about buying iPhones and Mac Problem iPhone Buy iPhone Volume Mac OS, Mac Pro Apples big "Back to the Mac" event The tail end of antenna-gate, signal issues,22 glass cracking, daylight savings bug © 2011 Networked Insights

Topic Trending - Moms and CPG “baby, baby clothes” Moms discussing fabric care “saving money” drives this acceleration around experiences a lone “cloth diapers” spike around Christmas23 © 2011 Networked Insights

Sentiment Analysis vs. Semantic AnalysisSentiment Analysis Semantic Analysis Semantic ValueIgnores most Data Analyzes all Data Understand the Entire ConversationResults Determined Results Determined Understand the Actualby Chance by Data Conversation Communicates Actually UnderstandHigh Subjective Error the Conversation Subjectivity 24 © 2011 Networked Insights

What if I need Sentiment Analysis 3 Questions to Ask your Provider: • What is the inter-reader agreement of your manually scored sentiment data? • When you manually score/label posts with sentiment, to how many readers do you give each post? • For what type of posts was your solution designed and tested?25 © 2011 Networked Insights

We fuel insights, helping brands and their agencies make better marketing decisions.T.R. Fitz-Gibbon, Chief Scientist Founded in 2006 by industry leaders and608-237-1867 seasoned entrepreneurs in the fields of socialtr.fitzgibbon@networkedinsights.com media and customer intelligence. Headquartered in Madison, WI, with officesnetworkedinsights.com in New York and Chicago.

http://info.networkedinsights.com	4
http://www.gleanster.com	4
https://si0.twimg.com	3
http://www.slideshare.net	1

Why Your Sentiment Is Wrong by Networked Insights

by Networked Insights

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 12

Statistics