#like or #fail
How Can Computers Tell the Difference?
Mark Cieliebak
27.3.2014
Institute of Applied Information Systems (InIT)
ZHAW
Zurich University of Applied Sciences
- 10'000 students, 230 professo...
About Me
27.3.2014 Mark Cieliebak 3
Mark Cieliebak
Institute of Applied Information Technology (InIT)
ZHAW, Winterthur
Ema...
What is "Text Analytics"?
Synonyms: Text Mining, Natural Language Processing (NLP)
4Picture Sources: see slide "References...
Sample Application: Social Media
Monitoring
Text Analytics Components:
• Find relevant documents
• Hot topic Analysis
• se...
Sentiment Analysis for Product Reviews
"… WiFi Analytics is a free Android app that I find very
handy when it comes to tro...
Flavours of Sentiment Analysis
• Document Based
• Sentence Based
• Target-Specific
• Rating Prediction
27.3.2014 Mark Ciel...
Simple Sentiment Analysis
Idea: Count number of positive and negative words
"This analysis is good[+1]." +1 (pos)
"I find ...
Improvements
Sample Text
• "This analysis is good."
"This analysis is excellent."
• "The car is really very
expensive."
• ...
Improvements 2
Sample Text
• This analysis is not good.
• The car is appealing and I do
not find it expensive.
• I do not ...
Linguistic Analysis
-> RULE: Invert scores of words being in the same phrases as negation.
“I do not find the car expensiv...
Technology Stack
for Sentiment Analysis on Web Documents using Linguistic Analysis
Linguistic Framework
• Gate: text analy...
Main Approaches to Sentiment Analysis
Rule-Based Corpus-Based
27.3.2014 Mark Cieliebak 13
Predicted
Label
[9]
[10]
Corpus-Based Sentiment Analysis
27.3.2014 Mark Cieliebak 14
Predicted
Label
[10]
Corpus-Based Sentiment Analysis
Annotated Corpus
Sentence Polarity
This analysis is good. Pos
It looks awful. Neg
This car...
Sample Features for Tweets
• Word ngrams: presence or absence of contiguous sequences of 1, 2, 3,
and 4 tokens; noncontigu...
How good are Sentiment Analysis Tools?
27.3.2014 Mark Cieliebak 17
Evaluation
joint work with Oliver Dürr and Fatih Uzdilli [15]
• 7 Public Text Corpora
– Single Statements
– Different Medi...
Sentiment Analysis Tools
AlchemyAPI www.alchemyapi.com
Lymbix www.lymbix.com
ML Analyzer www.mashape.com/mlanalyzer/ml-ana...
Tool Accuracy
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accuracy
Best Tool per Corpus
Worst Tool per Corpus
20
61%
40%
Avg.
27.3.2014 Ma...
Tool Accuracy
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accuracy
Best Tool per Corpus
Worst Tool per Corpus
Overall Best Tool
21
61%
40%...
Summary
"They all suck…and we suck, too."
CEO of a Sentiment
Analysis Company
27.3.2014 Mark Cieliebak 22
State of the Art in Academia
SemEval 2013: Competition on Semantic Analysis
• Task 2A: Contextual Polarity Disambiguation
...
SemEval 2013 Task 2B
Results
• 36 Submissions
• Average F1 = 54%
• Only 9 teams above 50%, all others below!
• Best Team: ...
Can a Meta-Classifier do better?
1st Approach: Majority Classifier
 Sentiment with most votes chosen
Illustration:
2527.3...
Tool Accuracy
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accuracy
Best Tool per Corpus
Worst Tool per Corpus
2627.3.2014 Mark Cieliebak
Majority Classifier Close to Best Tool
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accuracy
Best Tool per Corpus
Worst Tool per Corpus
Maj...
2nd Approach: Random-Forest
2827.3.2014 Mark Cieliebak
api1 api2 api3 api9 annotation
Text 1 + - + o +
Text 2 - + o + -
Te...
Before Random Forest
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accuracy
Best Tool per Corpus
Worst Tool per Corpus
Majority Classifier
2...
Random Forest beats Best Single Tool
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Accuracy
Best Tool per Corpus
Worst Tool per Corpus
Major...
Challenges in Sentiment Analysis
• "Salaries for software engineers are extremely high."
 Context Dependencies
• “What a ...
Talk in Short!
1. Sentiment analysis is an
important challenge
2. Commercial tools classify 6
out of 10 docs wrong
3. Acad...
References
[1] http://www.teleportmyjob.com/blog/wp-content/uploads/2013/05/find_job.jpg
[2] http://screenshots.de.sftcdn....
Upcoming SlideShare
Loading in …5
×

#like or #fail - How Can Computers Tell the Difference?

985 views

Published on

Sentiment analysis appears to be one of the easier tasks in the realm of text analytics: given a text like a tweet or product review, decide whether it contains positive or negative opinion. This task is almost trivial for humans, but it turns out to be a true challenge for automated systems.

We explain the intrinsic difficulties of automated sentiment analysis; describe existing solution approaches; analyze performance of state-of-the-art sentiment analysis tools; and show how to improve their analysis accuracy using machine learing technologies.

Presentation at Machine Learning Meetup Zurich, 27.3.2014

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
985
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

#like or #fail - How Can Computers Tell the Difference?

  1. 1. #like or #fail How Can Computers Tell the Difference? Mark Cieliebak 27.3.2014
  2. 2. Institute of Applied Information Systems (InIT) ZHAW Zurich University of Applied Sciences - 10'000 students, 230 professors (FTE) InIT Institute of Applied Information Systems - more than 30 professors - Labs on Cloud Computing, Data Science etc. 2 [11] 27.3.2014 Mark Cieliebak
  3. 3. About Me 27.3.2014 Mark Cieliebak 3 Mark Cieliebak Institute of Applied Information Technology (InIT) ZHAW, Winterthur Email: ciel@zhaw.ch, Website: www.zhaw.ch/~ciel Text Analytics Open Data Automated Test Generation Research Interests Social Network Analysis
  4. 4. What is "Text Analytics"? Synonyms: Text Mining, Natural Language Processing (NLP) 4Picture Sources: see slide "References" [6] [5] [3] [2] [1] [4] 27.3.2014 Mark Cieliebak
  5. 5. Sample Application: Social Media Monitoring Text Analytics Components: • Find relevant documents • Hot topic Analysis • sentiment analysis 27.3.2014 Mark Cieliebak 5 [7]
  6. 6. Sentiment Analysis for Product Reviews "… WiFi Analytics is a free Android app that I find very handy when it comes to troubleshooting and monitoring a home network. " [8] 27.3.2014 Mark Cieliebak 6
  7. 7. Flavours of Sentiment Analysis • Document Based • Sentence Based • Target-Specific • Rating Prediction 27.3.2014 Mark Cieliebak 7
  8. 8. Simple Sentiment Analysis Idea: Count number of positive and negative words "This analysis is good[+1]." +1 (pos) "I find it beautiful[+1] and good[+1]." +2 (pos) "It looks terrible[-1]." -1 (neg) "This car has a blue color." 0 (neu) POSITIVE: good love nice ... NEUTRAL: hello see I … NEGATIVE: bad hate ugly ... Use Sentiment-Dictionary: 27.3.2014 Mark Cieliebak 8
  9. 9. Improvements Sample Text • "This analysis is good." "This analysis is excellent." • "The car is really very expensive." • "This car has an appealing design and comfortable seats, but it is expensive." Solution • Fine-Grained Dictionaries: "This analysis is good [+2]. " "This analysis is excellent [+3]. • Detect Booster Words: "The car is really very expensive[-1 -1 -2] ." • New Category "Mixed": "This car has an appealing[+1] design and comfortable[+1] seats, but it is expensive[-1]. "  mix(+2/-1) 27.3.2014 Mark Cieliebak 9
  10. 10. Improvements 2 Sample Text • This analysis is not good. • The car is appealing and I do not find it expensive. • I do not find the car expensive and it is appealing. Solution • Invert all Scores: "This analysis is not[*-1] good[+3]." • Invert only score of words occuring after the negation: "The car is appealing[+3] and I do not[*-1] find it expensive[-2]"  Need to “understand” the sentence  Need linguistic analysis! 27.3.2014 Mark Cieliebak 10
  11. 11. Linguistic Analysis -> RULE: Invert scores of words being in the same phrases as negation. “I do not find the car expensive[+2] and it is appealing[+3].” → +5 (pos) Sentence Sentence Conj. Sentence Noun Phrase Verb Phrase Verb Adverb Verb Noun Phrase Adj. Noun Phrase Verb Phrase Det. Det Noun Det. Verb Participle I do not find the car expensive and it is appealing 27.3.2014 Mark Cieliebak 11
  12. 12. Technology Stack for Sentiment Analysis on Web Documents using Linguistic Analysis Linguistic Framework • Gate: text analysis framework with experimentation GUI; includes sentence splitter, tokenizer, chunker, pos-tagger, etc; plugin- mechanism for new libraries; very widely used. http://gate.ac.uk/ • OpenNLP: Java-based, easy to use; includes tokenization, sentence segmentation, part-of-speech tagging, named entity extraction, etc; by Apache Foundation. http://opennlp.apache.org/ Web Boilerplate Extraction • Boilerpipe: removes ads, menus, HTML-tags etc. http://boilerpipe-web.appspot.com/ Language Analysis • Nutch language identifier plugin: 14 languages preimplemented, „easy“ to add more. http://wiki.apache.org/nutch/LanguageIdentifierPlugin Sentence Splitting • OpenNLP: very easy to use, in Java. http://opennlp.apache.org/ • Punkt. In Python http://nltk.org/api/nltk.tokenize.html Part-of-Speech Tagger • TreeTagger: language-independent, several languages implemented;. http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/ • TweetNLP: POS-tagger just for Twitter. http://www.ark.cs.cmu.edu/TweetNLP/ Chunker • TreeTagger: supports English, German, French. http://www.cis.uni-muenchen.de/~schmid/tools/TreeTagger/ Dictionaries • WordNet: large lexical database of English. http://wordnet.princeton.edu/ • SentiWordNet: English dictionalry with sentiment annotations. http://sentiwordnet.isti.cnr.it/ 27.3.2014 Mark Cieliebak 12
  13. 13. Main Approaches to Sentiment Analysis Rule-Based Corpus-Based 27.3.2014 Mark Cieliebak 13 Predicted Label [9] [10]
  14. 14. Corpus-Based Sentiment Analysis 27.3.2014 Mark Cieliebak 14 Predicted Label [10]
  15. 15. Corpus-Based Sentiment Analysis Annotated Corpus Sentence Polarity This analysis is good. Pos It looks awful. Neg This car has a blue color. Neu This car has an appealing design, comfortable seats, but it is expensive. Mix This car has a very appealing design, comfortable seats, but it is really expensive. Mix This analysis is not good. Neg This car has an appealing design, comfortable seats and it is not expensive. Mix This movie was like a horror event. Neg This car is appealing and is not expensive. Mix ... ... 27.3.2014 Mark Cieliebak 15
  16. 16. Sample Features for Tweets • Word ngrams: presence or absence of contiguous sequences of 1, 2, 3, and 4 tokens; noncontiguous ngrams • POS: the number of occurrences of each part-of-speech tag • Sentiment Lexica: each word annotated with tonality score (-1..0..+1) • Negation: the number of negated contexts • Punctuation: the number of contiguous sequences of exclamation marks, question marks, and both exclamation and question marks • Emoticons: presence or absence, last token is a positive or negative emoticon; • Hashtags: the number of hashtags; • Elongated words: the number of words with one character repeated (e.g. ‘soooo’) from: Mohammad et al., SemEval 2013 27.3.2014 Mark Cieliebak 16
  17. 17. How good are Sentiment Analysis Tools? 27.3.2014 Mark Cieliebak 17
  18. 18. Evaluation joint work with Oliver Dürr and Fatih Uzdilli [15] • 7 Public Text Corpora – Single Statements – Different Media Types • Tweet, News, Review, Speech Transcript – Total: 28653 Texts • 9 Commercial APIs – Stand-alone – Free for this evaluation – Arbitrary Text NEGATIVEPOSITIVE OTHER ( neutral / mixed ) 1827.3.2014 Mark Cieliebak
  19. 19. Sentiment Analysis Tools AlchemyAPI www.alchemyapi.com Lymbix www.lymbix.com ML Analyzer www.mashape.com/mlanalyzer/ml-analyzer Repustate www.repustate.com Semantria www.semantria.com Sentigem www.sentigem.com Skyttle www.skyttle.com Textalytics core.textalytics.com Text-processing www.text-processing.com 27.3.2014 Mark Cieliebak 19
  20. 20. Tool Accuracy 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Accuracy Best Tool per Corpus Worst Tool per Corpus 20 61% 40% Avg. 27.3.2014 Mark Cieliebak
  21. 21. Tool Accuracy 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Accuracy Best Tool per Corpus Worst Tool per Corpus Overall Best Tool 21 61% 40% 59% Avg. 27.3.2014 Mark Cieliebak
  22. 22. Summary "They all suck…and we suck, too." CEO of a Sentiment Analysis Company 27.3.2014 Mark Cieliebak 22
  23. 23. State of the Art in Academia SemEval 2013: Competition on Semantic Analysis • Task 2A: Contextual Polarity Disambiguation • Task 2B: Message Polarity Classification Corpus: 15'000 tweets from 2012 on various topics annotated via Mechanical Turk 27.3.2014 Mark Cieliebak 23
  24. 24. SemEval 2013 Task 2B Results • 36 Submissions • Average F1 = 54% • Only 9 teams above 50%, all others below! • Best Team: F1 = 69% • For comparison: best commercial tool has F1 = 62% 27.3.2014 Mark Cieliebak 24
  25. 25. Can a Meta-Classifier do better? 1st Approach: Majority Classifier  Sentiment with most votes chosen Illustration: 2527.3.2014 Mark Cieliebak api1 api2 api3 api4 api5 api6 api7 Majority Text 1 + + - o - + o + Text 2 - + + - - - - - Text 3 - o + + + + - + Text n o o + o - o o o
  26. 26. Tool Accuracy 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Accuracy Best Tool per Corpus Worst Tool per Corpus 2627.3.2014 Mark Cieliebak
  27. 27. Majority Classifier Close to Best Tool 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Accuracy Best Tool per Corpus Worst Tool per Corpus Majority Classifier 2727.3.2014 Mark Cieliebak
  28. 28. 2nd Approach: Random-Forest 2827.3.2014 Mark Cieliebak api1 api2 api3 api9 annotation Text 1 + - + o + Text 2 - + o + - Text 3 - o - + - Text 4 + o + - + Text 5 + o + o o Text 6 + o o - o Text 7 + - + o unknown Text 8 + + o - unknown Text 9 o - + o unknown Random Forest Classifier + + o Train Train Train Train Predict Predict Predict Train Train
  29. 29. Before Random Forest 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Accuracy Best Tool per Corpus Worst Tool per Corpus Majority Classifier 2927.3.2014 Mark Cieliebak
  30. 30. Random Forest beats Best Single Tool 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Accuracy Best Tool per Corpus Worst Tool per Corpus Majority Classifier Random Forest Classifier 3027.3.2014 Mark Cieliebak
  31. 31. Challenges in Sentiment Analysis • "Salaries for software engineers are extremely high."  Context Dependencies • “What a great car, it stopped working the second day.”  Sarcastic Statements • "#YouCantDateMe if u still sag ur pants super hard...dat shit is played the fuck out!!! "  Informal Language • Huge Effort for New Languages 27.3.2014 Mark Cieliebak
  32. 32. Talk in Short! 1. Sentiment analysis is an important challenge 2. Commercial tools classify 6 out of 10 docs wrong 3. Academic systems are not really better 4. Random forest outperform even the best tools 27.3.2014 Mark Cieliebak 32 [14] [12] [13]
  33. 33. References [1] http://www.teleportmyjob.com/blog/wp-content/uploads/2013/05/find_job.jpg [2] http://screenshots.de.sftcdn.net/de/scrn/3340000/3340669/swiftkey-23-630x535.png [3] https://lh4.ggpht.com/Fs4HLaMCgmY6O7BWGKZB8LdOM8GmiRAbKUcdJysNp0k74xmPMTSGVHigwvaLy_Tw=w300 [4] http://researcher.watson.ibm.com/researcher/files/us-bansal/watson2.jpg [5] http://www.betadaily.com/wp-content/uploads/2011/02/BusinessPlanExecutiveSummary.jpg [6] http://everycook.org/cms/en/hardware-en/pictures-en# [7] http://cdn.fulltraffic.net/images/blog/social-media-monitoring-process.png [8] http://www.pcmag.com/article2/0,2817,2426416,00.asp [9] http://i1059.photobucket.com/albums/t424/shmi11y/UoS%20LingSite/tompushedthecar.png [10] http://bigsnarf.files.wordpress.com/2013/04/supervised.png [11] http://www.alumni.sml.zhaw.ch/portals/0/Volkartgeb%C3%A4ude%20gek%C3%BCrzt%202.jpg [12] http://www.bitspeed.com/wp-content/uploads/2011/09/Sign-multi-cast.jpg [13] http://wwwdelivery.superstock.com/WI/223/1829/PreviewComp/SuperStock_1829-47184.jpg [14] http://www.odditycentral.com/art/arborsculpture-the-art-of-turning-young-trees-into-living-works-of-art.html [15] Cieliebak, Dürr, Uzdilli, ESSEM 2013 27.3.2014 Mark Cieliebak 33

×