Targeting Sentiment Expressions through Supervised Ranking of Linguistic Configurations

2,348 views

Published on

A talk given at ICWSM 2009. The original paper can be found at http://www.jasonkessler.com/icwsm09.pdf

Published in: Technology, Education
  • ICWSM 2009 paper here: http://aaai.org/ocs/index.php/ICWSM/09/paper/view/190/413
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Targeting Sentiment Expressions through Supervised Ranking of Linguistic Configurations

  1. 1. Targeting Sentiment Supervised Ranking of Linguistic Configurations Jason Kessler Nicolas Nicolov Indiana University J.D. Power and Associates, McGraw Hill
  2. 2. Sentiment Analysis ―While the dealership was easy to find and the salesman was friendly, the car I bought turned out to be a disappointment.‖ • Bag of words: – Two positive terms, one negative term – Conclusion: author likes the car
  3. 3. What if we knew the sentiment targets? ―While the dealership was easy to find and the salesman was friendly, the car I bought turned out to be a disappointment.‖
  4. 4. Outline • Sentiment expressions • Finding sentiment targets • Previous work • Our approach: supervised ranking • Evaluation
  5. 5. Sentiment Expressions • Single or multi-word phrases – Express evaluation • Contextual polarity – I like the car (positive) – It is a lemon (negative) – The camera is not small (negative) • Assume annotation of sentiment expressions, their polarity
  6. 6. Targets • Target = word or phrase which is the object of evaluation • Sentiment expressions only link to physical targets:  Bill likes to drive.  Bill likes to drive the car. • Multiple targets possible: — Bill likes the car and the bike.
  7. 7. Targets (2) Some mentions are not targets. – Sue likes1 Al’s car1. Tricky cases: – The car2 frightens2 Mary. – Mary4’s dislike3 of Bill’s car3 is a turn-off4 for him. – Look at those pancakes5. My mouth is watering5.
  8. 8. Problem • Given annotation of mentions and sentiment expressions • Identify targets of all sentiment expressions
  9. 9. Manual Annotations Entity-level sentiment: Positive John recently purchased a digital camera. PERSON CAMERA TARGET TARGET TARGET COREF It had a great zoom lens, a mildly disappointingflash, CAMERA CAMERA-PART CAMERA-PART PART-OF PART-OF Entity-level sentiment: Mixed TARGET LESS and was very compact.He also considered a Cannon PERSON CAMERA FEATURE-OF TARGET MORE which, while priced highly had a better flash. CAMERA-FEATURE CAMERA-PART DIMENSION
  10. 10. Other Annotations • Sentiment expressions • Intensifiers, negators, neutralizers, committers • Targets, opinion holders • Mentions and semantic types • Coreference, part-of, feature-of, instance-of • Entity-level sentiment • Comparisons and their arguments
  11. 11. Corpus Size/Statistics • Micro-averaged harmonic mean of precision between annotator pairs • Sentiment expressions: 76.84 • Mentions: 87.19 • Targets: 81.55 Sentiment Domain Docs Tokens Sentences Expressions Mentions Cars 111 80,560 4,496 3,353 16,953 Camera 69 38,441 2,218 1,527 9,446 Total 180 119,001 6,614 4,880 26,399
  12. 12. Baseline - Proximity • Proximity approach: – Nearest mention selected as target – Break ties by preferring right-hand mention – Breaks on: Sue likes1 Al’s car1.
  13. 13. Baseline – One Hop • Run a dependency parser – Mentions that govern or are governed by SE – Use Stanford dependency parser – Partially breaks on: DOBJ Sue likes1 Al’s car1. NSUBJ POSS M. de Marneffe, B. MacCartney & C. Manning. 2006. ―Generating typed dependency parses from phrase structure parses‖. LREC 2006.
  14. 14. Previous Work – Decision List • Decision list of dependency paths: – Ordered list of 41 labeled dependency paths between sentiment expression and mention – Top path connecting a sentiment expression to a mention mention is the target DOBJ DOBJ … 4. SE –DOBJ Mention 5. SE –NSUBJ Mention Sue likes1 Al’s car. It1 upset1 Amy. … NSUBJ POSS NSUBJ Sample list slice Kenneth Bloom, Navendu Garg & Shlomo Argamon. 2007. ―Extracting Appraisal Expressions‖. NAACL-HTL 2007.
  15. 15. Our Approach • Learning to target from a corpus: – Bill likes1 the car1 and Sarah knows it. – Classification: • Three independent binary classifier calls • features(like, car) =? Target/Not Target • features(like, Bill) =? Target/Not Target • features(like, Sarah) =? Target/Not Target
  16. 16. Our Approach • Supervised Ranking – Bill likes1 the car1 and Sarah knows it. – Rank Bill, car, and Sarah by likelihood of being a target of like • Ensure car is ranked the highest – Learn score function s to appx. rank: • Input: features relating sentiment expression, mention • Output: number that reflects rankings • s(features(like, car)) < s(features(like, Bill)) • s(features(like, car)) < s(features(like, Sarah))
  17. 17. Our Approach • Learn score function given ranks: – Given: • My car gets good1 gas milage1. – Ranks for good: gas mileage: 0, car: 1, my: 1, • It handles2 well2. – Ranks for well: handles: 0, it: 1 – For score function s ensure that: • s(features(good, gas mileage)) < s(features(good, car)) • s(features(good, gas mileage)) < s(features(good, my)) • s(features(well, handles)) < s(features(well, it)) – Ensure difference ≥ 1
  18. 18. Our Approach • Use RankSVM to perform supervised ranking Joachims, T. 2002. Optimizing search engines using clickthrough data. KDD. • Features – Incorporate syntax (dependency parse) – Extract labeled-dependency paths between mentions and sentiment expressions
  19. 19. Features AUX DOBJ Paul likes1 to drive the blue car1 NSUBJ DET XCOMP Feature: likes blue car Example # tokens distance 3 # sentiment expressions 0 between # mentions between 0 Lexical path to drive the Lexical stem path to drive the POS path TO, VBD, DT Encoded as Stem + labeled dep. path like :: ↓XCOMP, ↓DOBJ binary features Labeled dependency path ↓XCOMP, ↓DOBJ Semantic type of mention Car POS tags of s.exp., mention VBP, NN
  20. 20. Results – All parts-of-speech • 10 fold cross validation over all data 90 80 70 60 50 Precision 40 Recall 30 F-score 20 10 0 Proximity One hop Decision List RankSVM
  21. 21. Results - Verbs Problem: John likes1 the car1 (-dobj) vs. The car2 upset2 me. (-nsubj) 90 80 70 60 50 Precision 40 Recall 30 F-Score 20 10 0 Proximity One hop Decision List RankSVM
  22. 22. Results - Adjectives AMOD Problems: terrible horrible, no good, very bad, movie. DEP 90 80 70 60 50 Precision 40 Recall 30 F-Score 20 10 0 Proximity One hop Decision List RankSVM
  23. 23. Future work – Apply techniques to targeting intensifiers, etc. – Inter-sentential targeting – Domain adaptation – Other approaches Kobayashi et al. (2006), Kim and Hovy (2006) Conclusions – Proximity works well – Substantial performance gains from supervised ranking and syntactic and semantic features
  24. 24. Thank you! Special thanks to: • Prof. Martha Palmer • Prof. Jim Martin • Dr. Miriam Eckert • Steliana Ivanova, • Ron Woodward • Prof. Michael Gasser • Jon Elsas
  25. 25. Dependency Features DOBJ AUX Paul likes1 to drive the blue car1 NSUBJ AMOD DET XCOMP Group sentiment expressions/mentions as single node: AUX DOBJ Paul likes1 to drive the blue car1 NSUBJ DET XCOMP
  26. 26. Dependency Features AUX DOBJ Paul likes1 to drive the blue car1 NSUBJ DET XCOMP Like, blue car: ↓XCOMP, ↓DOBJ Great1 car1 Great, car: ↑AMOD AMOD ↓ in front of grammatical relation indicates path is followed ↑ indicates path is followed in opposite direction
  27. 27. Previous Work • Kim & Hovy (2006) – Use FrameNet-based semantic role labeler on sentences with verb/adjective SEs – Some frame elements are considered always targeting (e.g. stimulus, problem) stimulus experiencer Bill2’s handling1 of the situation1 annoyed2 Sam. agent problem S.Kim & E.Hovy. 2006. ―Extracting Opinions, Opinion Holders, and Topics Expressed in Online News Media Text‖. Sentiment and Subjectivity in Text, ACL 2006.
  28. 28. Previous Work • Kobayashi et al. (2006) – Corpus based, statistical machine learning approach (Japanese product review corpus) – Determining winner reducible to binary classification • Bill likes1 the eraser1 and Sarah knows it. – Produces training data: » Features(Bill, eraser | like, sentence) -> Right » Features(eraser, Sarah | like, sentence) -> Left – To find like’s target » Winner of Bill vs. eraser competes against Sarah » Two calls to binary classifier – What features to use?, can’t have multiple targets Nozomi Kobayashi, Ryu Iida, Kentaro Inui, and Yuji Matsumoto. 2006. Opinion Mining on the Web by Extracting Subject-Attribute-Value Relations. In AAAI-CAAW 2006.
  29. 29. Our Approach • Supervised ranking (RankSVM): – Training data partitioned into subsets – Instances xi in each subset (k) are given relative rankings, PREF function give difference in ranking – Score function s should reflect partial orderings – We use SVMLight implementation Joachims, T. 2002. Optimizing search engines using clickthrough data. KDD. (Formulation from Lerman et al. EACL’09)
  30. 30. JDPA Sentiment Corpus

×