Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Jennifer Prendki, Principal Data Scientist, @WalmartLabs at MLconf SF 2016

593 views

Published on

Review Analysis: an Approach to Leveraging User-Generated Content in the Context of Retail: What are customers really thinking? What are they looking for specifically when shopping for a product? And if they are satisfied with their purchase, what is the main reason?

Today’s technology offers many different avenues for customers to express themselves, set their expectations in writing, and share their opinion, frustration or satisfaction regarding all kinds of products and services. ‘Leaving a review’ has become an integral part of the purchase process. Through reviews, customers are volunteering invaluable information that can be turned into insights that would help drive business decisions (if you are a retailer), or help you make a successful purchase (if you are a customer). Yet the amount of data available to make these decisions is oftentimes extremely large, and it might be difficult for a human to read and synthesize all that has been said about their product of interest.

Review analysis and opinion mining offer solutions to automate the analysis of customer feedback through large-scale machine learning, natural language processing and sentiment analysis, and allow retailers to better understand their customers… as well as their data.

In this talk, I will present the various ways in which machine learning techniques can be used to extract the most significant features for a given category of products. I will then dig into a process aiming at identifying the sentiments relative to these features, and a useful way to aggregate this information into insights that are both usable and readable by any user. I will end with mentions to some of the challenges met when trying to extract objective information from a data source likely tainted with human subjectivity in an ever-changing market.

Published in: Technology
  • Be the first to comment

Jennifer Prendki, Principal Data Scientist, @WalmartLabs at MLconf SF 2016

  1. 1. Review Analysis: An Approach to Leveraging User-Generated Content in the Context of Retail Jennifer Prendki, Principal Data Scientist Walmart Global e-Commerce California, USA The Machine Learning Conference, San Francisco, CA 11/11/2016
  2. 2. Outline • Business motivation • Algorithm Pipeline • Feature Space Computation • Sentiment Capture • Real-Life Examples and Results • Future Work and Conclusions 2  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  3. 3. Business Motivation 3  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  4. 4. Business Motivation 4  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  5. 5. Business Motivation 5  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  6. 6. Business Motivation 6  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  7. 7. Business Motivation “ I bought this for my daughter to do her college work on, it's been great, no problems so far. “ [SuperMom72] “ Works like a charm, would definitely recommend to anyone on a budget. “ [Vamsy] “ Fast CPU but slow disk drive slows everything down. ” [TalonBay] “ I don't do gaming or downloading movies or music, so for those folks I can't speak to the performance. But for surfing the web, checking email, etc., this computer will save you time for watching the little ball spin!” [Anonymous] 7  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  8. 8. Review Analysis: A Current Landscape • Sentiment analysis • Best known use case: Social Media Analysis/Tweets Why tweets?  shorter, condensed, highly sentimental content • Movie review analysis: Kaggle: Analysis of the ‘Rotten Tomatoes’ Dataset • Regarding product review analysis • Little to no papers regarding product review analysis at commercial scale • Shortage of work regarding combination of topic modeling and sentiment analysis 8 Our research: Combine feature computation and sentiment analysis to summarize reviewers’ opinions about a specific product  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  9. 9. Algorithm Pipeline 9 Product 𝛼 Product 𝛽 Product 𝛾 Review Review Review Review Review Review Fc Category C Feature Space Computation F 𝛼 F 𝛽 F 𝛾 Feature Space Reduction  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  10. 10. Algorithm Pipeline 10 Product 𝛼 Product 𝛽 Product 𝛾 Review Review Review Review Review Review Category C F 𝛼 F 𝛽 F 𝛾 Sentiment Sentiment Sentiment Sentiment Sentiment Sentiment Sentiment Computation For Each Review … Sentiment Computed For Relevant Features  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  11. 11. Algorithm Pipeline 11 Product 𝛼 Product 𝛽 Product 𝛾 Review Review Review Review Review Review Category C Sentiment Sentiment Sentiment Sentiment Sentiment Sentiment 𝜎𝑡, 𝛼, 𝑓 𝜎𝑡, 𝛽, 𝑓 𝜎𝑡, 𝛾, 𝑓 ∀ 𝑡 ∈ 𝜏 ∀ 𝑓 ∈ F 𝛼 ∀ 𝑡 ∈ 𝜏 ∀ 𝑓 ∈ Fβ ∀ 𝑡 ∈ 𝜏 ∀ 𝑓 ∈ F 𝛾  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions Sentiment Computation For Each Review … Sentiment Computed For Relevant Features
  12. 12. Feature Space Computation • Textual reviews go through a careful process: • TF/TF-Idf transform on documents • Stop words removal, stemming, part-of-speech selection • Spell-checking • etc. • ‘Synonym’ computation • Can be done using Word Embedding (glove, word2vec) • Can be done building synonym graph using dictionary/Wikipedia • Is complex and tricky, context-sensitive, unsupervised 12 In short: Creating synonym sets is difficult, and challenging as an online algorithm In short: Preprocessing crucial to extracting relevant features  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  13. 13. [0-1] Intensity of negativity in sentence {'neg': 0.0, 'neu': 0.58, 'pos': 0.42, 'compound': 0.4404} [0-1] Intensity of neutrality in sentence [0-1] Intensity of positivity in sentence [-1,1] Combination of positive and negative sentiments. Allows positive and negative to ‘compensate’ one another Sentiment Capture with Vader VADER: Valence Aware Dictionary and sEntiment Reasoner • Is a Python sub-module found of the nltk module • Is a lexicon and rule-based sentiment analysis tool • Is specifically attuned to sentiments expressed in social media • Is fully open-sourced, developed and licensed by MIT 13 Sentiment is not boolean posneg neu Sentiment as a PDF  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  14. 14. Sentiment Capture with Vader 14 ”This computer is good deal.” “This computer is a bad deal.” pos 0.42 0.0 neu 0.58 0.533 neg 0.0 0.467 compound 0.4404 -0.5423 ”This computer is not powerful.” “This computer is not that powerful.” “This computer is not powerful, but I like it anyways.” “This computer is not that powerful, but I like it anyways.” pos 0.0 0.0 0.0 0.252 neu 0.632 0.682 0.618 0.619 neg 0.368 0.318 0.382 0.129 compound -0.3252 -0.3252 -0.5157 0.3786  Vader is sensitive to adverbs, punctuation, case, emoticons and nuances…  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  15. 15. Sentiment Capture with Vader 15 Product A “The design and picture quality are amazing!” I love it! Just perfect for people on a budget. And it is beautifully designed!! “Pretty good, but I am not a fan of the design.” “I don’t think it’s possible to find better for the price” design Product B “I just HATE the design!!” “Okay computer. Wish I read the other reviews first.” design picture quality picture quality + 0.39 + 0.39 + 0.56 ~ 0.48 - 0.62  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  16. 16. Sentiment Capture with Vader 16 Product A design Product B design picture quality picture quality design picture quality battery value design battery CPU processor 0.48 0.39 0.46 0.62 0.49 NA NA NA 0.75 0.87NANA 0.43 NA NA 3 1 4 2 1 0.39 0.60 NA NA 1 1 1 NA NA NA NA 0.62 ✍ Scraping Summarizing Sentiment Intensity + 0.39 + 0.56 ~ 0.49 - 0.62 + 0.39 Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  17. 17. Results Discussion: Real Life Example 17 Product BProduct A  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  18. 18. Results Discussion: Real Life Example 18 screen design quality performance NEG NEU POSscreen design performance quality screen design quality performance Some dissatisfaction with overall quality Reviewers are rather happy with keyboard Weight is better for product A than for product B Customers satisfied with keyboard, display, screen, design, …Product B’s weakness is battery life The product’s features are well documented Product A Product B  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  19. 19. Results Discussion: Real Life Example 19 This laptop exceeds my expectations. It's fast, it's powerful, it's compact and great to travel with. “The screen is amazing and the keyboard too. the weight is so light, it's become my portable.” It’s durable, good keyboard, decent screen, and a good battery life. Plasticky build quality but holds up with my rough and tough handling. Is is surprisingly light. Keyboard is the best but it tales a bit of getting used to[…] Very light to carry and the carbon color gives an elegant finishing touch. Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions Product A 11 reviews, ~ 53 words per review
  20. 20. Results Discussion: Real Life Example The design is what caught my eye. Everything about this laptop is okay, except the battery life. Overall a great laptop with good display and build quality, solid performance and sleek design the only major concern is battery life. The touch screen is absolutely first rate […], and the back-lit keyboard has just the right feel. This is the best computer I've ever owned. […]. I love the backlit keyboard, the easily adjustable resolution and the long battery life. Pros: great screen, keyboard feels nice, best touchpad, very fast, extremely light, built durable Cons: battery life is less than competitors […]. It's really light weight yet really durable. I love the keyboard and mouse pad. 28  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions Product B 40 reviews, ~ 83 words per review
  21. 21. Conclusion and Future Work Work in Progress • Synonym computation: work in progress • Observed bias in sentiment, needs particular attention • Alternative when no/little reviews exist? Potential future applications • Offer a snapshot of product reviews to customers • Assist customers in finding similar items with enhanced feature(s) • Process seller satisfaction information/rating • Customer email processing, determine subject of request automatically 21  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  22. 22. References [1] Gensim https://radimrehurek.com/gensim/models/word2vec.html [2] GloVe http://nlp.stanford.edu/projects/glove/ [3] Wordnet https://wordnet.princeton.edu/ [4] nltk.stem http://www.nltk.org/api/nltk.stem.html [5] nltk.vader Paper: VADER: A Parsimonious Rule-based Model for Sentiment Analysis of Social Media Text, C.J. Hutto, Eric Gilbert Code: http://www.nltk.org/_modules/nltk/sentiment/vader.html 22  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  23. 23. Questions? 23
  24. 24. Back-Up Slides
  25. 25. Sentiment Capture with Vader 25 >>> sentence1 = ”This computer is a good deal." {'neg':0.0,'neu':0.58,'pos':0.42,'compound':0.4404} >>> sentence2 = “This computer is a very good deal.” {'neg':0.0,'neu':0.61,'pos':0.39,'compound':0.4927} >>> sentence3 = “This computer is a very good deal!!” {'neg':0.0,'neu':0.57,'pos':0.43,'compound':0.5827} >>> sentence4 = “This computer is a very good deal!! :-)” {'neg':0.0,'neu':0.441,'pos':0.559,'compound':0.7462} >>> sentence5 = “This computer is a VERY good deal!! :-)” {'neg':0.0,'neu':0.393,'pos':0.607,'compound':0.8287} >>> sentence6 = “This computer is a very bad deal!! :-(” {'neg':0.588,'neu':0.412,'pos':0.0,'compound':-0.7987} Adverb addition Punctuation addition Emoticon addition Case enhancement Inverse polarity  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  26. 26. Ratings vs Sentiment Analysis 26 Ratings (number of stars) Average sentiment from text review negativeneutralpositive  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  27. 27. Ratings vs Sentiment Analysis 27 Good reviews Bad reviews user bias = 𝑛 𝑝𝑜𝑠 − 𝑛(𝑛𝑒𝑔) 𝑛 𝑝𝑜𝑠 + 𝑛(𝑛𝑒𝑔) where: pos = number of prior good reviews neg = number of prior bad reviews  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions
  28. 28. Review Bias 28 • Where is subjectivity coming from? • Language bias / gender bias / etc. • Vader package biases due to development specificity? (remember: originally developed for social media) • Incentivized customers/reviewers • Why is it important to correct for it? • Filtering/sorting with ratings doesn’t work as well as expected • Possible options • Filter reviews with large bias • Weight results • Re-center the output of Vader to fit our definition of ’neutrality’ In short: Biases in both ratings and textual sentiment, both need attention  Business Motivation  Algorithm  Feature Space Computation  Sentiment Capture  Real-Life Examples  Future Work and Conclusions

×