Successfully reported this slideshow.

Big Data & Sentiment Analysis


Published on

Big amount of information is available in textual form in databases or online sources, and for many enterprise functions (marketing, maintenance, finance, etc.) represents a huge opportunity to improve their business knowledge.

Sentiment analysis or opinion mining refers to the application of language processing to identify and extract subjective information in source materials. Generally speaking, sentiment analysis aims to determine the attitude of a speaker or a writer with respect to some topic or the overall contextual polarity of a document.

Published in: Business, Technology
  • Be the first to comment

Big Data & Sentiment Analysis

  1. 1. Sentiment Analysis Extract from various presentations: Bing Liu, Aditya Joshi, Aster Data … January 2012
  2. 2. Introduction Two main types of textual information: Facts and Opinions Most current text information processing methods work with factual information (e.g., web search, text mining) Sentiment analysis or opinion mining, computational study of opinions (sentiments, emotions) expressed in text Why opinion mining now? Mainly because of the Web huge volumes of opinionated text.
  3. 3. What is Sentiment Analysis? Identify the orientation of opinion in a piece of text (blogs, user comments, review websites, community websites, …), in others words determine if a sentence or a document expresses positive, negative, neutral sentiment towards some object? The movie was fabulous! [ Sentimental ] The movie stars Mr. X [ Factual ] The movie was horrible! [ Sentimental ]
  4. 4. SA at different levels His last movie was The movie was Great and interesting. The last movie was His police stopped The movie was interesting and very boring corruption great. fabulousdud. This one’s a Word-level SA Sentence-level SA Document-level SA fabulous interesting boring police (subj.) stopped (verb) corruption (obj.)
  5. 5. What is an Opinion? An opinion is a quintuple: (oj, fjk, soijkl, hi, tl) where – oj is a target object – fjk is a feature of the object oj – soijkl is the sentiment value of the opinion of the opinion holder hi on feature fjk of object oj at time tl – hi is an opinion holder – tl is the time when the opinion is expressed
  6. 6. Objective: structure the unstructured Objective: Given an opinionated document, – Discover all quintuples (oj, fjk, soijkl, hi, tl), • i.e., mine the five corresponding pieces of information in each quintuple With the quintuples, – Unstructured Text → Structured Data • Traditional data and visualization tools can be used to slice, dice and visualize the results in all kinds of ways • Enable qualitative and quantitative analysis With all quintuples, all kinds of analyses become possible
  7. 7. SA is not Just ONE Problem Track direct opinions: – document – sentence – feature level Compare opinions: different types of comparisons Detect opinion spam detection: fake reviews
  8. 8. Polarity Classifier First eliminate objective sentences, then use remaining sentences to classify document polarity (reduce noise)
  9. 9. Level of Analysis We can inquire about sentiment at various linguistic levels: Words – objective, positive, negative, neutral Clauses – “going out of my mind” Sentences – possibly multiple sentiments Documents
  10. 10. Words Adjectives – objective: red, metallic – positive: honest, important, mature, large, patient – negative: harmful, hypocritical, inefficient – subjective (but not positive or negative): curious, peculiar, odd, likely, probable Verbs – positive: praise, love – negative: blame, criticize – subjective: predict Nouns – positive: pleasure, enjoyment – negative: pain, criticism – subjective: prediction, feeling
  11. 11. Clauses Might flip word sentiment – “not good at all” – “not all good” Might express sentiment not in any word – “convinced my watch had stopped” – “got up and walked out”
  12. 12. Some Problems Which features to use? Words (unigrams), Phrases/n-grams, Sentences How to interpret features for sentiment detection? Bag of words (IR), Annotated lexicons (WordNet, SentiWordNet), Syntactic patterns, Paragraph structure Must consider other features due to… – Subtlety of sentiment expression • irony • expression of sentiment using neutral words – Domain/context dependence • words/phrases can mean different things in different contexts and domains – Effect of syntax on semantics
  13. 13. Some Applications Examples Review classification: Is a review positive or negative toward the movie? Product review mining: What features of the ThinkPad T43 do customers like/dislike? Tracking sentiments toward topics over time: Is anger ratcheting up or cooling down? Prediction (election outcomes, market trends): Will Obama or Republican candidate win? Etcetera
  14. 14. Aster Data position for Text Analysis Data Data Acquisition Acquisition Gather text from relevant sources (web crawling, document scanning, news feeds, Twitter feeds, …) Pre-Processing Pre-Processing Mining Mining Analytic Analytic Applications Applications Perform processing required to transform and store text data and information Apply data mining techniques to derive insights about stored information Leverage insights from text mining to provide information that improves decisions and processes (stemming, parsing, indexing, entity extraction, …) (statistical analysis, classification, natural language processing, …) (sentiment analysis, document management, fraud analysis, e-discovery, ...) Aster Data Fit Third-Party Tools Fit Aster Data Value: Massive scalability of text storage and processing, Functions for text processing, Flexibility to develop diverse custom analytics and incorporate third-party libraries