The Truth About Sentiment & Natural Language Processing

The Truth About Sentiment & Natural Language Processing The Truth About Sentiment & Natural Language Processing Summary Introduction.... 2 Artificial Intelligence’s difficulties with sentiment.... 3 Human analysis is an obligatory step when analyzing web content.... 4 Current technological advances.... 7 Conclusion.... 10Synthesio - The Truth About Sentiment & Natural Language Processing - March 2011 1

The Truth About Sentiment & Natural Language ProcessingIntroductionThe web has made it possible for brands to discover what people are sayingabout their brands online, either in mainstream media like online newspapersand magazines, or on social media.Consumers now search for opinions online before, during, and after a purchase. The next step for brands is findingout whether people are talking positively or negatively about their brand, and why. Some online ratings provide anumber but not the reasoning behind it, and may only present half of the story.Numerous companies have been working on text mining for close to 30 years in some cases, thus sentiment analy-sis is not a new area but it has become a hot topic thanks to social media. Social media monitoring companies, aswell as PR practitioners, and digital marketers in general, have waged debate over whether sentiment should beanalyzed by man or machine. Synthesio currently uses human analysts for sentiment analysis but can add naturallanguage processing capacities on a case-by-case basis.Although technology is quickly advancing to catch up on its lag behind human analysis, as we advance toward whatis referred to as the singularity, it seems as though the best option is currently combining both machine and man.Synthesio - The Truth About Sentiment & Natural Language Processing - March 2011 2

The Truth About Sentiment & Natural Language ProcessingArtificial Intelligence’s difficulty with sentimentOne way that researchers have attempted to classify sentiment is by creatinga “sentiment lexicon”Sentiment is not analyzed via artificial intelligence, as some people may be tempted to think. Rather, it is analyzedvia a systematic process that involves the use of a sentiment lexicon. This lexicon assigns a degree of positivity ornegativity to a word by itself that is then used to give meaning to the entirety of the article. This is a way of analyz-ing sentiment, then, by considering a type of inherent positivity or negativity of each word that would be used bysomeone to talk about your business or products. For example, “happy” would be deemed a positive word, as wellas “like” and “love”. At the opposite end of the spectrum we can see words like “hate”, “dislike”, etc.There are two problems with this methodology, however. The first problem is that this assigning of positive andnegative sentiment evaluates a word without the context of what is around it. The dictionary is extremely limitedin the number of words that will always attach a positive or negative sentiment to an expression. The second prob-lem is that researchers may assign different degrees of positivity to a word. Particularly in the case of ambiguousexpressions, a researcher may be more inclined to note a word as more or less positive.Text categorization classifies articles by topic1Text categorization does not classically look at the various features mentioned within one article. Sentiment analy-sis has traditionally been performed using technology that evaluates an article at a global level. Within one text,however, the topic may not be linked to the descriptors. For example, take the sentence: “This film should be bril-liant. It sounds like a great plot, the actors are first grade, and the supporting cast is good as well, and Stallone isattempting to deliver a good performance. However, it can’t hold up.” The sentence should be positive, given thenumber of positive descriptors. It is only at the end that a human can identify the finality of the judgment that isoverall negative.The dictionaries used are developed through analysis of various factors, including sentiment polarity and degreesof positivity (“like” vs “dislike”; relatedness of topics..), identifying which parts of a document contain subjectivecontent (subjectivity detection and opinion Identification), identifying which parts of a document regard the samesubject before analyzing (joint topic-sentiment analysis), and determining the political orientation of a text (view-points and perspectives).Other non-factual information in the text can also be taken into account. For example, there are six “universal” emo-tions: anger, disgust, fear, happiness, sadness, and surprise that may be analyzed, as well as term presence, termfrequency, syntax, and negation.The majority of sentiment analysis literature has focused on text written inEnglishThis means that for the time being, most of the resources that have been developed for automatic sentiment analy-sis have been developed in English and for the English language. We looked at this with Seth Grimes, a textanalytics expert, later in this document in an exclusive interview, but there have traditionally been two types of so-lutions. One solution for multilingual resources has been using bilingual dictionaries to transfer the corpus, mean-ing finding parallels for all of the rules that were applied to the English texts. A second solution has been to applysentiment analysis to a translated version of the text, but accuracy rates may be questionable.Seth Grimes, expert in NLP There are companies that propose sentiment analysis in one language (typically English) while others propose an analysis in 10 different languages. Linguistic approaches (lexicons and dictionaries) may be used for several languages, but they have incomplete sentiment capabilities in most of them. Translat- ing linguistic content in French or Chinese, for example, can’t possibly offer the best results.1 Opinion mining and sentiment analysis, 20082 IdemSynthesio - The Truth About Sentiment & Natural Language Processing - March 2011 3

The Truth About Sentiment & Natural Language ProcessingHuman analysis is an obligatory step when analyzingweb contentMachines are capable of deciphering meaning from large amounts ofinformationAn advantage of having an automation of text analysis is that computers are able to work on large pieces of textthat are homogenous in form and written in one language much more quickly than a human ever could. Much asin the same way that macros in Excel accelerate the speed at which a human may advance, having algorithms treatinformation can accelerate sentiment analysis. The text must be written using a specific vocabulary, however, withvery little variability, in order to obtain high levels of accuracy.Collocations and complex syntactic patterns have been found to be useful indetecting subjectivity3Some technology experts have attempted to create syntactic relations within feature sets that are then tested ontext corpuses to “train” the software and allow for the detection of subjective expressions. This is done by creatingsyntactic templates that are run through a training corpus, generating extraction patterns for every time the tem-plates appear. For example, <x> pleased me should match any time the word “pleased” is present. There are certainlimitations to this technique, as the software will then search for specific syntactic expressions and not exact wordsentences. When analyzing for sentiment, then, this is only the first step in identifying if there is sentiment presentat all.Online reviews have had the most success with NLP online“Opinion-oriented information extraction” is advancing in identifying subjects in a text and their relationship withthe words around them that give them their context. Nouns in online reviews are particular in that they most likely– but not always – pertain to the product or service being reviewed. The context is similarly most likely – but notalways – the reviewer’s opinion of such product or service.Whereas other online media, like blog posts, may post various opinions throughout one post, with both positive andnegative sentiment attached accordingly, online reviews are one type of media that is typically focused on uniquelyone subject. A heuristic for NLP software has been to detect adjectives that are in the same sentence as the feature/product/service being evaluated. These can then be analyzed by manual or semi-manual rules or lexicons.Specialist in PR Relations KD Paine explains: “Computers can do a lot of things well, but differentiating between positive and negative comments in consumer generated media isn’t one of them. The problem with consumer generated media is that it is filled with irony, sarcasm and non-traditional ways of expressing sentiment. That’s why we recommend a hybrid solution. Let computers do the heavy lifting, and let humans provide the judgment.” –KD Paine3 Learning Extraction Patterns for Subjective Expression4 Opinion mining and sentiment analysis, 2008Synthesio - The Truth About Sentiment & Natural Language Processing - March 2011 4

The Truth About Sentiment & Natural Language ProcessingCurrent technological advancesTechnology is continually progressingMao and Lebanon are two researchers who proposed using “isotonic conditional random fields” to analyze senti-ment at sentence level5. They created mathematical calculations to determine sentiment, given that certain wordsmay be strongly positive or negative and thus affect the “local sentiment” positively or negatively. These could benew models for programming machines to determine sentiment within certain probabilities by also incorporatingthe author into the equation. Uses like these are interesting because human reviewers do not always agree, either. “I, for one, welcome our new computer overlords” – Ken Jennings, Jeopardy contestant Watson, a question-and-answer computer developed by IBM, made history on Jeopardy this year, an American game show renowned for its difficult question- and-answer format, by making an appearance against two top champions. Con- testants typically study volumes of encyclopedias in order to arrive at the final round, but IBM put their supercomputer Watson to the test – and he won. Programmed not only to buzz in according to the level of certainty he had for each question, Watson was trained to answer in the form of a question and decipher the complex language that goes into a game of Jeopardy. The category names are often puns, as well as the “answers” (which serve as questions)6. IBM proved that their technology has advanced to the point where it can intelligently parse language and weigh different parts of a phrase. Researchers scanned some 200 million pages of content — or the equivalent of about one million books — into the system but were unable to teach it to avoid all traps. During the practice session, one wrong answer led to a string of wrong answers, as the machine veered into a wrong direction.The web is comprised of many different types of media, both mainstream andsocialSome media online are more “fact-based,” such as newspapers or general news, while other are inherently more“opinion-based,” like Twitter, Facebook, and forums. Still other media may be one or the other, like blogs, all of whichmakes it difficult for automated sentiment analysis technology to differentiate between subjective and objectiveinformation. For example, if we look at the sentence “the battery lasts 2 hours” versus “the battery only lasts 2 hours,”there is a sentiment that is implied in the second sentence that is not in the first.Social media has also engendered new forms of expression via an “SMS-like” writing on social media that makestext analysis more complicated. Emoticons may or may not help, and slang is more commonly used in social media,along with misspellings and bad grammar, or poor syntax like missing or added characters. Take, for example, “ohmy gooooooood WTF did you see Biebur’s concert? It was aewsome! I lved it.” New forms of association and waysof depicting negative sentiment have also arisen, including ironic or sarcastic phrasing. “Another winner from thealmighty Microsoft,” for example, or most recently “Charlie Sheen is a winner.”Automated sentiment analysis cannot understand sentiment in the context ofyour business goalsOne factor that many automated proponents have struggled to respond to is analyzing text in the context of a busi-ness. For example, “Nike reports that profits rose” and “Adidas reports that profits rose” are both positive sentenceswhen evaluated with no context. If Nike is the firm listening to social media, however, the second phrase is suddenlynot as positive. The “goodness” or “badness” depends on whether the client is Nike or Adidas.5 Isotonic Conditional Random Fields and Local Sentiment Flow6 IBM and the Jeopardy! Challenge - Video - WiredSynthesio - The Truth About Sentiment & Natural Language Processing - March 2011 5

The Truth About Sentiment & Natural Language ProcessingBeyond looking at whether the information is positive or negative for a client, automated text analysis may extractinformation that the company already knows or does not wish to focus on. For example, the level at which machinescan decipher meaning is often limited to what brands already know. If a machine is told to analyze the top trendsaround a brand, it may include information that the brand already knows.Automated analysis is limited in analyzing sentiment for several topics withinan articleOnly now are certain technologies emerging that can analyze sentiment at a feature level, but in general automatedsentiment analysis technology has difficulty distinguishing sentiment between one topic and another, particularlyif more than one are mentioned in the same sentence.A blog post may be positive in the first sentence and negative in the second, or there may be one overall senti-ment for the blog post with positive and negative comments. “Much work on analyzing sentiment and opinionsin politically oriented text focuses on general attitudes expressed through texts that are not necessarily targetedat a particular issue or narrow subject.” A blogger, for example, may compare 2 products within the same post (ormore). Posts on a forum are often responses to earlier posts, and the lack of context makes it difficult for machinesto decipher whether the post is in agreement or disagreement.7 Opinion mining and sentiment analysis, 2008Synthesio - The Truth About Sentiment & Natural Language Processing - March 2011 6

The Truth About Sentiment & Natural Language ProcessingThe future of semantic technologyAn interview with Seth Grimes, an “Analytics visionary” “Watson”, the IBM computer won on the game show, Jeopardy, created a huge buzz around “his” technology. Why do you think there was so much buzz? Getting a computer to play Jeopardy was a great stunt. IBM made the technology do something that ev-eryone can understand. It was a “stunt,” however, because the ability to win Jeopardy is not in high demand in busi-ness or society. Nonetheless, Watson’s Jeopardy playing helps the non-technologist public understand the potentialand the reality of the technology.Question-answer systems are already out there, automating responses to business questions – for instance, forcontact-center support, customer inquiries, and online commerce – no requirement for a live person on the line.Right now Watson is focused on extracting factual information, but the technology could be working on sentimentvia a sentiment “annotator.” Then we won’t be limited to asking questions about facts. We’ll be able to ask aboutopinions and emotions.(An annotator analyzes text and marks it up with meaning, or attributes, features in the text. For example, a nameidentity annotator finds geographic locations and “marks them up”, finding semantic meaning. Annotating pattern-based entities can find addresses, identity location numbers by looking for patterns, and other annotators can markup other parts of the text.)How accurate can this technology be?Accuracy goals, and the amount of work you put into meeting them, should be decided in light of the businessproblem. Some problems will be solvable even with low levels of precision (e.g., positive versus negative sentimentclassification) while you might need higher precision for other applications. “Recall,” the ability to identify all ap-plicable cases, is also factored into accuracy measurements.My impression is that most sentiment tools that extract entities have out-of-the-box accuracy (without training)of something like 40-50% but can be “trained” (by having humans create marked-up samples or language rules orcorrect the tool) to reach above the 80% level. I saw one claim of 98% accuracy, which is laughable and ludicrous.The only way you can do this is by highly restricting the problem and tailoring the solution and being more lenienton what counts as accurate or not.It matters most, first that you identify that there is sentiment there at all, without even identifying if it is positiveor negative, and then passing materials on for human or machine classification. With machine filtering and humansanalyzing, for certain problems, you can yield high levels of accuracy. If you really want the machine to do every-thing, you need to do a lot more work or you will get much lower levels of accuracy over all, but again, decisionsshould be made based on business needs and also the nature of source materials.Let me add that I consider that while tools that analyze only at the message or document level may be accurate,the results they produce will also often be far less than useful. Think about it. It might be helpful if you’re running,say, a hotel group with 4,200 hotels, to know that (making up numbers) 77% of reviews were overall positive, 17%neutral, and 6% negative. Wouldn’t it be far more helpful to know, by hotel, opinion details? You want to knowwhen a reviewer found that room cleanliness and staff friendliness were exemplary but that noise was a problem.The details in a net positive review are not typically going to be all positive, and only by knowing sentiment at adetailed, “feature,” level can you reinforce what’s great and correct what’s not.By the way, let’s not overstate the accuracy of human sentiment analysis. The best study I’ve seen of accuracy wasdone at the University of Pittsburgh in 2005. While they found only 82% human agreement in annotating for senti-ment Results jumped to over 90% when they removed uncertain cases (when they subtracted cases where peoplesaid they weren’t sure).Synthesio - The Truth About Sentiment & Natural Language Processing - March 2011 7

The Truth About Sentiment & Natural Language ProcessingAre there certain online channels (among forums, blogs, Twitter, etc) that are easier to analyze using text mining asopposed to others?To really do it well you have to go to the feature level (to the individual item). You need strong natural languageprocessing (NLP) to do that right.Twitter is interesting because it is very hard to express more than one idea in a given tweet. Most tweets focus on asingle idea which, in theory, should make it easy to analyze. The problem is, people use a lot of slang and abbrevia-tion, which makes it difficult to analyze, as opposed to a blog or article. Also, a tweet is often part of a conversation.Very few tweets stand on their own; many including an article link or are responses to someone, for example. Oth-ers are part of multi-way conversations, and you very often need to understand the whole conversation to get thecontext. Most of the tools that are out there don’t do that; they don’t reach “through the tweet” to take into accountthe threaded nature of Twitter conversations. The more text there is, the easier it is to analyze, but at the same timethe shorter it is the more focused it’s going to be.But let’s move from ease of analysis to business value delivered.Applications like Synthesio’s get a lot of visibility because so many people use social media, but customer service isthe sentiment-analysis application that has probably delivered the clearest business benefits, the greatest businessvalue. Contact centers and surveys provide important data that is more focused than material out on the web, as-sociated with actual customers and transactions. You’ll get greater benefit tying customer feedback to social mediadata, rather than if you spend your funds broadly listening to people that are expressing opinion in a void, withoutcontext.There’s no denying the potential benefit in broad social-media monitoring and engagement, however. People willtell you what they like about your product (or don’t) and will post things that can be analyzed and shown to be indi-cators of their intent (to buy, to complain, or cancel their service, etc.) This information can be used to fix problems:the customer-service scenario. Answering a customer to make that person happy can turn them into a “net pro-moter,” and the information can be used to improve quality so the problems don’t happen to other people. Postedand analyzed information – beyond-polarity (positive/negative) intent signals – can also be used by companies toidentify and act on opportunities. This is engagement that not only reactively responds to particular commentsabout products and services. It’s engagement that proactively creates new and higher-value customers.What recent advances have you seen in sentiment analysis technology?The latest advances in analysis do go beyond “polarity” or “valence” (positive, negative, neutral), and I don’t justmean by rating sentiment on a scale from -10 to +10 to capture “intensity”: an advance, but we can do more. Forexample, you might look at sentiment in the terms of emotional categories such as “angry”, “sad”, or “happy,” abouta hotel service, for example. I’m sure we can all think of ways that automated understanding of emotional tone canbe useful in business contexts.Then there are the “intent signals” I was just discussing: sentiment as an indicator of plans, or actions.You’re going to get the most flexibility in creating business-suited categorizations via statistical approaches. Thatis, the analyst sets up categories that make sense and drags and drops documents into the different categories for“training” purposes. The machine uses statistical similarity measures to discover what the items in the categoryhave in common in order to automate classification.Further, the market is beginning to understand that influence is best measured by ability to affect business. Cer-tainly influence is correlated with the number of Facebook friends, Twitter followers, and retweets, but what shouldinterest far more is how those measures translate into inquiries, sales, and monetizable perceptions. A person isinfluential for real if he or she drives business transactions.And the market is understanding just how shallow many of the listening tools are – treating social media as a silo,completely unlinked to enterprise systems and actual business transactions, using simple keyword lists for senti-ment classification, and applying sentiment analysis only at message, article, or document level – and that theycan and should do better, including by joining the abilities of humans, who judge me and discern, and the power ofmachines, which are fast, work 24 hours per day, and can tap huge volumes of social, online, and enterprise informa-tion that are beyond human analysis regardless of cost.Synthesio - The Truth About Sentiment & Natural Language Processing - March 2011 8

The Truth About Sentiment & Natural Language ProcessingConclusionNo social media monitoring vendor would dare to pretend that technology can accurately (or even near-accurate-ly) assess sentiment on a specific topic. At subtopic-level (such as what we do at Synthesio), it is completely im-possible. However, NLP can at least help identify trends at a macro level such as hot topics or aggregate changesin sentiment over time. The theory is that even if the sentiment marking is inaccurate (even by an order of magni-tude), by tracking and trending it over time we can watch the pattern for changes because we are assuming thatthe level of inaccuracy will be consistent over time... However, there is no proof of this yet. About Synthesio Synthesio is a global, multi-lingual Social Media Monitoring and research company, utilizing a powerful hybrid of tech and human monitoring services to help Brands and Agencies collect and analyze consumer conversations online. The result is actionable analytics and insights that provide an accurate snapshot of a brand and help answer the ultimate questions – how are we really doing right now, and how can we make it better. Founded in 2006, the company has grown to include analysts who provide native-language monitoring and analytic services in over 30 lanuages worldwide. Brands such as Toyota, Microsoft, Sanofi, Accor, Orange and many other well-known companies turn to Synthesio for the data they need to engage with their markets, an- ticipate and prepare for emerging crisis situations, and prepare for new product or new campaign launches.Synthesio - The Truth About Sentiment & Natural Language Processing - March 2011 9

http://synthesio.com	482
http://mbatazzi.dev.synthesio.com	15
http://translate.googleusercontent.com	1

The Truth About Sentiment & Natural Language Processing

by Synthesio

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 498

Statistics

The Truth About Sentiment & Natural Language Processing Document Transcript