Social text sentiment and tone analysis [aai 201] - (4160)

1,253 views
1,188 views

Published on

Published in: Technology
1 Comment
2 Likes
Statistics
Notes
No Downloads
Views
Total views
1,253
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
17
Comments
1
Likes
2
Embeds 0
No embeds

No notes for slide

Social text sentiment and tone analysis [aai 201] - (4160)

  1. 1. Social Text, Sentiment and ToneAnalysisAdvanced Analytics and InsightsRuben Pertusa Lopez, SolidQ, Business Intelligence DPE & MAP 2013 (rpertusa@solidq.com)Paco Gonzalez, SolidQ, Mentor (paco@solidq.com) April 10-12 | Chicago, IL
  2. 2. Please silencecell phones April 10-12 | Chicago, IL
  3. 3. Paco & Ruben• Conference Speakers• Based in Europe• Book Authors• PHD Candidate on Data Mining• Project Managers• Microsoft Certified Professionals 3
  4. 4. Goals• This session will help us understand how to analyze sentiment / tone using cutting edge Microsoft Technologies• This is NOT a research session on NLP (Natural Language Processing). Don’t be scared! ☺ 4
  5. 5. Agenda• Overview of Sentiment Analysis• Gathering and Storing Data• Sentiment Analysis Techniques• Business Analytics for Sentiment & Structured data, together 5
  6. 6. Overview of SentimentAnalysis April 10-12 | Chicago, IL
  7. 7. What is Sentiment?Feelings, Opinions, Emotions• Like• Dislike• Good• Bad• … 7
  8. 8. Some examples paulo @paulomors Barcelona Beating Milan 4-0 and a gallon of Coke, best day of the year ☺ ☺ αυѕтιη ✰ @AustinJohnsto A 2 hour delay is like when a restaurant only has Pepsi products. Itll work but ill still be very disappointed. Trace @trace_haf Drinking more Coke than water every day does no good for you health. ☺? ? DONALD @donald150 My budget decides whether coke or pepsi, always buy the cheaper 8
  9. 9. Not only Twitter! The Walking Dead Season 3 (3300 customer reviews) Awesome season opener! October 15, 2012 By K. Erwin After sticking through season two, which was alot of looking for a little girl and standing around on a farm waiting for something to happen, I hoped that theyd pick up the pace a little with season three. The premiere doesnt dissapoint! The Walking Dead Season 3 Mid-Premiere (72K Likes, 7K comments) Timothy Berteau You guys are crazy. One boring episode and you think the show sucks, did you forget how awesome the two episodes before it were? Not every episode is going to be action packed. Season 1 and 2 had some very boring moments as well. 18 February at 00:58 via mobile · 5 likes 9
  10. 10. DEMOLooking at some Twitter Sentiment 10
  11. 11. Surrounded by opinions 12 Tb 21 Pb Hadoop day cluster 7 Pb Month (search queries info) 1 Tb 7 Tb Tweets Data day day 75 Mi 4B Scores Graph day edg/day Millions of opinions 11
  12. 12. Valuable Information for our BusinessQuestions• Is this review positive or negative?• Do this Twitter User like or dislike my new show?• What are they saying about our company or services?• How are Facebook User’s Attitudes about the next election? Is that the only valuable info? 12
  13. 13. What is Sentiment Analysis?Text Categorization (Opinion Mining)• Positive / Negative / Neutral• 1, 2, 3, 4, 5 Stars• For / Against Sentiment Analysis Text IN OUT Category Techniques Photos Videos, etc. 13
  14. 14. DEMOTwitter for Analytics 14
  15. 15. Gathering and Storing Data April 10-12 | Chicago, IL
  16. 16. Gathering data from sourcesPublic/Private APIs• Limitations• Privacy• Format & Structure of source data• UpdatesWebcrawlers? Call the cops! 16
  17. 17. How may it look?JSON Example (1 tweet) 17
  18. 18. Storing the gathered dataWhy?• Valuable data!• Store now. Figure out laterHow?• Structured (Relational Database – SQL Server)• Semi-Structured (Hadoop Cluster – Microsoft HDInsight) 18
  19. 19. Doctor, We need some helpThe 4 V’s Volume Velocity Variety Variability 19
  20. 20. Doctor, We need some helpThe 4 V’s Volume Velocity Variety Variability 20
  21. 21. Start building our system Extract Extract Transform Load Load 21
  22. 22. DEMOLet’s gather some Twitter data ☺ 22
  23. 23. Sentiment AnalysisTechniques April 10-12 | Chicago, IL
  24. 24. Understanding the problem Data != Sentiment DataIdentify relevant parts• Nouns• Adjectives/Adverbs• VerbsDrop anything else 24
  25. 25. Sentiment Analysis TechniquesTechniques• Natural Language Processing• Basic Statistics • Clustering • Fuzzy Components • Classification • EstimationToolsHDInsight, SSIS, SSAS DM, FullText Search (Semantic Search) 25
  26. 26. Dictionaries Tone Index Dictionary Positive 10 amazing Positive 9 awesome Positive 8 bestMatch words Positive 7 excellent Positive 6 exciting“Coke is the best & coolest drink Positive 5 greaton the market” Positive Positive 4 3 good rocks Positive 2 cool Positive 1 :) Neutral 0 Negative -1 :( Negative -2 poor Negative -3 bad Negative -4 criticizedSome Other Database Example Negative -5 attacked Negative -6 humiliatedSentiWordNet Negative -7 suckshttp://sentiwordnet.isti.cnr.it/ Negative Negative -8 -9 terrible horrible Negative -10 worthless 26
  27. 27. A closer look at the example “Coke is the best & coolest drink on the market” Tone Index Dictionary Positive 10 amazing Fuzzy Match Positive 9 awesome Positive 8 best Positive 7 excellent “Best” matches “best” =+ 8 Positive 6 exciting Positive 5 great “coolest” matches “cool” =+ 2 Positive Positive 4 3 good rocks Positive 2 cool Total = + 10 Positive 1 :) Neutral 0 ☺ Negative Negative -1 -2 :( poor 27
  28. 28. DEMOGive me your sentiment! 28
  29. 29. Sentiment Analysis ChallengesPhrase level polarity• “The US fears happy caffeine consumers”.Context polarity• Context: The date that Steve Jobs died.• Opinion 1: “This is the worst day of my life”.• Opinion 2: “The world is a better place today”.Irony, Sarcasm, Irregular forms, Pragmatic information… 29
  30. 30. It looks like… Extract Extract Transform Load Load SA Techniques ☺ 30
  31. 31. Combining SentimentAnalysis with StructuredData April 10-12 | Chicago, IL
  32. 32. MORE Valuable Information for ourBusinessQuestions• Is this positive/negative review affecting my sales?• Have we increased our sales because of positive tweets? Can we learn anything from opinions to improve our products?• How are people responding to this campaign? 32
  33. 33. Bring our cubes into a new worldNew ways of analysisBetter decisions• Drive next campaign, new wave of products, etc.Estimate future figures ☺ + 33
  34. 34. DEMOOur cubes have feelings ☺ 34
  35. 35. Final picture Extract Extract Transform Load Load SA Techniques ☺ + 35
  36. 36. Brief SummaryDuring this session..• Business Insights from social opinions• Sentiment Analysis loves Big Data ☺• Microsoft Technologies can help us with some SA Techniques• There is a huge competitive advantage by using Sentiment Analysis 36
  37. 37. QUESTIONSPositive / Negative / Neutral ones ☺ 37
  38. 38. Contact us!Rubén Pertusa López (rpertusa@solidq.com)@rpertusaData Platform Engineer, SolidQMicrosoft Active Professional 2013Paco González (paco@solidq.com)@pacosqlMentor, SolidQ 38
  39. 39. Win a Microsoft Surface Pro!Complete an online SESSION EVALUATIONto be entered into the draw.Draw closes April 12, 11:59pm CTWinners will be announced on the PASS BAConference website and on Twitter.Go to passbaconference.com/evals or follow the QR code link displayed onsession signage throughout the conference venue.Your feedback is important and valuable. All feedback will be used to improveand select sessions for future events.
  40. 40. Platinum Sponsor Thank you!Diamond Sponsor April 10-12, Chicago, IL

×