Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Online Reputation Monitoring in Twitter
from an Information Access Perspective
Damiano Spina
damiano@lsi.uned.es
@damiano1...
In Collaboration with
University of Amsterdam
●

Julio Gonzalo

●

Maarten de Rijke

●

Enrique Amigó

●

Edgar Meij (Yaho...
Arab Spring in Egypt, Jan 2011
Online Reputation Monitoring (ORM)
●

Reputation/public image is key for entities:
–

Companies, Organizations, Personalit...
Online Reputation Monitoring (ORM)
●

Reputation/public image is key for entities:
–

●

Companies, Organizations, Persona...
Online Reputation Monitoring (ORM)
●

Reputation/public image is key for entities:
–

●

Companies, Organizations, Persona...
Automatic Tools for ORM
Information Access (IA) techniques for
-Tracking Relevant Mentions
- Sentiment Analysis
- Discover...
Problem
●

Lack of standard benchmarks

for evaluation
Problem
●

Lack of standard benchmarks

for evaluation

●

It is hard for the analysts to know
how automatic tools will pe...
Goals
●

Formalize the Online Reputation Monitoring
problem as scientific challenges
Goals
●

Formalize the Online Reputation Monitoring
problem as scientific challenges
–

Build standard test collections

–...
Goals
●

Formalize the Online Reputation Monitoring
problem as scientific challenges
–
–

Organize International evaluatio...
Outline
●

Online Reputation Monitoring in Twitter
Outline
●

Online Reputation Monitoring in Twitter

●

Formalization from an Information Access perspective
–

Tasks Defin...
Outline
●

Online Reputation Monitoring in Twitter

●

Formalization from an Information Access perspective
–
–

●

Tasks ...
Outline
●

Online Reputation Monitoring in Twitter

●

Formalization from an Information Access perspective
–
–

●

Tasks ...
Online Reputation Monitoring in
Twitter
●

Analysts' daily work
–

Focus on a given entity of interest
Online Reputation Monitoring in
Twitter
●

Analysts' daily work
–

Focus on a given entity of interest

–

Recall oriented...
Online Reputation Monitoring in
Twitter
●

Analysts' daily work
–

Focus on a given entity of interest

–

Recall oriented...
Why Twitter?
●

●

(Bad) news spread earlier/faster/more unpredictable
than any other source in the Web
Most popular micro...
Why Twitter?
●

●

(Bad) news spread earlier/faster/more unpredictable
than any other source in the Web
Most popular micro...
Online Reputation Monitoring in
Twitter
Online Reputation Monitoring in
Twitter

?
Problem Formalization
ORM from an Information Access Perspective
Filtering Task
●

Is the tweet related to the entity of interest?

●

Example: Suzuki

related

unrelated
Filtering Task
●

Is the tweet related to the entity of interest?

●

Example: Suzuki

related
●

●

unrelated

Input: Ent...
Polarity for Reputation Task
●

●

Does the tweet affect negatively/positively to the reputation
of the entity?
Example: G...
Polarity for Reputation Task
●

●

●

●

Does the tweet affect negatively/positively to the reputation
of the entity?
Exam...
Topic Detection Task
●

What are the topics discussed in the tweets?
Topic Detection Task
●

●

●

What are the topics discussed in the tweets?

Input: Entity of interest (name + representati...
Topic Priority Task
●

What is the priority of each topics
in terms of reputational issues?

●

Input: Topics

●

Output: ...
Evaluation Framework
●

Reusable Test Collections

●

Evaluation Measures
–

Compare systems to annotated ground truth
Evaluation Framework
●

Reusable Test Collections

●

Evaluation Measures
–

●

Compare systems to annotated ground truth
...
RepLab: Evaluating Online Reputation
Management Systems
●

Organized as CLEF Labs
Cross-Language Evaluation Forum
RepLab: Evaluating Online Reputation
Management Systems
●

Organized as CLEF Labs
Cross-Language Evaluation Forum

●

2 ed...
Building Test Collections
Annotation Process
RepLab 2013 Annotation Tool
The RepLab 2013 Dataset
Evaluation
Why we Need All this Stuff?
●

To Evaluate Automatic Systems

●

To be able to answer the questions:
–

Which system perfo...
Automatic Solutions for ORM:
Filtering + Topic Detection
Evaluation: Filtering Task

Automatic systems can significantly help
when there is enough training data for each entity (7...
Evaluation: Filtering Task

Automatic systems can significantly help
when there is enough training data for each entity (7...
Evaluation: Topic Detection

Much more difficult than the Filtering Task
Evaluation: Topic Detection

Much more difficult than the Filtering Task
What performed better in RepLab?
UNED_ORM:
Cluste...
Topic Detection Approach
●

●

Tweet -> Set of Wikipedia Concepts/Articles

Clustering: Tweets sharing x% of identified
Wi...
Wikification: Commonness probability
WP concept c, n-gram q

q=“ferrari”
Wikification: Commonness probability
WP concept c, n-gram q

q=“ferrari”
Wikification: Commonness probability
WP concept c, n-gram q

q=“ferrari”

COMMONNESS "Ferrari S.p.A.", "ferrari" =

4
= 0....
Putting the Human in the Loop
Building Semi-Automatic Tools for
ORM
ORMA: A Semi-Automatic Tool for
Online Reputation Monitoring

J. Carrillo de Albornoz, E. Amigó, D. Spina, J. Gonzalo
ORMA...
Basic Filtering Approach
Basic Filtering Approach
Support Vector Machines (SVM)

Related/Unrelated

Training tweet

Test tweet
(unknown label)
Bag ...
Active Learning for Filtering

M. H. Peetz, D. Spina, M. de Rijke, J. Gonzalo
Towards an Active Learning System for Compan...
Active Learning for Filtering
●

Margin Sampling (confidence of the classifier)

●

After inspecting 2% of test data (30 o...
Active Learning for Filtering
●

Margin Sampling (confidence of the classifier)

●

After inspecting 2% of test data (30 o...
Conclusions
Conclusions
●

Online Reputation Monitoring in Twitter
Conclusions
●

Online Reputation Monitoring in Twitter

●

Formalized as Information Access Tasks
–

Reusable Test Collect...
Conclusions
●

Online Reputation Monitoring in Twitter

●

Formalized as Information Access Tasks
–
–

●

Reusable Test Co...
Conclusions
●

Online Reputation Monitoring in Twitter

●

Formalized as Information Access Tasks
–
–

●

Reusable Test Co...
Online Reputation Monitoring in Twitter
from an Information Access Persepective
Damiano Spina
damiano@lsi.uned.es
@damiano...
Online Reputation Monitoring in Twitter from an Information Access Perspective
Online Reputation Monitoring in Twitter from an Information Access Perspective
Upcoming SlideShare
Loading in …5
×

Online Reputation Monitoring in Twitter from an Information Access Perspective

663 views

Published on

Slides of my talk about the research I'm doing for my PhD thesis, given at Grasia, UCM (http://grasia.fdi.ucm.es/) on January, 2014

Published in: Social Media
  • Be the first to comment

  • Be the first to like this

Online Reputation Monitoring in Twitter from an Information Access Perspective

  1. 1. Online Reputation Monitoring in Twitter from an Information Access Perspective Damiano Spina damiano@lsi.uned.es @damiano10 UNED NLP & IR Group January 29, 2014 FdI UCM, Madrid, Spain
  2. 2. In Collaboration with University of Amsterdam ● Julio Gonzalo ● Maarten de Rijke ● Enrique Amigó ● Edgar Meij (Yahoo! Barcelona) ● Jorge Carrillo de Albornoz ● Mª Hendrike Peetz ● Irina Chugur ● Tamara Martín Llorente & Cuenca ● ● Ana Pitart ● LiMoSINe EU Project www.limosine-project.eu Vanessa Álvarez Adolfo Corujo
  3. 3. Arab Spring in Egypt, Jan 2011
  4. 4. Online Reputation Monitoring (ORM) ● Reputation/public image is key for entities: – Companies, Organizations, Personalities
  5. 5. Online Reputation Monitoring (ORM) ● Reputation/public image is key for entities: – ● Companies, Organizations, Personalities Social Media: – Necessity (and opportunity) of handling the public image of entities on the Web
  6. 6. Online Reputation Monitoring (ORM) ● Reputation/public image is key for entities: – ● Companies, Organizations, Personalities Social Media: – Necessity (and opportunity) of handling the public image of entities on the Web – Online Reputation Managers/Analysts ● Handle the reputation of an entity of interest (i.e., customer) ● Among other tasks, monitoring Social Media (manually!) – Early detection of issues/conversations/topics that may damage the reputation of the entity of interest
  7. 7. Automatic Tools for ORM Information Access (IA) techniques for -Tracking Relevant Mentions - Sentiment Analysis - Discover Keywords/Topics
  8. 8. Problem ● Lack of standard benchmarks for evaluation
  9. 9. Problem ● Lack of standard benchmarks for evaluation ● It is hard for the analysts to know how automatic tools will perform on their real data
  10. 10. Goals ● Formalize the Online Reputation Monitoring problem as scientific challenges
  11. 11. Goals ● Formalize the Online Reputation Monitoring problem as scientific challenges – Build standard test collections – Organize International evaluation campaigns – Bring together ORM and IA experts from Industrial and Academic communities
  12. 12. Goals ● Formalize the Online Reputation Monitoring problem as scientific challenges – – Organize International evaluation campaigns – ● Build standard test collections Bring together ORM and IA experts from Industrial and Academic communities Propose automatic solutions that may assist the reputation manager, reducing the effort in their daily work
  13. 13. Outline ● Online Reputation Monitoring in Twitter
  14. 14. Outline ● Online Reputation Monitoring in Twitter ● Formalization from an Information Access perspective – Tasks Definition – Evaluation Framework
  15. 15. Outline ● Online Reputation Monitoring in Twitter ● Formalization from an Information Access perspective – – ● Tasks Definition Evaluation Framework How much of the problem can be solved automatically? – Filtering – Topic Detection
  16. 16. Outline ● Online Reputation Monitoring in Twitter ● Formalization from an Information Access perspective – – ● Tasks Definition Evaluation Framework How much of the problem can be solved automatically? – – ● Filtering Topic Detection Putting the Human in the Loop: A Semi-Automatic ORM Assistant
  17. 17. Online Reputation Monitoring in Twitter ● Analysts' daily work – Focus on a given entity of interest
  18. 18. Online Reputation Monitoring in Twitter ● Analysts' daily work – Focus on a given entity of interest – Recall oriented ● They have to check all potential mentions! ● Also filter out not relevant mentions manually
  19. 19. Online Reputation Monitoring in Twitter ● Analysts' daily work – Focus on a given entity of interest – Recall oriented ● ● – They have to check all potential mentions! Also filter out not relevant mentions manually They make a summary to report to the client periodically Summary – ● What is being said about the entity in Twitter? What are the topics that may damage its reputation?
  20. 20. Why Twitter? ● ● (Bad) news spread earlier/faster/more unpredictable than any other source in the Web Most popular microblogging service – >230M monthly active users – 5k tweets published per second
  21. 21. Why Twitter? ● ● (Bad) news spread earlier/faster/more unpredictable than any other source in the Web Most popular microblogging service – – ● >230M monthly active users 5k tweets published per second Challenging for Information Access – Little context (only 140 characters) – Non-standard, SMS-like language
  22. 22. Online Reputation Monitoring in Twitter
  23. 23. Online Reputation Monitoring in Twitter ?
  24. 24. Problem Formalization ORM from an Information Access Perspective
  25. 25. Filtering Task ● Is the tweet related to the entity of interest? ● Example: Suzuki related unrelated
  26. 26. Filtering Task ● Is the tweet related to the entity of interest? ● Example: Suzuki related ● ● unrelated Input: Entity of interest (name + representative URL) + tweets that potentially mention the entity Output: Binary classification at tweet-level (relevant/not relevant)
  27. 27. Polarity for Reputation Task ● ● Does the tweet affect negatively/positively to the reputation of the entity? Example: Goldman Sachs
  28. 28. Polarity for Reputation Task ● ● ● ● Does the tweet affect negatively/positively to the reputation of the entity? Example: Goldman Sachs Input: Entity of interest (name + representative URL) + Stream of tweets that potentially mention the entity Output: Multi-class classification at tweet-level (positive/negative/neutral)
  29. 29. Topic Detection Task ● What are the topics discussed in the tweets?
  30. 30. Topic Detection Task ● ● ● What are the topics discussed in the tweets? Input: Entity of interest (name + representative URL) + Stream of tweets that mention the entity Output: Topics (Cluster of tweets)
  31. 31. Topic Priority Task ● What is the priority of each topics in terms of reputational issues? ● Input: Topics ● Output: Ranking of Topics – Alerts go first
  32. 32. Evaluation Framework ● Reusable Test Collections ● Evaluation Measures – Compare systems to annotated ground truth
  33. 33. Evaluation Framework ● Reusable Test Collections ● Evaluation Measures – ● Compare systems to annotated ground truth Evaluation Campaigns – Involve community – Compare different approaches
  34. 34. RepLab: Evaluating Online Reputation Management Systems ● Organized as CLEF Labs Cross-Language Evaluation Forum
  35. 35. RepLab: Evaluating Online Reputation Management Systems ● Organized as CLEF Labs Cross-Language Evaluation Forum ● 2 editions so far (+1 this year) – RepLab 2012 ● Filtering and Polarity for Reputation ● Topic Detection and Topic Priority as Monitoring Pilot Task – RepLab 2013 – RepLab 2014 (in progress) E. Amigó, J. Carrillo de Albornoz, I. Chugur, A. Corujo, J. Gonzalo, T. Martín, E. Meij, M. de Rijke, D. Spina Overview of RepLab 2013: Evaluating Online Reputation Monitoring Systems Proceedings of the Fourth International Conference of the CLEF initiative. 2013.
  36. 36. Building Test Collections
  37. 37. Annotation Process
  38. 38. RepLab 2013 Annotation Tool
  39. 39. The RepLab 2013 Dataset
  40. 40. Evaluation
  41. 41. Why we Need All this Stuff? ● To Evaluate Automatic Systems ● To be able to answer the questions: – Which system performs better? – Can tasks be solved automatically?
  42. 42. Automatic Solutions for ORM: Filtering + Topic Detection
  43. 43. Evaluation: Filtering Task Automatic systems can significantly help when there is enough training data for each entity (750 tweets)
  44. 44. Evaluation: Filtering Task Automatic systems can significantly help when there is enough training data for each entity (750 tweets) How? * Supervised learning POPSTAR (Univ. of Porto): Features: Twitter metadata, textual features, keyword similarity + external resources such as the entity’s homepage, Freebase and Wikipedia.
  45. 45. Evaluation: Topic Detection Much more difficult than the Filtering Task
  46. 46. Evaluation: Topic Detection Much more difficult than the Filtering Task What performed better in RepLab? UNED_ORM: Clustering of wikified tweets Tweets are represented as Bag of Wikipedia Concepts Tweet content linked to Wikipedia concepts based on intra-Wikipedia links
  47. 47. Topic Detection Approach ● ● Tweet -> Set of Wikipedia Concepts/Articles Clustering: Tweets sharing x% of identified Wikipedia articles are grouped together D. Spina, J. Carrillo de Albornoz, T. Martín, E. Amigó, J. Gonzalo, F. Giner UNED Online Reputation Monitoring Team at RepLab 2013 CLEF 2013 Labs and Workshops Notebook Papers. 2013.
  48. 48. Wikification: Commonness probability WP concept c, n-gram q q=“ferrari”
  49. 49. Wikification: Commonness probability WP concept c, n-gram q q=“ferrari”
  50. 50. Wikification: Commonness probability WP concept c, n-gram q q=“ferrari” COMMONNESS "Ferrari S.p.A.", "ferrari" = 4 = 0.57 (4 + 2 + 1)
  51. 51. Putting the Human in the Loop
  52. 52. Building Semi-Automatic Tools for ORM
  53. 53. ORMA: A Semi-Automatic Tool for Online Reputation Monitoring J. Carrillo de Albornoz, E. Amigó, D. Spina, J. Gonzalo ORMA: A Semi-Automatic Tool for Online Reputation Monitoring in Twitter 36th European Conference on Information Retrieval (ECIR). 2014.
  54. 54. Basic Filtering Approach
  55. 55. Basic Filtering Approach Support Vector Machines (SVM) Related/Unrelated Training tweet Test tweet (unknown label) Bag of Words: Tokenization + Preprocessing + Term Weighting Filtering Classifier 0.42 F: Similar to best RepLab
  56. 56. Active Learning for Filtering M. H. Peetz, D. Spina, M. de Rijke, J. Gonzalo Towards an Active Learning System for Company Name Disambiguation in Microblog Streams CLEF 2013 Labs and Workshops Notebook Papers. 2013.
  57. 57. Active Learning for Filtering ● Margin Sampling (confidence of the classifier) ● After inspecting 2% of test data (30 out of 1500 tweets): – 0.42 -> 0.52 F(R,S) (19.2% improvement) – Higher than the best RepLab contribution
  58. 58. Active Learning for Filtering ● Margin Sampling (confidence of the classifier) ● After inspecting 2% of test data (30 out of 1500 tweets): – – ● 0.42 -> 0.52 F(R,S) (19.2% improvement) Higher than the best RepLab contribution The cost of initial training data can be reduced substantially: – Effectiveness: 10% training + 10% test for feedback = 100% training
  59. 59. Conclusions
  60. 60. Conclusions ● Online Reputation Monitoring in Twitter
  61. 61. Conclusions ● Online Reputation Monitoring in Twitter ● Formalized as Information Access Tasks – Reusable Test Collections – Systematic Evaluation
  62. 62. Conclusions ● Online Reputation Monitoring in Twitter ● Formalized as Information Access Tasks – – ● Reusable Test Collections Systematic Evaluation Can tasks be solved automatically? – Filtering: Almost solved with enough training data (0.49F, 0.91 accuracy) – Topic: Systems are useful but not perfect
  63. 63. Conclusions ● Online Reputation Monitoring in Twitter ● Formalized as Information Access Tasks – – ● Reusable Test Collections Systematic Evaluation Can tasks be solved automatically? – – ● Filtering: Almost solved with enough training data (0.49F, 0.91 accuracy) Topic: Systems are useful but not perfect We need the expert in the loop – With a substantial reduction of manual effort
  64. 64. Online Reputation Monitoring in Twitter from an Information Access Persepective Damiano Spina damiano@lsi.uned.es @damiano10 UNED NLP & IR Group January 29, 2014 FdI UCM, Madrid, Spain

×