Credibility Ranking of Tweets during High Impact Events


Published on

Twitter has evolved from being a conversation or opinion sharing medium among friends into a platform to share and disseminate information about current events. Events in the real world create a corresponding spur of posts (tweets) on Twitter. Not all content posted on Twitter is trustworthy or useful in providing information about the event. In this paper, we analyzed the credibility of information in tweets corresponding to fourteen high impact news events of 2011 around the globe. From the data we analyzed, on average 30% of total tweets posted about an event contained situational information about the event while 14% was spam. Only 17% of the total tweets posted about the event contained situational awareness information that was credible. Using regression analysis, we identified the important con- tent and sourced based features, which can predict the credibility of information in a tweet. Prominent content based features were number of unique characters, swear words, pronouns, and emoticons in a tweet, and user based features like the number of followers and length of username. We adopted a supervised machine learning and relevance feedback approach using the above features, to rank tweets according to their credibility score. The performance of our ranking algorithm significantly enhanced when we applied re-ranking strategy. Results show that extraction of credible information from Twitter can be automated with high confidence.

Published in: Technology, Travel
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Credibility Ranking of Tweets during High Impact Events

  1. 1. Credibility  Ranking  of  Tweets   during  High  Impact  Events   Adi$  Gupta  &  Ponnurangam  Kumaraguru   PSOSM@WWW   April  17,  2012  
  2. 2. Problem  MoOvaOon                    IIIT-­‐Delhi   2  
  3. 3. Problem  MoOvaOon   Informa$on   Opinion   Spam                    IIIT-­‐Delhi   3  
  4. 4. Outline   •  •  •  •  •  •  •      Research  statement   Architecture   Data  collecOon   Analysis   Results   ImplementaOon   Future  direcOon                    IIIT-­‐Delhi   4  
  5. 5. Research  Statement   •  IdenOfy  parameters  that  affect  credibility  of   content  on  TwiTer   •  Develop  a  semi-­‐automated  algorithm  to   assess  credibility  of  tweets                    IIIT-­‐Delhi   5  
  6. 6. Terminology   TWEET:  A  status  (140   chars)   HASHTAG   RETWEET   USER   PROFILE   URL   USER  NAME  @screen_name   FOLLOWERS   Tweets   @-­‐MENTIONS                    IIIT-­‐Delhi   6  
  7. 7. Credibility   •  “The  quality  of  being  trusted  and  believed  in.”     •  In  this  research   –  Assess  the  credibility  of  the  informaOon  in  the   content  of  a  tweet  (message)  by  a  user  on  TwiTer.     –   A  tweet  is  said  to  contain  credible  informaOon   about  a  news  event,  if  you  trust  or  believe  that   informaOon  in  the  tweet  to  be  correct  /  true.                    IIIT-­‐Delhi   7  
  8. 8. News  on  TwiTer   News  on   Twi7er   Topics  on   Twi7er   News   Events   E.g.  #Irene,   #Libyacrisis     Credible   Informa$on   Chit-­‐Chat   Fake  news  /  Rumors  / Spam  /  Personal   Opinions   E.g.   #nothingwrongwith,   #goodmorningtwiTer             Non-­‐ Credible   Informa$on          IIIT-­‐Delhi   8  
  9. 9. Our  ContribuOons   •  30%  of  tweets  provide  informaOon  (17%  credible  informaOon)   and  14%  was  spam     •  Linear  logisOc  regression     –  Content  based:  #unique  characters,  swear  words,   pronouns  and  emoOcons   –  User  based:  #followers  and  length  of  username     •  Present  automated  algorithm  (supervised  ML  and  relevance   feedback)  to  assess  credibility  in  tweets                    IIIT-­‐Delhi   9  
  10. 10. Data  StaOsOcs   Total  tweets 35,748,136 Total  unique  users 6,877,320 Tweets  with  URLs 4,973,457 Number  of  singleton  tweets 22,481,898 Number  of  re-­‐tweets  /  replies 13,266,238 Start  date 12th  July,  2011 End  date 30th  August,  2011 •  High  impact  events:   –  Greater  25K  tweets   –  More  than  48  hours  in  trending  topics                    IIIT-­‐Delhi   10  
  11. 11. Data  StaOsOcs                    IIIT-­‐Delhi   11  
  12. 12. Data  StaOsOcs   Events 542,685 #ukriots, #londonri- ots, #prayforlondon Libya Crisis 389,506 libya, tripoli Earthquake in Virginia 277,604 #earthquake, Earth- quake in SF JanLokPal Bill Agitation 182,692 Anna Hazare, #jan- lokpal, #anna Apple CEO Steve Jobs resigns 158,816 Steve Jobs, Tim Cook, Apple CEO US Downgrading 148,047 S&P, AAA to AA Hurricane Irene 90,237 Hurricane Irene, Tropical Storm Irene Google acquires Motorola Mobility 68,527 Google, Motorola Mobility News of the World Scandal 67,602 Rupert Murdoch, #murdoch Abercrombie & Fitch stocks drop 54,763 Abercrombie & Fitch, A&F Muppets Bert and Ernie were gay 52,401 Bert and Ernie Indiana State Fair Tragedy 49,924 Indiana State Fair Mumbai Blast, 2011 32,156 #mumbaiblast, Dadar, #needhelp New Facebook Messenger   Trending Topics UK Riots   Tweets 28,206 Facebook Messenger                    IIIT-­‐Delhi   12  
  13. 13. Architecture                    IIIT-­‐Delhi   13  
  14. 14. Human  AnnotaOon   •  For  each  tweet:   –  Tweet  contains  informaOon  about  the  event.  Rate  the  credibility  of   informaOon  present:   •  Definitely  Credible   •  Seems  Credible   •  Definitely  Incredible   •  I  can’t  Decide   –  Tweet  is  related  to  the  news  event,  but  contains  no  informaOon   –  Tweet  is  not  related  to  news  event   –  Skip  tweet     •  Each  tweet  annotated  by  3  people   •  Inter-­‐annotator  agreement  (Cronbach  Alpha)  =  0.748     •  30%  of  tweets  provide  informaOon  (17%  credible  informaOon)  and   14%  was  spam                    IIIT-­‐Delhi   14  
  15. 15. ANALYSIS                    IIIT-­‐Delhi   15  
  16. 16. Feature  Sets   Message based features Source based features Length of the tweet Registration age of the user Number of words Number of unique characters Number of statuses Number of hashtags Number of followers Number of retweets Number of swear language words Number of friends Number of positive sentiment words Number of negative sentiment words Is a verified account Tweet is a retweet Length of description Number of special symbols [$, !] Length of screen name Number of emoticons [:-), :-(] Tweet is a reply Has URL Number of @- mentions Ratio of followers to followees Number of retweets Time lapse since the query Source based features Has URL Registration age of the user Number of URLs Use of URL shortener service Number of statuses Message based features Number of followers Length of the tweet Number of words                    IIIT-­‐Delhi   16  
  17. 17. PRF   •  PRF  (Pseudo  Relevance  Feedback)     –  Extract  k  ranked  documents  and  then  re-­‐rank   those  documents  according  to  a  defined  score     –  Re-­‐ranking  based  on  ‘context’  of  the  event     –  Top  n  unigrams  based  on  BM25  metric                    IIIT-­‐Delhi   17  
  18. 18. Algorithm                    IIIT-­‐Delhi   18  
  19. 19. EvaluaOon  Metric   EvaluaOon  Metric:  NDCG  (Normalized  Discounted  CumulaOve  Gain)           NDCG  is  the  standard  metric  used  to  evaluate  “graded”  results                    IIIT-­‐Delhi   19  
  20. 20. Ranking  Results   •  Tweet  and  user  based  features  contribute  in  determining  the  credibility  –  it   maTers  “what  you  post  and  who  you  are”     •  Context  based  (PRF)  ranking  greatly  enhances  the  performance  (upto  .74   NDCG)                    IIIT-­‐Delhi   20  
  21. 21. Web-­‐portal  ImplementaOon                    IIIT-­‐Delhi   21  
  22. 22. LimitaOons  &  Future  Work   •  Human  input  required   –  Need  to  develop  self  learning  (completely   automated)  soluOons   •  Analyze  events  with  a  greater  temporal   variaOon   •  Understanding  user’s  perspecOve  of  credibility   of  content  on  TwiTer                    IIIT-­‐Delhi   22  
  23. 23. Challenges   •  •  •  •      Large  volume  of  data  being  generated   Real-­‐Ome  soluOons  needed   Only  140  characters   Informal  language                    IIIT-­‐Delhi   23  
  24. 24. Acknowledgements   •  All  members  of  our  research  group   •  Dept.  of  InformaOon  Technology,  Government   of  India                    IIIT-­‐Delhi   24  
  25. 25. References   •  C.  CasOllo,  M.  Mendoza,  and  B.  Poblete.  InformaOon  Credibility  on  TwiTer.   In  WWW,  pages  675–684,  2011.   •  J.  Chen,  R.  Nairn,  L.  Nelson,  M.  Bernstein,  and  E.  Chi.  Short  and  tweet:   experiments  on  recommending  content  from  informaOon  streams.  CHI   ’10,  pages  1185–1194,  2010.   •  J.  Ratkiewicz,  M.  Conover,  M.  Meiss,  B.  Gon  ̧calves,  S.  PaOl,  A.  Flammini,   and  F.  Menczer.  Truthy:  mapping  the  spread  of  astroturf  in  microblog   streams.  WWW  ’11.   •  S.  E.  Robertson,  S.  Walker,  and  M.  Beaulieu.  Okapi  at  trec-­‐7:  automaOc  ad   hoc,  filtering,  vlc  and  interacOve  track.  IN,  1999.   •  T.  Sakaki,  M.  Okazaki,  and  Y.  Matsuo.  Earthquake  shakes  twiTer  users:   real-­‐Ome  event  detecOon  by  social  sensors.  WWW  ’10,  2010.   •  S.  Verma,  S.  Vieweg,  W.  J.  Corvey,  L.  Palen,  J.  H.  MarOn,  M.  Palmer,  A.   Schram,  and  K.  M.  Anderson.  Nlp  to  the  rescue?  extracOng  “situaOonal   awareness”  tweets  during  mass  emergency.  ICWSM,  2011.                    IIIT-­‐Delhi   25  
  26. 26. QuesOons?                    IIIT-­‐Delhi   26  
  27. 27.             Thank  You!  
  28. 28. For  any  further  informaOon,  please  write  to   28