Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TweetCred: Real-Time Credibility Assessment of 
 Content on Twitter @ Socinfo Conference 2014

1,382 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

TweetCred: Real-Time Credibility Assessment of 
 Content on Twitter @ Socinfo Conference 2014

  1. 1. Real-Time Credibility Assessment of Content on Twitter" Adi$ Gupta*, Ponnurangam Kumaraguru*, Carlos Cas$llo~ and Patrick Meier~ *Indraprastha Ins$tute of Informa$on Technology ~Qatar Compu$ng Research Ins$tute November 11, SocInfo’14
  2. 2. cerc.iiitd.ac.in TweetCred"
  3. 3. cerc.iiitd.ac.in Non-trustworthy Content" FAKE RUMORS 3 $
  4. 4. cerc.iiitd.ac.in 4
  5. 5. cerc.iiitd.ac.in Research Contributions" – Semi-­‐supervised ranking model for scoring tweets according to their credibility in real-­‐$me – TweetCred - Live deployment: 1,400+ TwiRer users - Credibility score computed for 7+ Million tweets - Evaluated TweetCred in terms of response $me, effec$veness and usability 5
  6. 6. cerc.iiitd.ac.in Methodology" 6
  7. 7. cerc.iiitd.ac.in Training Data" – 500 Tweets per event – Used CrowdFlower Event Tweets Users Boston Marathon Blasts (2013) 7,888,374 3,677,531 Typhoon Haiyan / Yolanda (2013) 671,918 368,269 Cyclone Phailin (2013) 76,136 34,776 Washington Navy yard shoo$ngs (2013) 484,609 257,682 Polar vortex cold wave (2014) 143,959 116,141 Oklahoma Tornadoes (2013) 809,154 542,049 Total 10,074,150 4,996,448 7
  8. 8. cerc.iiitd.ac.in Annotation" – Step 1 - R1. Contains informa$on about the event - R2. Is related to the event, but contains no informa$on - R3. Not related to the event - R4. Skip tweet *45% (class R1), 40% (class R2), and 15% (class R3) – Step 2 - C1. Definitely credible - C2. Seems credible - C3. Definitely incredible - C4. Skip tweet. *52% (class C1), 35% (class C2), and 13% (class C3) 8
  9. 9. cerc.iiitd.ac.in Credibility Modeling " 9 Feature set Features (45) Tweet meta-­‐data Number of seconds since the tweet; Source of tweet (mobile / web/ etc); Tweet contains geo-­‐coordinates Tweet content (simple) Number of characters; Number of words; Number of URLs; Number of hashtags; Number of unique characters; Presence of stock symbol; Presence of happy smiley; Presence of sad smiley; Tweet contains `via'; Presence of colon symbol Tweet content (linguis$c) Presence of swear words; Presence of nega$ve emo$on words; Presence of posi$ve emo$on words; Presence of pronouns; Men$on of self words in tweet (I; my; mine) Tweet author Number of followers; friends; $me since the user if on TwiRer; etc. Tweet network Number of retweets; Number of men$ons; Tweet is a reply; Tweet is a retweet Tweet links WOT score for the URL; Ra$o of likes / dislikes for a YouTube video
  10. 10. cerc.iiitd.ac.in Ranking Results" 10 AdaRank Coord. Ascent RankBoost SVM-­‐rank NDCG@25 0.6773 0.5358 0.6736 0.3951 NDCG@50 0.6861 0.5194 0.6825 0.4919 NDCG@75 0.6949 0.7521 0.689 0.6188 NDCG@100 0.6669 0.7607 0.6826 0.7219 Time (training) 35-­‐40 secs 1 min 35-­‐40 secs 9-­‐10 secs Time (tesNng) <1 sec <1 sec <1 sec <1 sec
  11. 11. cerc.iiitd.ac.in TweetCred Demo" 11
  12. 12. cerc.iiitd.ac.in Implementation"
  13. 13. cerc.iiitd.ac.in Feedback by Users" 13
  14. 14. cerc.iiitd.ac.in Usage Statistics" Date of launch of TweetCred 27 Apr, 2014 Credibility score seen by users total 5,438,115 Credibility score seen by users unique 4,540,618 Unique TwiRer users 1,127 Feedback was given for tweets 1,273 Unique users who gave feedback 263 14 *at the $me of paper submission
  15. 15. cerc.iiitd.ac.in Users of TweetCred" – Live deployment – Sample users: - Emergency departments - Firefighters - Journalists / news media - General users - Researchers (Requested API tokens) 15
  16. 16. cerc.iiitd.ac.in Evaluation" – Response Time – Usability Evalua$on – User Feedback 16
  17. 17. cerc.iiitd.ac.in Response Time" 17
  18. 18. cerc.iiitd.ac.in Usability Survey" – Online survey - Par$cipants: 67 – Ques$ons: - System Usability Scale (SUS) ques$ons - Demographic ques$ons – SUS Score: 70 - 74% of par$cipants -­‐> TweetCred easy to use - 81% of par$cipants -­‐> Would like to use TweetCred 18
  19. 19. cerc.iiitd.ac.in User Feedback" Observed 95% Conf. interval 19 Agreed with score 40.14 (36.73, 43.77) Disagreed with score 59.85 (55.68, 64.26) Disagreed: score should be higher 48.62 (44.86, 52.61) Disagreed: score should be lower 11.23 (9.82, 13.65) Disagreed by 1 point 8.71 (7.17, 10.50) Disagreed by 2 points 14.29 (12.29, 16.53) Disagreed by 3 points 12.8 (10.91, 14.92) Disagreed by 4 points 10.91 (9.17, 12.89) Disagreed by 5 points 6.52 (5.19, 8.08)
  20. 20. cerc.iiitd.ac.in Crisis Events Bias" 20
  21. 21. v
  22. 22. cerc.iiitd.ac.in Future Work" – Personaliza$on - Users trust contacts in their network more (users they follow, retweet or men$on) – Intersec$on between psychology literature on informa$on credibility & credibility of content on TwiRer 22
  23. 23. Thank you! hRp://twitdigest.iiitd.edu.in/TweetCred/ cerc.iiitd.ac.in

×