Your SlideShare is downloading. ×
0
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
  Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Toward Formal Reasoning with Epistemic Policies about Information Quality in the Twittersphere

339

Published on

Presentation by VIStology Inc. at Fusion 2011 Conference, Chicago, IL. July 2011. …

Presentation by VIStology Inc. at Fusion 2011 Conference, Chicago, IL. July 2011.

Automatically evaluating the reliability and credibility of messages on Twitter.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
339
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Toward Formal Reasoning with Epistemic Policies About Information Quality in the Twittersphere<br />Brian Ulicny <br />VIStology, Inc. <br />bulicny@vistology.com<br />Mieczyslaw Kokar <br />Northeastern University and VIStology, Inc.<br />kokar@coe.neu.edu<br />VIStology, Inc - Fusion 2011<br />1<br />
  • 2. Arab Spring Uprisings 2011<br />2<br />VIStology, Inc - Fusion 2011<br />
  • 3. Situation Awareness (?):Al Jazeera’s Twitter Monitor<br />3<br />VIStology, Inc - Fusion 2011<br />
  • 4. Situation Awareness:Attention Spikes from Twitter<br />4<br />VIStology, Inc - Fusion 2011<br />
  • 5. Situation Awareness: Flu Trends from Social Media<br />Detecting influenza outbreaks by<br />analyzing Twitter messages<br />AronCulotta<br />arXiv:1007.4748v1 [cs.IR] 27 Jul 2010<br />5<br />VIStology, Inc - Fusion 2011<br />
  • 6. Twitter as Open Source Intel<br />6<br />VIStology, Inc - Fusion 2011<br />
  • 7. 7<br />VIStology, Inc - Fusion 2011<br />Confidence = <Reliability, Credibility><br />
  • 8. Problem Statement<br />How can we assess not only the volume of tweets per time period<br />And the frequency of terms they contain<br />But the reliability, credibility & confidence of the information they convey<br />In a potentially adversarial situation?<br />8<br />VIStology, Inc - Fusion 2011<br />
  • 9. Naïve STANAG 2022 for Twitter<br />Reliability = F: Cannot Be Judged<br />All “sources not used in the past”<br />Credibility = 1: Confirmed by Other Sources<br />More than two string identical tweets?<br />Or Credibility = 3, Possibly True <br />Because Sources not Independent<br />Because Path between all sources in Twitter graph <br />9<br />VIStology, Inc - Fusion 2011<br />
  • 10. Need<br />Tractable Way to Calculate:<br />Twitter Source Reliability<br />Twitter Content Credibility<br />Twitter Source Independence<br />Where <br />Entire Twitter graph contains 105 Million Users<br />As of April, 2010<br />55 Million Tweets per Day<br />3 Billion Requests per day to Twitter API<br />10<br />VIStology, Inc - Fusion 2011<br />
  • 11. The Argument from Google<br />There are too many Twitter sources to evaluate their reliability directly.<br />However, Google has shown that there is great value in using eigenvector centrality (PageRank) as a proxy for reliability.<br />Therefore, we assume that a PageRank-like metric correlates with Reliability because<br />(1) We assume that people do not pass along information they believe to be unreliable<br />(2) Eigenvector centrality/retweet influence, unlike simple indegree centrality, is difficult to fake.<br />11<br />VIStology, Inc - Fusion 2011<br />
  • 12. Not Every Twitter User is Real<br />CENTCOM<br />Operation Earnest Voice<br />12<br />VIStology, Inc - Fusion 2011<br />
  • 13. TunkRank as Reliability<br />Influence(X) = Expected number of people who will read a tweet that X tweets, including all retweets of that tweet. For simplicity, we assume that, if a person reads the same message twice (because of retweets), both readings count.<br />If X is a member of Followers(Y), then there is a 1/||Following(X)|| probability that X will read a tweet posted by Y, where Following(X) is the set of people that X follows.<br />If X reads a tweet from Y, there’s a constant probability p that X will retweet it.<br />D. Tunkelang. 2009. A Twitter Analog to PageRank. <br />http://thenoisychannel.com/2009/01/13/a-twitter-analog-to-pagerank/<br />13<br />VIStology, Inc - Fusion 2011<br />
  • 14. TunkRank as Reliability <br />TunkRankvs Indegree Centrality (log scale)<br />Mapping TunkRank to STANAG 2022 Reliability <br />14<br />VIStology, Inc - Fusion 2011<br />
  • 15. Unreliability Indicators<br />If X retweets a message, e.g:<br />RT @Whitehouse Zombie uprising in Scranton<br />And there is no corresponding original tweet<br />Then X is E: Unreliable.<br />If X tweets a message with the same URL (shortened or dereferenced) <br />But different content<br />More than twice<br />Then X is D: Not Usually Reliable.<br />(On the other hand: Verification: Reliability )<br />15<br />VIStology, Inc - Fusion 2011<br />
  • 16. Source Independence<br />There is a path connecting (nearly) every user in the Twitter graph.<br />This does not mean that there is no source independence in Twitter.<br />We count any sources as independent if they originate the message, and <br />The shortest path between them is ≥ 4.<br />In T.H. dataset, 4/20 tweets cite same NY Times URL via 3 shortened URLs. <br />So, not independent.<br />Other news sources: 2 cite Guardian, 1 BBC, 1 Der Spiegel, 1 WaPo, 1 Times of London<br />No explicit Retweets<br />No Implicit Retweets<br />=> 16 originating sources<br />Compute distance between remaining sources<br />16<br />VIStology, Inc - Fusion 2011<br />
  • 17. Sameness of Content<br />String identical tweets are not independent. Implicit retweets<br />@BWJones: Tim Hetherington, photographer and 'Restrepo' co-director, killed in Misrata, Libya http://nyti.ms/dIm29T4/20/2011 6:16:25 PM<br />@Frieze_magazine: Tim Hetherington, photographer and 'Restrepo' co-director, killed in Misrata, Libya http://nyti.ms/dIm29T4/20/2011 7:01:30 PM<br />Custom Regexes to handle dead/alive<br />Tweet =~ (<subject> .* (dead|died|killed|notalive|RIP) ) &&<br />Tweet !~ (<subject> .* (not (dead|died|killed)) => Dead<br />Tim Hetherington, Restrepo director has been killed in Misurata<br />OR: Tweet =~ (<subject>.*(alive|(not (killed|dead|died)) &&<br />Tweet !~ (<subject> .* (not alive|RIP) => Alive<br />E.g. C H still alive. (true positive) Wish T H were still alive (false positive)<br />Misses: C H in serious condition ( |= alive)<br />>2x P vs not-P: Confirmed P; not-P: Improbable; > 1.5x P vs not-P: Probably True P, Doubtful not-P; ~same P, not-P: Possibly-true P, Possibly-true not-P<br />435 Tweets report C H dead; vs 7 C H alive: Confirmed: C H Dead; Improbable: C H not Dead.<br />17<br />VIStology, Inc - Fusion 2011<br />
  • 18. Recap: Algorithm<br />Identify set of Tweets by Search API on name<br />Classify into Dead/Alive content<br />Calculate TunkRank on Users<br />Discount false retweeters<br />Calculate Source Independence<br />Group same media URLs; retweets, implicit retweets<br />Calculate distance between sources for joint network two hops out for each source.<br />@NYTImesPhoto: An attack in Misurata, Libya today killed the photographer Tim Hetherington. 4/20/2011 7:11:15 PM<br />TunkRank: 99th percentile; > 5 independent sources assert T H died; 0 alive<br /><A:Completely Reliable, 1:Confirmed by Other Sources><br />@Cmovila: Sad news Tim Hetherington died in Misrata now when covering the front line. 4/20/2011 4:39:57 PM<br />TunkRank: 0th Percentile; > 5 Independent sources assert T H died; 0 alive<br /><E: Unreliable; 1:Confirmed by Other Sources><br />T H Alive: 5: Improbable><br />18<br />VIStology, Inc - Fusion 2011<br />
  • 19. Notional Architecture<br />VIStology, Inc - Fusion 2011<br />19<br />Twitter <br />Search API<br />Tweet to RDF<br />Conversion<br />Message Classifier<br />Twitter<br />API<br />BaseVISor<br />Inference<br />Engine<br />TunkRank<br />API<br />Distance<br />Calculator<br />Tweets Augmented with STANAG 2022<br />Assessments<br />
  • 20. Conclusions<br />Treating all Tweets as equally legitimate OK in non-adversarial, high volume situations.<br />As OSINT, Tweets need to be evaluated according to the STANAG 2022 rubric<br />We have outlined tractable ways to calculate reliability (TunkRank), credibility (sameness of content) and source (in)dependence.<br />By converting Tweets to RDF, we can reason about them formally with a formal reasoner (BaseVISor)<br />Future work: Do large scale demonstration showing efficacy in distinguishing low-confidence death rumors from high-confidence death notices on Twitter<br />20<br />VIStology, Inc - Fusion 2011<br />
  • 21. Questions?<br />21<br />VIStology, Inc - Fusion 2011<br />

×