Information Credibility on Twitter

Information Credibility on TwitterCarlos Castillo1, Marcelo Mendoza2,3, Bárbara Poblete2,4 1. Yahoo! Research Barcelona 2. Yahoo! Research Latin America 3. Universidad Federico Santa María, Chile 4. Dept. Cs. de la Computación, Universidad de Chile WWW 2011 Hyderabad, India.

http://xkcd.com/723/

A “small” earthquake in Chile• Sat Feb 27, 2010. 03:34 local time• 7th largest recorded in history – 8.8 Mw, 90 s, 500-600 casualties – Haiti: 7.7 Mw, 60 s, 90k-300k casualties• Communications – Almost impossible for 2-3 hours – First video images 6-7 hours after quake3 Information Credibility on Twitter

Chileans love Twitter• Prominent role for communications – online and offline• All public figures tweet• Well integrated with traditional media – Since ~1yr before earthquake (elections)6 Information Credibility on Twitter

Twitter helped, but ...• Large majority of tweets were very helpful• Some tweets were not – False tsunami warnings – False reports of looting – ...9 Information Credibility on Twitter

Manual classification results

Can we automaticallydetect false tweets?

Can we automaticallydetect false tweets? Of course not.

Information credibility• Perceived quality• Made of multiple dimensions• Examples: – Same article written by male/female author – Same headline in news/twitter – ...14 Information Credibility on Twitter

Supervised classification• Events (tweet sets) from TwitterMonitor – [Mathioudakis & Koudas 2010]• Labels from Amazons Mechanical Turk• Decision trees15 Information Credibility on Twitter

Labeling: News or Chat• 383 events from TwitterMonitor.net [Mathioudakis & 30% 36% Koudas]• 7 evaluators per event• >=5 agreement 35% News Chat Unsure16 Information Credibility on Twitter

Spreading a specific news/event -OR- Conversation or comments among friends.

Labeling: Credible or Not 19%• 747 events automatically classified as news 41% 9%• 7 evaluators per event• >=5 agreement 32% Almost Likely to Almost Unsure certainly be false certainly true false18 Information Credibility on Twitter

Almost certainly true Likely to be true Likely to be falseAlmost certainly false

Examples credible/not-credibleUsers believed these Users did not believe these• US construction plunges 10% • Lennon lyrics auctioned for 1.2m (true)• Tropical storm in the gulf • Suspicious vehicle causes NYC• Markets down on Portugal, scare (false alarm: no bomb Greece was found, they were gas canisters)• Yankees Bob Sheppard dies • Free video calling app for Android (spam) • Warnings on BART murder trial veredict (partially true: no widespread riots)20 Information Credibility on Twitter

Credible tweets for users tend to ...• Have a URL• Dont have exclamation marks• Express a negative sentiment• Are re-posted by prolific users• Are re-posted by well-connected users21 Information Credibility on Twitter

A = credible B = non-credible

4 features related to the share of the top-1:URL, user mention, author, hashtag

Summary• Detected past events on twitter• Took a sample of 10 tweets per event• Asked users if they believe the tweets• Built an automatic classifier• Got 86% accuracy on balanced classes24 Information Credibility on Twitter

Issues• Online classification• Other types of misinformation: – Spam • Commercial • Non-commercial (rogue cyberactivism) – Astroturfing • “Grassroot” political campaigns• ...25 Information Credibility on Twitter

Thank you!C. Castillo, M. Mendoza and B.Poblete: chato@yahoo-inc.com“Information Credibility on Twitter”

Information Credibility on Twitter

by Carlos Castillo on Apr 11, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Information Credibility on Twitter — Presentation Transcript