Finding High-Quality Content in Social Media

Eugene Agichtein Emory University Atlanta, USA Carlos Castillo Debora Donato Aris Gionis Yahoo! Research Barcelona, Spain Gilad Mishne Yahoo! S&A Sciences Santa Clara, CA, USA

User-generated content ≠ Traditional publishing E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Chris Anderson: “The Long Tail”. Hyperion, 2006. Frequency Traditional publishing User- generated Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Chris Anderson: “The Long Tail”. Hyperion, 2006. Quantity User- generated Traditional publishing Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

<!-- E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Quantity ? Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Chris Martin from Coldplay in The Rolling Stone, Fortieth Aniversary, July 2007. “We think it's all about quality over quantity now, because there's so much noise everywhere, there's no point in putting anything out unless it's fucking amazing.” Quantity Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Quantity User- A generated Traditional publishing Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Quantity User- F.A. generated Traditional publishing Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

--> E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

? Quantity Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

(Hard) problem Quantity Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Question + “Stars” Best answer Picked by votes -or- Picked by asker All answers + “Thumbs up” + “Thumbs down” E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

¼ questions want an opinion: informal polls ¾ questions seek for information or advice E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Q. Su, D. Pavlov, J.-H. Chow, W. C. Baker. “Internet-scale collection of human-reviewed data”.WWW'07. 17%-45% of answers were correct 65%-90% of questions had at least one correct answer E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

There are top contributors ... ... but they don't have all the answers E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Task: find high-quality items Quantity Quality E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Existing tools ● Link-based ranking methods ● Propagation of trust/distrust ● Automatic text analysis ● Usage mining ● ... E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Sources of information ● Content analysis ● Usage data (clicks) ● Community ratings E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Sources of information ● Content analysis (with errors) ● Clicks (with noise) ● Community ratings (sparse, with spam) E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Text analysis Clicks Community E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Text analysis Readability statistics Language modeling E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Text analysis Readability statistics Language modeling Punctuation density Capitalization errors Number of words + spacing density, sylablles per word,... E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

G. Mishne, D. Carmel, R. Lempel: “Blocking blog spam with language model disagreement”. AIRWeb'05 Text analysis Readability statistics Language modeling Language model disagreement Distributions of word n-grams and part-of-speech sequences when|how|why -- “to” -- verb “how to identify ...” when|how|why – verb – verb – pronoun – verb “how do I remove ...” E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Clicks If we know that a question is clicked 100 times, and another question is clicked 10,000 times ... ... we still know nothing E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Clicks Clicks Per-category average Per-category stdev. Question age E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Power laws E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

P. Jurczyk, E. Agichtein: “Discovering authorities in Q.A. communities by using link analysis” CIKM'07 Askers Answerers E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Community answers votes + votes - picks as best E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Community Degree-based metrics # answers given # answers received # votes + given # votes + received etc... E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Community Propagation-based metrics 1. Pagerank score 2. HITS hub score 3. HITS authority score Computed on each graph E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Text analysis Clicks Community Relations Training labels Learning E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Question quality High Medium Low High 15% Answer Medium 76% quality Low 9% 100% E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Question quality High Medium Low High 15% 8% Answer Medium 76% 74% quality Low 9% 18% 100% 100% E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Question quality High Medium Low High 41% 15% 8% Answer Medium 53% 76% 74% quality Low 6% 9% 18% 100% 100% 100% E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Question quality High Medium Low High 41% 15% 8% Answer Medium 53% 76% 74% quality Low 6% 9% 18% 100% 100% 100% Question quality and answer quality are not independent E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Relations: questions Answers to AQ questions asked Answers to the question being evaluated Questions asked Q Question being A A U evaluated Answers given AQ U Q A U Votes given VV User asking question A U Answerers of question being evaluated E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Relations: answers Answers to AQ questions asked Answer being QA Questions asked evaluated A U AQ Answers given Answerer VV Votes given Asker of question A being answered Other answers to the U Q A same question Question being A answered E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Text analysis Clicks Community Relations Training labels Learning E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

J. H. Friedman: “Stochastic gradient boosting”. Comp. Stat. Data. Anal., 38(4), 367-378, 2002. Text analysis Clicks Community Relations Labeled data: 6K questions Learning: 8K answers stochastic gradient boosted trees Evaluation: Precision, Recall (F1); Area under ROC curve E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Task: high-quality questions Precision Recall AUC N-grams (N) 65% 48% 0.52 N+ text analysis 76% 65% 0.65 N+ clicks 68% 57% 0.58 N+ relations 74% 65% 0.66 All 79% 77% 0.76 E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Task: high-quality answers Precision Recall AUC N-grams (N) 67% 86% 0.81 N + text analysis 71% 93% 0.88 N + clicks - - - N + relations 69% 85% 0.82 All 73% 91% 0.87 E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

In the paper ... ● Framework for quality estimation in social media ● Graph-based model of contributor relationships ● Details on the relative importance of (sets of) features E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

What did we learn? ● Human assessments for this task – ... have relatively low agreement ● Classifying questions/answers – ... is substantially different from document classification ● Look at orthogonal feature spaces E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Future work Text analysis Clicks Community Relations Relational learning Learning E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

Thank you! E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

E. Agichtein, C. Castillo, D. Donato, A. Gionis, G. Mishne: Finding High-Quality Content in Social Media. WSDM'08.

ROC curve: high-quality questions Best N-grams

ROC curve: high-quality answers Best N-grams

http://www.chato.cl	129
http://i-seoexpert.ru	73
http://feeds.feedburner.com	64
http://seo.wikimart.ru	44
http://chato.cl	40
http://www.slideshare.net	2
http://twitter.com	1
http://kanokim.blogspot.com	1

Finding High-Quality Content in Social Media

by Carlos Castillo on Feb 12, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

8 Embeds 354

Statistics

Finding High-Quality Content in Social Media — Presentation Transcript