Wikimania 2009 - Yahoo Answers Community Moderation

Yahoo! Answers Community Moderation
Micah Alpern, Director of User Experience Yahoo! Search

Yahoo! Answers Ask, Answer, Discover
1.2 million Q&A per day
Experiential Knowledge

Activity

Answers is a marketplace
People want lots of good answers to their questions quickly.
This means wide distribution of your freshest content.
Higher visibility for abuse in Questions than Answers.
“ If you don't fix it right away, it's not worth fixing.” by getxsickphotography

Traditional Moderation Failed
As Answers grew traditional moderation methods failed to scale with the community.
Manual customer care systems:
Had slow response times
Treated all abuse reports the same
Had high false positive and false negative rates
Were high cost and scaled up with traffic

Machine learning? A team of PHDs worked on a black box abuse classifier.
Pre-filter of bad content to be reviewed by Customer Care.
Could only handle questions, not the answers
After a while that broke as well
60% of reports were fake abuse by users trying to report others.

Community Moderation
Deployed a new Community Moderation system
Empowered trusted Answers users to help moderate content by allowing their report abuse actions to automatically delete content.
System didn’t reveal reputation scores to users,
Encouraged them to report accurately so they could gain more community influence.
Illustration by Bryce Glass To address these challenges:

Reputation System

Answers Moderation Model Hide Content Item? Content Reporter Author

Answers Moderation Model REPORTER Abuse Suspicion Reputation REPORTER Community Investment Reputation User’s Abusive Reputation Confirmed Abuse Reporter Reputation Abuse Reporter Bootstrap Reputation Abuse Reporter Reputation CONTENT ITEM Abuse Reputation AUTHOR Overall Asker Reputation AUTHOR Overall Answerer Reputation AUTHOR Overall Asker Reputation Question Quality Reputation AUTHOR Overall Answerer Reputation Answer Quality Reputation Hide Content Item?

the story of a report an abuse report comes in We need to decided: Who do we believe? Do we hide the content? A question is asked.. Author Reporter

The content A question is asked..
What do we know about the content?
Have others reported it
Machined learned “Junk detector” score
Language specific
Requires tons of training data
!= a quality detector
CONTENT ITEM Abuse Reputation

the reporter What do we know about the reporter? Reporter CONTENT ITEM Abuse Reputation
Past reporting performance
and it’s outcome
08.17.08 4:41pm 08.20.08 6:23 pm 10.04.08 1:08 pm 11.06.08 7:03 am Past reporting behavior hidden not hidden hide overturned Outcome not hidden Confirmed Abuse Reporter Reputation Abuse Reporter Reputation For many reporters we don’t have enough feedback.

How do we solve the cold start problem?

Cold Start: Reporter Reputation Reporter
Suspicion: Implied Karma
Hard evidence
Examine signals across synonymous identities
Confirmed Abuse Reporter Reputation Abuse Reporter Reputation CONTENT ITEM Abuse Reputation Take this result and update the running reputation of the content reported. Reporter’s Abuse Suspicion Reputation Reporter’s Community Investment Reputation User’s Abusive Reputation Abuse Reporter Bootstrap Reputation

Compared to Illustration by Bryce Glass

Do we trust the author? How good are they at Asking questions? How good are they at Answering? Do they have a history of abuse? AUTHOR Overall Asker Reputation AUTHOR Overall Answerer Reputation AUTHOR Overall Asker Reputation Question Quality Reputation AUTHOR Overall Answerer Reputation Answer Quality Reputation Users Abusive Reputation

we make a decision an abuse report comes in A question is asked.. Reporter Author

decisions can be revised an abuse report comes in A question is asked.. Reporter Trust level is re-instated and temporary protection assigned. Trust level for Reporting is decreased. Author

Naive moderation
CraigsList
After 3 flags the post is deleted
Other systems: Ignore new user flags
In Answers:
Its harder to remove content from good contributors.
Easier for a even a new person to remove a spammer.

So how did we do?

Accuracy

Concussion
Reputation system helped Answers:
Scale & empower the community to mange abuse
Provide a better experience to Askers and Answerers

Questions? Bryce Glass Randy Farmer Ori Zaltzman http://buildingreputation.com / Micah Alpern [email_address] twitter:malpern

SPAM
“ We have other methods for identifying and dealing with spam, but as a very popular site we are a target for spammers so are continually upping our game to find new ways to identify them quicker and remove their content faster (though we shouldn’t share the methods we take). We recognize spam damages the experience for our users and so are committed to reducing spam on Answers as much as possible.”

Chat and Low quality content
“ You mention this a bit in the presentation, but generally we acknowledge there are different types of questions being asked and answered on the site and accept that currently we’re limited in how we are identifying and highlighting knowledge based Q&A vs Social Q&A. However, we are looking at ways to easier promote the type of content a user wants to see.”

Accuracy
“ focus is on experiential knowledge, practical help based on someone’s opinion or experience. Different type of knowledge than covered by wikipedia. We don’t want to publish content which is misleading, incorrect or harmful but use the “report abuse” system and our trusted users to balance any dubious advice.

http://buildingreputation.com	258
http://www.linkedin.com	15
http://www.slideshare.net	10
http://www.buildingreputation.com	5
http://www.linkedin-alpha.com	3
http://acontrer-2-md.linkedin.biz:8080	2
http://twitter.com	2
https://twitter.com	2
http://webcache.googleusercontent.com	1
http://www.buildingreputationsystems.com	1

Wikimania 2009 - Yahoo Answers Community Moderation

by Micah Alpern on Sep 01, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

10 Embeds 299

Statistics

Wikimania 2009 - Yahoo Answers Community Moderation — Presentation Transcript