1.
Moderating for Fractions
of a Penny per Comment
@justinisaf
The Communal Group
SPRINT San Francisco 2014

2.
Ohai!
Principal at
The Communal Group

3.
Ohai!
I used to manage the HuffPost Community
We pre-moderated 450,000 comments a day
with 28 moderators
*This is rough initial data, scraped
from the huffingtonpost.com
Please do not share

4.
Disclaimer
I do not claim to have thought of everything, for every situation, nor
do I intend to. These things worked in at least one situation and are
not guaranteed to work everywhere.
Individual results may vary.
There is no substitute for experience and timely, intelligent decision
making. This discussion is designed to explore several aspects of
community building which, when used collectively, could produce
positive results.
:)

5.
Why
It’s a super unsexy cost center
But it’s essential
So let’s make it cheap and best

6.
Why
Industry standard is about $0.25 per
comment for low-scale and $0.04 at scale
I’m going to show you how to do it for $0.04
at low-scale and $0.005 at scale

7.
Terms
Active vs Reactive moderation - are you actively looking for
good/bad content or are you waiting for “flags” before you react
Pre- and Post- moderation - are you looking at content before or
after it’s posted

8.
Moderation Types
Housekeeping - Maintaining baseline standards, designed to avoid
offensive content, legal headaches and rapid decline to “no, you’re a
d00die head” by deleting “bad” comments.
Curating - Attempting to improve the level of discussion, dialog and
debate by promoting “good” comments.
Mediating - Actively engaging with members to work through
disagreements and fights without necessarily deleting or promoting
content.
(Tip: Watch the first sentence, last sentence and first sentence of the last paragraph)

9.
Baselines - Performance
Professional moderators working at @ 85-90% accuracy:
Housekeeping - 400 - 600 comments per hour
Curating - 250 - 350 comments per hour
Mediating - 20-30ish comments per hour
(Tip: Moderators get faster and better with time, but don’t hire below 250)

10.
Baselines - Salaries
Remote moderators start at (plus benefits):
Housekeeping - $12 an hour/$28K per year ($0.03 cpc)
Curating - $15 an hour/$30-32K per year ($0.06 cpc)
Mediating - Priceless
(Tip: Partner with organizations that place wounded vets or people with disabilities)

11.
Baselines - Outsourcing
$25-$35 per hour or
$0.17-$0.30 cents per comment
0-10% Account maintenance fees
50-400 comments per hour
(Tip: Outsourcing and Offshoring are VERY different)

12.
Embrace the Machines
People are very good at nuance, but bad at volume
Machines are very good at volume, but bad at nuance

13.
Embrace the Machines
Machine assisted Human moderation lets machines
take care of “sure” tasks, and passes “unsure” tasks to
humans
Human assisted Machine moderation puts humans
adjusting and auditing machines, without actively
moderating themselves

14.
Embrace the Machines
Machine assisted Human moderation can give you
approximately a 4:1 leverage on your humans
(getting us to about $0.0075 cpc)
More importantly, it frees up time and resources for more Curation
which is more valuable for the community and more interesting for
moderators

15.
Design, for Moderation
Highlighting trigger words can give moderators a 50% increase in speed
Scoop some of the shit out of your brain and observe that "being talked to" was not what was described at all. Condescending sexist
pestering is not "being talked to," and no one here complained about normal civil non-sexist "being talked to" at all. Making cracks
about a woman only watching sports to please her man, demanding she answer quizzes to prove her "true fan" status, and just
generally expecting women to drop everything they're doing and submit to whatever the men demanding their attention want is
not "being talked to." The fact that you think these things are normal polite conversation that a person would have to be antisocial
to want to avoid means you undoubtedly treat women exactly this way and see no problem with it.
You aren't confused, you're an asshole.

16.
Design, for Moderation
Highlighting trigger words can give moderators a 50% increase in speed
Scoop some of the shit out of your brain and observe that "being talked to" was not what was described at all. Condescending sexist
pestering is not "being talked to," and no one here complained about normal civil non-sexist "being talked to" at all. Making cracks
about a woman only watching sports to please her man, demanding she answer quizzes to prove her "true fan" status, and just
generally expecting women to drop everything they're doing and submit to whatever the men demanding their attention want is
not "being talked to." The fact that you think these things are normal polite conversation that a person would have to be antisocial
to want to avoid means you undoubtedly treat women exactly this way and see no problem with it.
You aren't confused, you're an asshole.

17.
Design, for Moderation
Hotkeys for publish and delete (~5% increase in speed)
Size 14 Times New Roman (or Arial) (~7.5% increase over other fonts)
40 characters per line (~3% increase over 80 characters per line)
User/story context at the top left of the comment (increases accuracy at speed)
Pagination, not infinite scroll (gives a psychological “end” to an endless job)
“Special” comments at random times (breaks monotony)
Automatic audits and reporting (increases accuracy and consistency)
Special shout out to Mountain Dew and 6 hour shifts

18.
Design, for Moderation
All in, a well designed moderation tool will give you
another ~200 comments an hour
(bringing us down to $0.005 for machine assisted human mods)

19.
Utilization
We’ve assumed 100% utilization so far
(that’s 3.5million comments per year)
If you have that “problem”, congrats
So let’s look at this in a real example

20.
Case Study - Salon.com
Media site dealing with politics
~75,000 comments per month
Volume peaks during daytime hours

21.
Case Study - Salon.com
Started with 2 part time mods, reactive
housekeeping moderation
Touched ~3-4% of the incoming volume
Up to 24 hour SLA
Cost per comment reviewed was ~$1

22.
Case Study - Salon.com
Now 1 full time moderator during peak hours
for real-time, active, post-moderation and
reviews overnight content during lulls
Plus outsourced, reactive, post-mod with a
30 minute SLA on per-comment pricing

23.
Case Study - Salon.com
Now they touch ~80% of incoming content
Within 5 minutes during the day and 30
minutes during nights and weekends
At a cost per comment reviewed of ~$0.04

24.
Case Study - Salon.com
~2500% increase in coverage,
for 96% reduction in cost per comment
to get better coverage

25.
Next Steps - Nascent Community
Fewer than 5K comments per month
Hands on by the owner/CM
Consider volunteer “leaders” to help you
Mediate, set the tone and identify promotable
content

26.
Next Steps - Small Community
5-50K comments per month
Outsource 24/7 flagged comment review
with a 30 minute SLA (assuming ~10% flag rate)
Continue hands on Mediation and step up
Curation

27.
Next Steps - Medium Community
50-200K comments per month
Outsource flagged comments
Hire in-house moderator(s) for peak hour
housekeeping (6 hour shifts 6 days a week)
CM/Owner kept for strategic interventions

28.
Next Steps - Large Community
200K+ comments per month
Hire 3 full time/2 part time moderators in-house
Turn on the machines
Begin managing through design

29.
Cat

30.
Cat

31.
Done
Thanks!
Questions?
@justinisaf / justin@communl.com / 415-789-ohai

32.
Moderation Types - Housekeeping
Most common type of moderation.
Often necessary. Often not sufficient.
Never ending, escalating battle.
Do:
First sentence, last sentence and the first sentence of
the last paragraph
Aim for a 15 minute SLA (no more than 60 minutes)
Leverage mutli-faceted filters
Ensure you have robust flagging
Lean on the side of do-removal
Be consistent
Dont:
Leave large time gaps in coverage (like weekends)
Blanket delete terms
Assume your users are unbiased when they flag

33.
Moderation Types - Curating
Most likely to increase the quality of discussion.
Often overlooked because the Risk Of Ignoring is low.
Automation tends to be harder.
Do:
Aim for a 1 hour SLA (no more than 24 hours)
Enlist users
Add a “good” flag
Lean on the side of don’t-promote
Dont:
Set an expectation of promotion
Favor certain users because it’s easy
Assume your users are unbiased when they flag
Create a “grind”

34.
Moderation Types - Mediation
Most likely to increase the quantity of discussion.
Often seen in early stage communities.
Hard to measure, but high long term return.
Do:
Be human
Be human
Remember that it’s just the internet - it’s not that bad
Be human
Dont:
Take sides
Quote the guidelines
Let an argument go for more than 3 replies
Let problem users become entrenched