Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Justin Isaf - How To Reduce Your Moderation Costs

5,081 views

Published on

Justin Isaf, formerly of the Huffington Post and now of Communl, will take you through a specific, detailed, process of reducing your moderation costs to fractions of pennies - and save millions in the process

Published in: Business
  • Be the first to comment

Justin Isaf - How To Reduce Your Moderation Costs

  1. 1. Moderating for Fractions of a Penny per Comment @justinisaf The Communal Group SPRINT San Francisco 2014
  2. 2. Ohai! Principal at The Communal Group
  3. 3. Ohai! I used to manage the HuffPost Community We pre-moderated 450,000 comments a day with 28 moderators *This is rough initial data, scraped from the huffingtonpost.com Please do not share
  4. 4. Disclaimer I do not claim to have thought of everything, for every situation, nor do I intend to. These things worked in at least one situation and are not guaranteed to work everywhere. Individual results may vary. There is no substitute for experience and timely, intelligent decision making. This discussion is designed to explore several aspects of community building which, when used collectively, could produce positive results. :)
  5. 5. Why It’s a super unsexy cost center But it’s essential So let’s make it cheap and best
  6. 6. Why Industry standard is about $0.25 per comment for low-scale and $0.04 at scale I’m going to show you how to do it for $0.04 at low-scale and $0.005 at scale
  7. 7. Terms Active vs Reactive moderation - are you actively looking for good/bad content or are you waiting for “flags” before you react Pre- and Post- moderation - are you looking at content before or after it’s posted
  8. 8. Moderation Types Housekeeping - Maintaining baseline standards, designed to avoid offensive content, legal headaches and rapid decline to “no, you’re a d00die head” by deleting “bad” comments. Curating - Attempting to improve the level of discussion, dialog and debate by promoting “good” comments. Mediating - Actively engaging with members to work through disagreements and fights without necessarily deleting or promoting content. (Tip: Watch the first sentence, last sentence and first sentence of the last paragraph)
  9. 9. Baselines - Performance Professional moderators working at @ 85-90% accuracy: Housekeeping - 400 - 600 comments per hour Curating - 250 - 350 comments per hour Mediating - 20-30ish comments per hour (Tip: Moderators get faster and better with time, but don’t hire below 250)
  10. 10. Baselines - Salaries Remote moderators start at (plus benefits): Housekeeping - $12 an hour/$28K per year ($0.03 cpc) Curating - $15 an hour/$30-32K per year ($0.06 cpc) Mediating - Priceless (Tip: Partner with organizations that place wounded vets or people with disabilities)
  11. 11. Baselines - Outsourcing $25-$35 per hour or $0.17-$0.30 cents per comment 0-10% Account maintenance fees 50-400 comments per hour (Tip: Outsourcing and Offshoring are VERY different)
  12. 12. Embrace the Machines People are very good at nuance, but bad at volume Machines are very good at volume, but bad at nuance
  13. 13. Embrace the Machines Machine assisted Human moderation lets machines take care of “sure” tasks, and passes “unsure” tasks to humans Human assisted Machine moderation puts humans adjusting and auditing machines, without actively moderating themselves
  14. 14. Embrace the Machines Machine assisted Human moderation can give you approximately a 4:1 leverage on your humans (getting us to about $0.0075 cpc) More importantly, it frees up time and resources for more Curation which is more valuable for the community and more interesting for moderators
  15. 15. Design, for Moderation Highlighting trigger words can give moderators a 50% increase in speed Scoop some of the shit out of your brain and observe that "being talked to" was not what was described at all. Condescending sexist pestering is not "being talked to," and no one here complained about normal civil non-sexist "being talked to" at all. Making cracks about a woman only watching sports to please her man, demanding she answer quizzes to prove her "true fan" status, and just generally expecting women to drop everything they're doing and submit to whatever the men demanding their attention want is not "being talked to." The fact that you think these things are normal polite conversation that a person would have to be antisocial to want to avoid means you undoubtedly treat women exactly this way and see no problem with it. You aren't confused, you're an asshole.
  16. 16. Design, for Moderation Highlighting trigger words can give moderators a 50% increase in speed Scoop some of the shit out of your brain and observe that "being talked to" was not what was described at all. Condescending sexist pestering is not "being talked to," and no one here complained about normal civil non-sexist "being talked to" at all. Making cracks about a woman only watching sports to please her man, demanding she answer quizzes to prove her "true fan" status, and just generally expecting women to drop everything they're doing and submit to whatever the men demanding their attention want is not "being talked to." The fact that you think these things are normal polite conversation that a person would have to be antisocial to want to avoid means you undoubtedly treat women exactly this way and see no problem with it. You aren't confused, you're an asshole.
  17. 17. Design, for Moderation Hotkeys for publish and delete (~5% increase in speed) Size 14 Times New Roman (or Arial) (~7.5% increase over other fonts) 40 characters per line (~3% increase over 80 characters per line) User/story context at the top left of the comment (increases accuracy at speed) Pagination, not infinite scroll (gives a psychological “end” to an endless job) “Special” comments at random times (breaks monotony) Automatic audits and reporting (increases accuracy and consistency) Special shout out to Mountain Dew and 6 hour shifts
  18. 18. Design, for Moderation All in, a well designed moderation tool will give you another ~200 comments an hour (bringing us down to $0.005 for machine assisted human mods)
  19. 19. Utilization We’ve assumed 100% utilization so far (that’s 3.5million comments per year) If you have that “problem”, congrats So let’s look at this in a real example
  20. 20. Case Study - Salon.com Media site dealing with politics ~75,000 comments per month Volume peaks during daytime hours
  21. 21. Case Study - Salon.com Started with 2 part time mods, reactive housekeeping moderation Touched ~3-4% of the incoming volume Up to 24 hour SLA Cost per comment reviewed was ~$1
  22. 22. Case Study - Salon.com Now 1 full time moderator during peak hours for real-time, active, post-moderation and reviews overnight content during lulls Plus outsourced, reactive, post-mod with a 30 minute SLA on per-comment pricing
  23. 23. Case Study - Salon.com Now they touch ~80% of incoming content Within 5 minutes during the day and 30 minutes during nights and weekends At a cost per comment reviewed of ~$0.04
  24. 24. Case Study - Salon.com ~2500% increase in coverage, for 96% reduction in cost per comment to get better coverage
  25. 25. Next Steps - Nascent Community Fewer than 5K comments per month Hands on by the owner/CM Consider volunteer “leaders” to help you Mediate, set the tone and identify promotable content
  26. 26. Next Steps - Small Community 5-50K comments per month Outsource 24/7 flagged comment review with a 30 minute SLA (assuming ~10% flag rate) Continue hands on Mediation and step up Curation
  27. 27. Next Steps - Medium Community 50-200K comments per month Outsource flagged comments Hire in-house moderator(s) for peak hour housekeeping (6 hour shifts 6 days a week) CM/Owner kept for strategic interventions
  28. 28. Next Steps - Large Community 200K+ comments per month Hire 3 full time/2 part time moderators in-house Turn on the machines Begin managing through design
  29. 29. Cat
  30. 30. Cat
  31. 31. Done Thanks! Questions? @justinisaf / justin@communl.com / 415-789-ohai
  32. 32. Moderation Types - Housekeeping Most common type of moderation. Often necessary. Often not sufficient. Never ending, escalating battle. Do: First sentence, last sentence and the first sentence of the last paragraph Aim for a 15 minute SLA (no more than 60 minutes) Leverage mutli-faceted filters Ensure you have robust flagging Lean on the side of do-removal Be consistent Dont: Leave large time gaps in coverage (like weekends) Blanket delete terms Assume your users are unbiased when they flag
  33. 33. Moderation Types - Curating Most likely to increase the quality of discussion. Often overlooked because the Risk Of Ignoring is low. Automation tends to be harder. Do: Aim for a 1 hour SLA (no more than 24 hours) Enlist users Add a “good” flag Lean on the side of don’t-promote Dont: Set an expectation of promotion Favor certain users because it’s easy Assume your users are unbiased when they flag Create a “grind”
  34. 34. Moderation Types - Mediation Most likely to increase the quantity of discussion. Often seen in early stage communities. Hard to measure, but high long term return. Do: Be human Be human Remember that it’s just the internet - it’s not that bad Be human Dont: Take sides Quote the guidelines Let an argument go for more than 3 replies Let problem users become entrenched

×