Designing Your Reputation System

Slide 1: Designing your reputation system (in 15 10! easy steps) Bryce Glass bryce@yahoo-inc.com IA Summit 2008 Miami, Florida

Slide 2: What will we cover? 1. First, let’s define our terms 2. Then we’ll review a number of common reputation patterns 3. Finally, I’ll pose a number of questions designed to assist you in employing these patterns properly. DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS With 15 questions, I may be over-reaching. But once I’d submitted the talk title to the Summit, I felt committed! Though some questions will be given a highly-abbreviated treatment, the set really does represent every issue I think you’d have to deal with when designing a reputation system. I will try to publish a complete and annotated version of the questions online after the summit.

Slide 3: What’s my reputation? • Interaction design lead for Yahoo! Reputation Platform • Consulted with ~10 different Yahoo! properties over 14 months ‣ To varying degrees. With mixed results • Benefited from user-testing of concepts DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS [Ask if anyone would like a little but of clarity around what a ‘platform’ is…] I won’t be discussing the platform specifically—these are best practices derived from working with customers. In fact, I am not actively working on the platform anymore, so—for me—this material serves as a nice capstone to a year’s efforts. Tho’ I’ll show examples from various Y! properties, not all of them utilize the platform. (In fact, most don’t.) So please don’t mistake this talk for a generalization about “The Yahoo way” of thinking about reputation. It’s -a- yahoo way. [Warn the audience that you’ll use the term ‘property’ alot, and probably interchangeably with Product, Site and Community.] I’ll attempt to summarize findings from user-testing, but confidentiality prevents me from citing specific examples, or giving user-quotes.

Slide 4: Also, great thanks to… • Yvonne French Product Manager for Reputation Platform • Randy Farmer Community Strategy Analyst (Hire Randy!) • Anne Binhack & Beverly Tseng Freeman User Researchers (Y! Answers & Community, respectively) • Various other folks from Yahoo! Answers, Sports, Buzz, Message Boards, EU Community, … DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 5: Reputation Your reputation in a community is equal to the sum of your past actions—whether good or bad—within that community. “Good” or “bad” is subjective and determined by the community itself. What the community values is good—what it abhors is bad. DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS I’m really hoping to avoid a “Webster’s Dictionary definition” slide, so I’ll just give you my own coarse way of thinking about reputations…

Slide 6: Reputation Systems Reputation systems attempt to mediate and automate this process: to take note of a community-member’s actions; assess the community’s reaction to them; and keep a running tally of the history of these actions. DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Optionally, the system may show this reputation to its possessor. Or back to the community at large. ‣ Not strictly incentive systems ‣ Not reputation management • Focused on People reputation, tho’ the same principles easily apply to content reputation • Digg • Google/Pagerank

Slide 7: [Discuss Amazon—Top 100 Reviewers.] Joshua Porter: “Is Harriet Klausner for real?” Avg. of 7 books a day! http://bokardo.com/archives/is-harriet-klausner-for-real/ [Xbox Live] Several different reputation indicators: appropriate for a gaming community. [starwars.yahoo.com] An -aggressive- attempt to bring gaming elements to a content-driven community [Ebay] •Points-based reputation, number of transactions. Plus some reciprocal peer testimonials. •Ebay has been doing significant work in revamping reputation over the last 1 or two years. I’ve failed to keep up with all of the changes, myself. (But worth a re-look.)

Slide 8: developer.yahoo.com/ypatterns/ DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS [Plug Design Pattern Library — Reputation Patterns being released soon. Perhaps as soon as next week. … Christian?]

Slide 9: Levels 5 4 3 2 1 NUMBERED Fanatic OR Trendsetter Enthusiast Listener Newbie NAMED DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Numbered: A family of reputations on a progressive continuum. Each level that is achieved is higher than the one before it. Levels are referred to by their number,which makes comparisons between levels very straightforward and easy-to-do. Named: A family of reputations on a progressive continuum. Each level that is achieved is higher than the one before it. Levels are given unique names, which can give them a fun and approachable quality. Quick comparisons between levels, however, become slightly more difficult.

Slide 10: Numbered: Yahoo Answers. Easy comparisons. Named: phpBB. Tho’ they don’t look like it, yes those are reputations. Fun! but… wha?

Slide 11: Identifying Labels Helpful Elite Of ficial DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS A family (one or more) of reputations that are not sequential in nature. Each reputation is crafted to identify and reward particular behaviors or qualities within a community. Identifying labels are helpful for consumers in identifying more-experienced contributors who possess these qualities. (eg, 'Helpful' guides, or 'Elite' reviewers.) Identifying Labels are not particularly useful for comparing one reputation-holder to another.

Slide 12: Yelp: Elite Status Get Satisfaction: Employee --> Official Rep. --> Admin. There’s a validation process, requiring an email address w/requested company to start. [Note that we’ll talk about externally-validated processes as reputation Inputs later.]

Slide 13: Points 1,103 180 10 DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS A cumulative count of the number of points that a user has earned within a community. The points generally come from performing one of a number of activities on the property. Do not confuse points with spendable currency, btw.

Slide 15: Top ‘X’ TOP 10 TOP 50 TOP 100 DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Contributors are grouped, numerically, into 'buckets' of performance, and top performers are acknowledged for their superior achievements. Top 10, 50 and 100 are some commonly-used groupings.

Slide 16: Amazon and Yahoo! Shopping: Amazon is kind of the canonical example. Y! Shopping? lesser known.

Slide 17: Collectible Achievements DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Provide some boon, or reward to users for attaining certain goals within the community. Make them a consistent family, or program, of collectibles. Enhance their fun appeal by: fetishizing them in some way—develop attractive trophies, icons or 'gamepieces' to represent each achievement; allow users to save them and put them on display; provide a healthy mix of difficulties—make some achievements very easy and quick (low-hanging fruit) while others require time and effort to conquer; unlock new achievements as easier ones are accomplished.

Slide 18: Y! Fantasy Sports: a fun and consistent set of collectibles. (Even a ‘case’ to keep them in.) Yahoo! Starwars: Nicely fetishistic, but they ‘break character’ with the universe. Also… ALWAYS MAKE THOSE ITEMS CLICKABLE. They should pay-off to a explanation of what they are. (Related: Question 7.)

Slide 19: Ranking (or ‘Leaderboards’) 1 DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Users' performance is tracked in some empirical and granular fashion (most typically points) in order to compare them against one another. Users accrue reputation by rising in the ranks, which necessarily comes at the expense of other users in the community. User rankings are displayed in Leaderboards.

Slide 20: Temporal Awards DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Users are rewarded or acknowledged for their contributions over a specific timeframe, or interval. Weekly- , Monthly- or Yearly Awards are common examples. These reputations, once earned, are never lost; they're useful both for heaping praise upon consistent top performers and for giving a wider number of users the opportunity to earn a reputation. Temporal Awards may also be used to award 'keepsake' reputations for special events or promotions.

Slide 21: Statistical Evidence DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Statistics about a user's participation or history in the community are displayed. No attempt is made to aggregate or abstract the statistics into higher-order patterns: rather, readers can decide for themselves the value of that user's contributions. Statistical evidence may also be used to validate another reputation pattern (eg. to provide justification for why someone has earned a Gold Medal.)

Slide 22: Peer Testimonials DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Members of the community are provided a mechanism to speak to the qualities of other members. Typically, - positive- testimonials are encouraged (often enforced by allowing the recipient the right to -approve- submitted testimonials before they are made publicly available. Often, peer testimonials are designed to encourage reciprocity.

Slide 23: User-to-User Awards DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Similar to Collectible Achievements. The significant difference is that these are peer-granted (and not ‘officiated’ through any broker.) Therefore, they’ll typically be more light-hearted fare, and probably not based on inputs reflecting Quality.

Slide 24: Yahoo’s BravoNation is a really simple, fun peer-to-peer Award granting product.

Slide 25: Thank You! DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS So… there you have it! Talk’s done! Of course, that’s not true… what we’ve looked at is a laundry list of reputation patterns, but no guidance on which one to use in what situation… so…

Slide 26: 1. What are your Business Goals? DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 27: 1. What are your Business Goals? • Drive user engagement? (However you define that.) • Promote a specific feature? • Acknowledge top contributors? • Increase content quality? • Promote user retention? DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Engagement — establish and measure an enduring relationship w/users. As simple as ‘time spent on site’ or as deep as “the number of comments left on posts not authored by ‘Friends’”. For more interesting ideas on which metrics to follow: http://www.horsepigcow.com/2007/10/03/metrics-for-healthy-communities/ Promote a feature – eg. a Top Reviewer’s Badge to encourage review-writing. (But think about how the system will have continued relevancy as initial goals are met, too.) Top Contributors — tread carefully, though. You’re just as likely to turn top contributors off if you don’t consider their other motivations. (See Question 3!) Also be aware: effect on those who aren’t acknowledged. Increase content quality by highlighting top-quality contributors or their contributions. You’re really increasing the perception of quality, which will hopefully influence the submission of higher-quality content. (The virtuous circle.) Don’t PUNISH low-quality content. Just let it lie unnoticed. Promote user retention, such as in…

Slide 28: Reputation to build loyalty • Collectible Achievements • Temporal Awards • Statistical Evidence • Tenure • Spectrum DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Fantasy Sports employs a number of Reputation Indicators. These promote loyalty from season to season because the cost of switching to another provider is high: lose your hard-won standing and have to start again from scratch. BUT… There’s room for improvement. Reputations are temporal (per season) and each new season, the rep-holder begins again at ‘level n00b.’ [How might the reputation system reward long-time, engaged and SUCCESSFUL players and managers?]

Slide 29: 2. What Community Spirit do you want to encourage? DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 30: The Competitive Spectrum Caring Collaborative Cordial Competitive Combative Members are Member goals are Members have their Members share the Members share motivated by largely shared ones. own intrinsic same goals, but opposing goals: in helping other Members work motivations, but must compete order for one members—giving together to achieve these goals need against each other member to achieve advice, solace or those goals. not conflict with to achieve them. these goals, others comfort. other members' must necessarily be goals. denied their own. DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS “Competitiveness” is but one facet of a Community’s spirit, but it’s probably the most compelling one to consider. This scale is intended to be subjective. You can find many examples that will break this model (and where you place an example on the spectrum will probably differ from where I would.) But it’s a good starting point for discussion. [Briefly walk through each stage in the spectrum]

Slide 31: Examples Caring Collaborative Cordial Competitive Combative • Support Groups • Wikipedia • Yahoo! Answers • Fantasy Sports • Halo on XBox Live • A LiveJournal blog • Dating Sites • Message Boards • Casual Gaming • Hot or Not • Yahoo! Health • Ebay • YouTube Expert Blogs DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Danah Boyd has published some good stuff on the value of LiveJournal community for at-risk teens. You may take issue with where I’ve placed some of these along the spectrum. (Dating sites and Ebay, for instance —was your first inclination to say ‘but those are competitive!! You’re competing with others on those sites!) [Point out that the consumers of these reputations, however, aren’t competing w/the rep-holders. They’re trying to find a good partner/match/someone they can trust. The goal is a successful collaboration!] [Mention Ben Brown’s lessons from Consumating: http://benbrown.com/says/category/ chickenwiremommy/ ]

Slide 32: The Competitive Spectrum Use Reputation to… Identify senior Identify community Show a member's Show a member's Show a member's community members with a history of level of history of members of good proven track-record participation, that accomplishment, accomplishments, standing, so that of being others may get a that others may including other others can find trustworthy general sense for acknowledge (and members' victories them for advice and partners. their interests, admire) their level and defeats against guidance. identity and values. of performance. them. Use reputation to establish bragging rights. DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 33: 3. What motivates your community members? DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 34: 3. What motivates your community members? • Your answer had better not be: The Incentive System • Also — don’t rely on altruism alone • Motivations will vary according to context, so it’s best to research your own community DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Earlier, I mentioned that we prefer to think in terms of Reputation, not Incentive. If you find yourself saying things like “Our users will do this because they want the points (or… the medals, the cash incentive, etc.)” that’s a red flag. You will be motivating the worst contributors (the extrinsically-motivated) to have the loudest voice in the community. Randy Farmer is fond of saying: “If your incentive system is users’ only incentive, then you’re SCREWED.”

Slide 35: Why do people write reviews? We brought 10 people (active contributors to various online communities) into the lab to compare and contrast ~16 different ways of expressing reputation. Brett W 27, Male dam T Chicago, IL (USA) A View Profile ale Grego 38, M , IA (USA) Iowa C ity 45, M ry B ale Colum 28px rofile bus (U 128x1 View P SA ) Bronze reviewer View P rofile 23 reviews written • 14 "Helpful" votes S cey Sta ) 1) tes ale (USA Level Fem k, NYpoints ( n • 2 "Help ful" vo Revie Tere 29, Yor 83 ritte 23 revw s Hot sa w iews w iews w shot! 128 46, Fe B Ne 13ilrev f e ritten • 14 "H x12 8px Quin male w Pro elpful" cy, K Y (U px Vie votes SA) 128 View 128x Prof ile T O r te s P welpful" vo 10 Top vie e 423 1 0 Re Reen • 2 "H 1 t revi ews view er vel w ritte t wri n • 85 Le reviews Ma 1 "H elpf ul" v 13 21 rk otes , Sa Male H nJ ose Vie ,C w A( Pro US Fu file A) nn y! (12 vot es) DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 36: We learned their preferences, but… We also learned what motivates them to participate in the first place. Interest in the Object Being Self-Interest Interest in Others Rated Have a voice Help Others “Fill a void” Practice a craft Share a unique perspective “Set the record straight” (or dissenting opinion) Chronicle life experiences Feel connected to others Reward the Business Be seen in a favorable light Entertain Others Punish the Business Express yourself (All thanks and credit to Beverly Tseng Freeman) DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 37: Yelp does a good job Reputations earned on Yelp include: 1st Reviewer (“Fill a Void”); Useful, Funny and Cool (Be Seen in a Favorable Light, Entertain Others, Help Others) DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Yelp has an extensive reputation system, with many reputations seemingly tailored to acknowledge this wide range of motivations for writing reviews. It’s possible to be many different kinds of ‘Yelper’ (funny, or helpful, or a pioneer) and still feel appreciated. The quality of submissions to Yelp reflects this: some are funny, some are quirky, some well- written and some are very autobiographical. All can be rewarded for their own peculiar qualities. Note, too, that Yelp! does -not- feature easy user-to-user comparisons. No leaderboards.

Slide 38: 4. Which entities will accrue reputation? DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 39: 4. Which entities will accrue reputation? • People? • Things? • Collections of things? Or all of these? DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Today’s talk is focused on People Reputation, but the exact same principles apply to Digg or Reddit, or ‘Honors’ badges on YouTube. And — as you’ll see — it’s impossible to discuss a person’s reputation without managing and tracking the reputation of the artifacts that they author. So let’s dissect, a bit, how this might work…

Slide 40: The same principles apply for people and ‘content’ “Jedi Master” DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Bob is our actor. The little blue guy represents “Content” of any type: a video, a blog entry, a comment on someone else’s blog, or even just a bookmark that I’ve saved. We’ll focus on Bob’s story, but we won’t get very far before we have to consider the other guy too...

Slide 41: Bob joins your community 10 DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS He has some amount of karma, by simple virtue of joining. (‘Firsts’ are good milestones to reward!) NOTE: Bob should never, ever again fall -below- this karma level. (The incentive would be too great at that point to abandon his identity and start over.)

Slide 42: He starts to participate 24 WRITES A REVIEW +4 LEAVES A COMMENT +1 POSTS BLOG ENTRY POSTS A ANSWERS +2 VIDEO A QUESTION +4 +3 DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Bob starts slow… leaves a comment on someone’s blog … then he posts a video of himself dancing to Soulja Boy… etc, etc. Soon, Bob is that engaged community contributor that we all hope for and actively using all the features that our community has to offer. Why are different activities weighted differently? Could be a number of things: the effort involved to produce the artifact; your business priorities (see Question 1); could be based on an economic model, where demand w/in a context determines an objects ‘value.’ BUT… plan on changing these weightings over time. Don’t lock your design into any constants.

Slide 43: The community responds 86 10 HELPFUL VOTES +40 THUMBED UP +6 READ 200 TIMES FLAGGED VOTED BEST +20 INAPPROPRIATE! ANSWER! -10 +6 DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS [Walk quickly through the various responses… note that Bob probably should have worn pants in his dancing video ;-) … Also point out that these events are INPUTS] These responses could take the form of atomic, singular events (a vote is cast, a button is clicked) or they could only apply once some aggregate milestone is reached. And—again—the karmic value of each event is weighted by you, the designer in consideration of your business goals, community goals and the questions that we’ll be discussing today. And that’s roughly how Bob’s reputation is generated. (Pending the rest of our questions, of course…)

Slide 44: Rate the thing, not the person At Yahoo! we try not to allow the community to rate Bob directly. •Keeps the focus on his works, not the quality of his character • Discourages ad-hominem attacks • Even positive-only votes can be manipulated DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Other communities may feel differently about this: HOT or NOT for example is entirely based around rating others. “Positive-only” votes offer only compliments, but these too should be directed at things, not people. (Nice article! instead of ‘You’re Cute’… less creepy.) There are other ways to abuse these too (Discussed in Question 14.)

Slide 45: 4. Which entities will accrue reputation? ✓ People And the things those people create… ✓ Videos ✓ Playlists ✓ Blog posts ✓ Message board ✓ Bookmarks posts ✓ Article submissions ✓ Reviews ✓ Collections ✓ “Social Objects” DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS So back to our original question… for -your- property, which entities should accrue reputation? We’ll assume that you DO want users to have reputation (the point of this talk, after all…) Everything else depends on the offerings for your property, and the types of reputation that you want to promote.

Slide 46: Message Board entities DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS Really we care about a contributor and her posts in this example. But there may be a very limited application of thread reputation to pay attention to: perhaps we’re interested in creating a ‘Conversation Starter’ reputation so we want to calculate thread reputation -only- for threads that she has initiated. So… entities may be conditional.

Slide 47: 5. Which inputs should you pay attention to? DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS

Slide 48: 5. Which inputs should you pay attention to? 1. Perform an audit of your property. 2. List all the actions that a user can take. 3. Note the ones that—in some way—make a value statement about the entities that you’ve defined (in the last question.) DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS We do so love our audits, don’t we? When listing actions, look beyond the surface: atomic actions that are readily apparent from scanning the UI — these are important, too! Record them. But also think about non-obvious actions, like: Reading an entry; bookmarking something. Inputs will be constrained by what your application is instrumented to record.

Slide 49: Consider YouTube… DESIGNING YOUR REPUTATION SYSTEM IN 15 EASY STEPS This is but a partial listing of all the potential ways you could make value judgements about a video on YouTube. How many views does it have? How many times viewed all the way through? How frequently is it shared/favorited or flagged? Ratings provide an Explicit indicator of Quality — most of the others indicate Interest alone. Quality is -implied-.

Slide 50: Typical ‘Message Board’ Inputs Action Affects… Indicates… Start new thread Thread Activity Post Interest (Implicit) Reply to a Post Per