Augmented Social Cognition: Who Edits Wikipedia? Ed H. Chi Augmented Social Cognition Area Palo Alto Research Center 2007-06-17 Ed H. Chi - Who writes Wikipedia? Image from: http://www.flickr.com/photos/ourcommon/480538715/

Wikipedia 2007-06-17 Ed H. Chi - Who writes Wikipedia?

High-end of the collaboration spectrum Groups utilize systems to make sense and share complex topics and materials. Wikipedia (social status) Slashdot (karma points) eHow.com Lostpedia.com 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Groups utilize systems to make sense and share complex topics and materials.

Wikipedia (social status)

Slashdot (karma points)

eHow.com

Lostpedia.com

Middle of the spectrum Systems that evolve structures that can be used to organize information. Del.icio.us Flickr YouTube Friendster 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Systems that evolve structures that can be used to organize information.

Del.icio.us

Flickr

YouTube

Friendster

Lightweight social processes Counting votes A way to increase signal-to-noise ratio Information faddishness Examples: Digg.com Most bookmarked items on del.icio.us Estimating the weight of an ox or temperature of a room The true value of a stock PageRank or Hub / Authority algorithms 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Counting votes

A way to increase signal-to-noise ratio

Information faddishness

Examples:

Digg.com

Most bookmarked items on del.icio.us

Estimating the weight of an ox or temperature of a room

The true value of a stock

PageRank or Hub / Authority algorithms

Layers of Models Needed 2007-06-17 Ed H. Chi - Who writes Wikipedia? Heavier collaboration Voting systems Digg.com Collaborative Creation Wikipedia Col. Information Structures Slashdot eHow.com Del.icio.us IBM dogear PageRank Flickr Understanding of micro-economics of foraging [PARC] Personal vs. group [Huberman, Adamic] Wisdom of Crowd [Surowieki] Information cascades [Anderson and Holt] Understanding of conflicts and coordination Wikipedia coordination costs [PARC] Invisible Colleges [Sandstrom] Interference effects [Pirolli] Co-laboratories [Olson and Olson] Community networks / Col. Problem solving [Carroll] Understanding of info and social networks Tag network analysis [PARC, Golder, Yahoo] Structural holes (info brokerage) [Burt] Network constraints and structure [various] Semantic of semiotic structures / words [IR, LSA]

Understanding of micro-economics

of foraging [PARC]

Personal vs. group [Huberman, Adamic]

Wisdom of Crowd [Surowieki]

Information cascades [Anderson and Holt]

Understanding of conflicts and coordination

Wikipedia coordination costs [PARC]

Invisible Colleges [Sandstrom]

Interference effects [Pirolli]

Co-laboratories [Olson and Olson]

Community networks / Col. Problem solving [Carroll]

Understanding of info and social networks

Tag network analysis [PARC, Golder, Yahoo]

Structural holes (info brokerage) [Burt]

Network constraints and structure [various]

Semantic of semiotic structures / words [IR, LSA]

Research Vision Augmented Social Cognition Cognition : the ability to remember, think, and reason; the faculty of knowing. Social Cognition : the ability of a group to remember, think, and reason; the construction of knowledge structures by a group. (not quite the same as in the branch of psychology that studies the cognitive processes involved in social interaction, though included) Augmented Social Cognition : Supported by systems, the enhancement of the ability of a group to remember, think, and reason; the system-supported construction of knowledge structures by a group. 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Cognition : the ability to remember, think, and reason; the faculty of knowing.

Social Cognition : the ability of a group to remember, think, and reason; the construction of knowledge structures by a group.

(not quite the same as in the branch of psychology that studies the cognitive processes involved in social interaction, though included)

Augmented Social Cognition : Supported by systems, the enhancement of the ability of a group to remember, think, and reason; the system-supported construction of knowledge structures by a group.

The first step in solving any interesting problem is to get some paper and pencil. John Tukey (not a direct quote) 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Increasing Coordination Cost in Wikipedia (joint work with Niki Kittur, Bongwon Suh, Bryan Pendleton) Published in CHI2007 conference: Aniket Kittur, Bongwon Suh, Bryan Pendleton, Ed H. ChiHe Says, She Says: Conflict and Coordination in Wikipedia. In Proc. of ACM Conference on Human Factors in Computing Systems (CHI2007), pp. 453--462, April 2007. ACM Press. San Jose, CA 2007-06-17 Ed H. Chi - Who writes Wikipedia?

What is Wikipedia? 2007-06-17 Ed H. Chi - Who writes Wikipedia? “ Wikipedia is the best thing ever. Anyone in the world can write anything they want about any subject, so you know you’re getting the best possible information.” – Steve Carell, The Office

Increasing Coordination Costs in Wikipedia Understanding coordination costs is vital for long-term viability of collaborative information environment Data: Entire dump on July 2, 2006 58 million revisions 4.7 million wiki pages 2.4 million article pages 800 gigabytes 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Understanding coordination costs is vital for long-term viability of collaborative information environment

Data:

Entire dump on July 2, 2006

58 million revisions

4.7 million wiki pages

2.4 million article pages

800 gigabytes

Less direct work Decrease in proportion of edits to article page 2007-06-17 Ed H. Chi - Who writes Wikipedia? 70%

Decrease in proportion of edits to article page

More indirect work Increase in proportion of edits to user talk 2007-06-17 Ed H. Chi - Who writes Wikipedia? 8%

Increase in proportion of edits to user talk

More indirect work Increase in proportion of edits to user talk Increase in proportion of edits to procedure 2007-06-17 Ed H. Chi - Who writes Wikipedia? 11%

Increase in proportion of edits to user talk

Increase in proportion of edits to procedure

More maintenance work Increase in proportion of edits that are reverts 2007-06-17 Ed H. Chi - Who writes Wikipedia? 7%

Increase in proportion of edits that are reverts

More wasted work Increase in proportion of edits that are reverts Increase in proportion of edits reverting vandalism 2007-06-17 Ed H. Chi - Who writes Wikipedia? 1-2%

Increase in proportion of edits that are reverts

Increase in proportion of edits reverting vandalism

Global level Conflict and coordination costs are growing Less direct work (articles) More indirect work (article talk, user, procedure) More maintenance work (reverts, vandalism) 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Conflict and coordination costs are growing

Less direct work (articles)

More indirect work (article talk, user, procedure)

More maintenance work (reverts, vandalism)

Conflict at the article level Conflict is growing at the global level We have some idea about where it is But what defines conflict at the local level? Build a characterization model of article conflict Identify metrics relevant to conflict Automatically identify high-conflict articles 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Conflict is growing at the global level

We have some idea about where it is

But what defines conflict at the local level?

Build a characterization model of article conflict

Identify metrics relevant to conflict

Automatically identify high-conflict articles

Measure of controversy “ Controversial” tag Use # revisions tagged controversial 2007-06-17 Ed H. Chi - Who writes Wikipedia?

“ Controversial” tag

Use # revisions tagged controversial

Page metrics Possible metrics for identifying conflict in articles 2007-06-17 Ed H. Chi - Who writes Wikipedia? Metric type Page Type Revisions (#) Article, talk, article/talk Page length Article, talk, article/talk Unique editors Article, talk, article/talk Unique editors / revisions Article, talk Links from other articles Article, talk Links to other articles Article, talk Anonymous edits (#, %) Article, talk Administrator edits (#, %) Article, talk Minor edits (#, %) Article, talk Reverts (#, by unique editors) Article

Possible metrics for identifying conflict in articles

Performance: Cross-validation 5x cross-validation, R 2 = 0.897 2007-06-17 Ed H. Chi - Who writes Wikipedia?

5x cross-validation, R 2 = 0.897

Performance: Cross-validation 5x cross-validation, R 2 = 0.897 2007-06-17 Ed H. Chi - Who writes Wikipedia?

5x cross-validation, R 2 = 0.897

Determinants of conflict   Revisions (talk)   Minor edits (talk)   Unique editors (talk)   Revisions (article)   Unique editors (article)   Anonymous edits (talk)   Anonymous edits (article) 2007-06-17 Ed H. Chi - Who writes Wikipedia? Highly weighted features of conflict model:

  Revisions (talk)

  Minor edits (talk)

  Unique editors (talk)

  Revisions (article)

  Unique editors (article)

  Anonymous edits (talk)

  Anonymous edits (article)

Highly weighted features of conflict model:

Model Generalization and Validation survey Applied model to untagged articles (100+ edits) Sampled range of predicted conflict scores Rated by expert Wikipedians Significantly correlated with predicted scores By rank correlation, p < 0.013 (Spearman’s rho) Validates characterization model Detects conflicts even for articles with no ground truth 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Applied model to untagged articles (100+ edits)

Sampled range of predicted conflict scores

Rated by expert Wikipedians

Significantly correlated with predicted scores

By rank correlation, p < 0.013 (Spearman’s rho)

Validates characterization model

Detects conflicts even for articles with no ground truth

Who edits Wikipedia? % of edits made by administrators 2007-06-17 Ed H. Chi - Who writes Wikipedia?

% of edits by 10k+ editors 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Word changes made by admins 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Shifting user population in Wikipedia (more and more bottom driven!) 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Proportion of edits made by top editors in Wikipedia 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Long tail of participation in Wikipedia 2007-06-17 Ed H. Chi - Who writes Wikipedia?

The participation architecture is a power law 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Only 60% of top 1% editors stay around month to month! 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Living Laboratory: Prototyping Social Applications on the Internet Create a Living Laboratory as a platform to develop, test, and market our innovations, and as a vehicle for creating collaborations and thought leadership. 2007-06-17 Ed H. Chi - Who writes Wikipedia?

WikiDashboard Joint work with Bongwon Suh, Aniket Kittur, Bryan Pendleton 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Risks for Using Wikipedia Factual accuracy Motives of editors Uncertain expertise Volatility Spotty coverage Unproven/non-independent source 2007-06-17 Ed H. Chi - Who writes Wikipedia? [Denning et al. 2005]

Factual accuracy

Motives of editors

Uncertain expertise

Volatility

Spotty coverage

Unproven/non-independent source

Social Dashboard Social translucent for effective communication and collaboration Make socially significant information visible and salient Support awareness of the rules and constraints Accountability for actions Wikis can be a prime candidate Every edit is logged and retrievable WikiScanner.com WikiRage.com Intellipedia 2007-06-17 Ed H. Chi - Who writes Wikipedia? [Erickson and Kellogg 2002]

Social translucent for effective communication and collaboration

Make socially significant information visible and salient

Support awareness of the rules and constraints

Accountability for actions

Wikis can be a prime candidate

Every edit is logged and retrievable

WikiScanner.com

WikiRage.com

Intellipedia

WikiDashboard Surfacing hidden social context to users For readers Any incidents in the past e.g. A sudden burst of edits? Who are the editors? What is their motivation / point of views / expertise / topics of interest Help them judging the quality/trustworthiness/usefulness of an article For writers Measure expertise / contribution / reputation Motivate them to be more active / responsible (?) 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Surfacing hidden social context to users

For readers

Any incidents in the past e.g. A sudden burst of edits?

Who are the editors?

What is their motivation / point of views / expertise / topics of interest

Help them judging the quality/trustworthiness/usefulness of an article

For writers

Measure expertise / contribution / reputation

Motivate them to be more active / responsible (?)

Article Dashboard 2007-06-17 Ed H. Chi - Who writes Wikipedia?

User Dashboard 2007-06-17 Ed H. Chi - Who writes Wikipedia?

Drilling Down List of every edits that a user made Let readers examine each individual revision for validity, which is hard to accomplish when only provided with aggregate visual summaries. 2007-06-17 Ed H. Chi - Who writes Wikipedia?

List of every edits that a user made

Let readers examine each individual revision for validity, which is hard to accomplish when only provided with aggregate visual summaries.

Augmented Social Cognition: From Social Foraging to Social Sensemaking 2007-06-17 Ed H. Chi - Who writes Wikipedia? Image from: http://www.flickr.com/photos/ourcommon/480538715/ Research Vision: Understand how social computing systems enhance the ability of a group of people to remember, think, and reason. Living Laboratory: Create breakthrough applications that harness collective intelligence to improve knowledge capture, transfer, and discovery.

Research Vision: Understand how social computing systems enhance the ability of a group of people to remember, think, and reason.

Living Laboratory: Create breakthrough applications that harness collective intelligence to improve knowledge capture, transfer, and discovery.