1. Wikipedia as
Participatory
Journalism: Reliable
Sources?
Metrics for evaluating collaborative media
as a news source
Andrew Lih, Assistant Professor
International Symposium on Online Journalism
April 17, 2004

2. Overview
Participatory journalism
Wikipedia introduction
Growth
Methodology
Results

3. Participatory Journalism
We Media, Bowman and Willis
The act of a citizen, or group of citizens, playing an
active role in the process of collecting, reporting,
analyzing and disseminating news and
information. The intent of this participation is to
provide independent, reliable, accurate, wide-
ranging and relevant information that a
democracy requires.

4. Forms
Web logs – one to many
Wikis – many to many

5. Wikis
Based on Hawaiian word for quick wikiwiki
Web site where any page can be edited by
users (Edit this page button)
Registered or anonymous users
Wikipedia encyclopedia format
How could it possibly work?

6. How could it work?
Neutral point of view (NPOV)
“The neutral point of view attempts to present
ideas and facts in such a fashion that both
supporters and opponents can agree.”

7. Journalistic decisions
Avoiding the use of word “terrorism,” very
much like Reuters
Intense debate on naming accuser in Kobe
Bryant case

9. Wikipedia Traffic Rank vs. Other Encyclopedias (2003-2004)
TrafficRank
Wikipedia.org
Britannica.com
Encyclopedia.com
500
2000
1000
4000
6000
Apr-03 Jul-03 Oct-03 Jan-04
10,000
12,000
Source: www.alexa.com

10. How could it work?
Neutral point of view (NPOV)
“The neutral point of view attempts to present
ideas and facts in such a fashion that both
supporters and opponents can agree.”

11. Research questions
What quantitative metrics for quality?
Analyze articles cited in the press
Any patterns or quality level of articles?
Effects of citation on article?

12. Metrics for each article
Rigor - number of edits
Diversity - number of unique editors
Taken from the edit history for each page
Metadata only, can be used for any language

14. Methodology
Establish benchmark by analyzing
representative general topics
Assumption that these would be exemplars
Measure median for those topics in Wikipedia
Edits: 61
Unique editors: 36.5

15. Methodology
Determine metrics for cited articles
Examine movement before citation, after
citation

16. Reputation of Major Encyclopedia Topics in English Wikipedia
(Diversity vs. Rigor)
Median
Sport
Buddhism
Law
History_of_China
Hinduism
World_War_II
Astronomy
Islam
Religion
Canada
Philosophy
Mathematics
World_War_I
Earth
Native_American
0
100
200
300
400
500
600
700
800
0 50 100 150 200 250 300 350
Diversity (number of unique editors)
Rigor(numberofedits)

17. Reputation of Major Encyclopedia Topics in Wikipedia
(Diversity vs. Rigor) using Logarithmic Scale
Colonial_America
History_of_the_Middle_East
World_War_II
Astronomy
Islam
Hinduism
History_of_China
Median
1
10
100
1000
1 10 100 1000
Diversity (number of unique editors)
Rigor(numberofedits)

18. News outlets citing Wikipedia
Aljazeerah.info
Arizona Republic
Associated Press
CBC Newsworld
Daily Telegraph online
Greenville News
Groklaw
Guardian
Houston Chronicle
JapanToday.com
Kansas City Star
Knoxville News Sentinel (Tennessee)
Loudon Times-Mirror
Lowell Sun Online
Metropolis
New Straits Times
NewScientist.com
OC Weekly
Sacramento Bee
Salt Lake Tribune
San Francisco Chronicle
San Jose Mercury News
Slashdot
Star-Telegram
Sydney Morning Herald
TechTV
The Age
The Dallas Morning News
The Globe and Mail
The New Zealand Herald
The Philadelphia Inquirer
The Pocono Record
The Straits Times
The Sydney Morning Herald
Time.com
Venezuelan News and Analysis
WOAI-TV San Antonio

19. Wikipedia Articles Cited in Press Jan 2003-March 2004 (Before Citation)
Highlighted Articles
X_Window_System
Who_Wants_to_Be_a_Millionaire
Vietnam_War
List_of_recipes
11_March_2004_Madrid_attacks
Saddam_Hussein
Fuck
Chen_Shui-bian
Communist_Party_of_China
0
100
200
300
400
500
600
700
800
900
1000
0 20 40 60 80 100 120 140 160 180
Diversity (number of unique editors)
Rigor(numberofedits)

20. Trends
Madrid attacks subject of constant updates
Asian topics have high edits, but low number
of contributors
Science and history have high showing

21. Wikipedia Articles Cited in Press Jan 2003-March 2004 (Before Citation)
1
10
100
1000
10000
1 10 100 1000
Diversity (number of unique editors)
Rigor(numberofedits)

22. Wikipedia Articles Cited in Press Jan 2003-March 2004 (After Citation)
1
10
100
1000
10000
1 10 100 1000
Diversity (number of unique editors)
Rigor(numberofedits)

23. Articles Moving Across Benchmark Point
62274522British_honours_system
63362811Suez_Crisis
54383528Leap_year
69385930Explosive_material
65404628Palestinian_Authority
70405330Execution_by_firing_squad
76412712Iain_Duncan_Smith
86413930Alexander_Graham_Bell
60434131Breakdancing
83435423Form_of_government
105446032Australian_dollar
95474716History_of_Iraq
98472717Lyndon_LaRouche
95482015Provisional_Irish_Republican_Army
102515022Weapons_of_mass_destruction
106545229Juliana_of_the_Netherlands
84624133Aramaic_language
2261102413Ahmed_Yassin
EUEEUE
AfterBeforeArticle name

24. Articles improved, without
direct link to current events
Alexander_Graham_Bell
Australian_dollar
Breakdancing
Execution_by_firing_squad
Explosive_material
Form_of_government
Lyndon_LaRouche
Suez_Crisis

25. Conclusions
Simple metrics for rigor and diversity can help
identify trends
Citations in press can lead to improvement of
articles
If news is the first draft of history, Wikipedia
is a “working draft of history”

26. Future Work
Content analysis in conjunction with
metadata
Detect edit wars, vandalism, which skew results
Analyze internal and external hyperlinks
Measure more limited window of time
More broad benchmark topics

27. Andrew Lih
alih@hku.hk
http://jmsc.hku.hk/