Slides from my Wikimania 2014 presentation on metrics standardization and the Vital Signs project.

1. Measuring Community Health
Vital Signs for Wikimedia Projects
Dario Taraborelli • Aaron Halfaker • Dan Andreescu
Wikimania 2014, London

3. metrics != science

4. metrics as filters

5. summer 2013

6. cohort-level metrics

7. cohort-level metrics project-level metrics

8. project-level metrics

9. project-level metrics

10. Existing data sources
wikistats report card
http://stats.wikimedia.org http://reportcard.wmflabs.org/

11. ENWIKI New Editors / day 1D: 21% 30D: 18% YTD:
20%
Vital Signs
• granular measurements of new user engagement, community size, and content growth
• aggregated daily / weekly / monthly
• for every single Wikimedia project
• visualizations + raw data
https://www.mediawiki.org/wiki/Analytics/Epics/Editor_Engagement_Vital_Signs
02/01: 1240
•
summer 2014

12. 4 categories of metrics
New users Community Content Curation
Newly registered users
New editors
New active editors
Productive new editors
Surviving new editors
Surviving new active editors
Active editors
Recurring old active editors
Re-activated editors
Unique editors
Unique anonymous editors
Unique editing bots
Unique page creators
Unique media creators
Edits
Anonymous edits
Bot edits
Pages created
Media uploaded
Pages deleted
Pages protected
Pages moved
Reverts
https://meta.wikimedia.org/wiki/Research:Metrics_standardization

13. metric definitions

14. Relevant
Measure quantities that describe important phenomena
Replicable
Make research easily replicable and verifiable
Transparent
Provide formal specifications, remove ambiguity
Consistent
Replace proprietary, ad-hoc metric definitions; compare apples to apples
Robust
Make metrics replicable via multiple data sources at any point in time
Granular
Computable at different time scales
Principles

15. Anatomy of a metric 1. specification

16. Anatomy of a metric 2. visualizations
registration
time
Activation Trial Survival

17. Anatomy of a metric 2. visualizations
New editor
Productive new editor
Surviving new editor

18. Anatomy of a metric 3. discussion

19. Anatomy of a metric 4. sensitivity analysis

20. Sensitivity analysis
https://meta.wikimedia.org/wiki/Research:Productive_new_editor
Does new editor productivity vary when we measure it over the first day or the first week?

21. Sensitivity analysis
https://meta.wikimedia.org/wiki/Research:New_editor
Should we define new editors based on activity in the article namespace only?

22. https://meta.wikimedia.org/wiki/Research:Rolling_monthly_active_editor
What segments of the population of a project drive total active editor numbers
Anatomy of a metric 5. segmentation

23. Use cases

24. Use cases
1. Data exploration
“Newly registered users on German and Dutch Wikipedia have a higher activation
rate than newbies who join the English Wikipedia”
“Spanish Wikipedia adds every day twice as many new editors than German
Wikipedia, despite having only half its new user activation rate”
2. Natural experiments
“A change in abuse filter rules on the Italian Wikipedia significantly increased new
editor survival”

25. Use cases
3. Projections and target setting
“To stop the active editor decline in the English Wikipedia, we should increase the
retention of existing users by 87% or increase the activation of new editors by 23%”
25.8%
37.3%
33.7%
3.1%
New active
Surviving new active
Recurring old active
Re-activated

26. What we’re building

27. Data generation
DEMO
https://metrics-staging.wmflabs.org/static/public/dash/

28. Information architecture
http://pauginer.github.io/prototypes/analytics-dashboard/

29. Beyond basic measurements
Beyond signals based on simple edit counts: quality, value added
Better segmenting of the editor population (classifying edit types)
Readership metrics (unique visitors, pageviews)
Feedback and evaluation

30. Questions?
dario@wikimedia.org [[User:DarTar]] @readermeter
ahalfaker@wikimedia.org [[User:Halfak (WMF)]] @halfak
dandreescu@wikimedia.org [[User:DAndreescu]] @DanAndreescu
Read more
https://meta.wikimedia.org/wiki/Research:Metrics_standardization
Image credits
Tim Sheerman-Chase. Fair, Barometer Detail (CC BY) https://www.flickr.com/photos/68932647@N00/7615682432/
Mark Dumont. Mad Science (CC BY) https://www.flickr.com/photos/23661161@N02/5826590955/
Future scientist experiments on sister http://www.gifbay.com/gif/future_scientist_experiments_on_sister-117763/
Joaquim Alves Gaspar. Vernier Caliper (CC BY SA) https://commons.wikimedia.org/wiki/File:Vernier_caliper.png