Metrics standardization
Dario Taraborelli • Aaron Halfaker
Wikimedia Research and Data showcase
March 2014
summer 2013
cohort-level metrics
cohort-level metrics project-level metrics
project-level metrics
project-level metrics
ENWIKI New Editors / day 1D: 21% 30D: 18% YTD:
20%
Editor engagement vital signs
key performance indicators for user engag...
Key metrics
New users Community Content Curation
Newly registered users
New editors
Productive new editors
Surviving new e...
Relevant
Measure quantities that describe important phenomena
Replicable
Make research easily reproducible and verifiable
...
Anatomy of a metric 1. specification
Anatomy of a metric 2. visualizations
registration
time
Activation Trial Survival
Anatomy of a metric 2. visualizations
New editor
Productive new editor
Surviving new editor
Anatomy of a metric 3. discussion
Anatomy of a metric 4. sensitivity analysis
Sensitivity analysis
https://meta.wikimedia.org/wiki/Research:Productive_new_editor
Does new editor productivity vary when...
Sensitivity analysis
https://meta.wikimedia.org/wiki/Research:New_editor
Does it really matter to limit new editor activat...
Sensitivity analysis
https://meta.wikimedia.org/wiki/Research:Surviving_new_editor
Does the length of the trial and surviv...
Why does this matter at all?
1. Data exploration
“Newly registered users on German and Dutch Wikipedia have a higher activ...
Metric specification
Sensitivity analysis
Parameter recommendation
Release
Evaluation
Evaluation
Metric specification
Sensitivity analysis
Parameter recommendation
Release
Evaluation
Evaluation
feedback
Questions?
dario@wikimedia.org
ahalfaker@wikimedia.org
Read more
https://meta.wikimedia.org/wiki/Research:Metrics_standard...
Upcoming SlideShare
Loading in …5
×

Metrics standardization. Wikimedia Research & Data Showcase, March 2014

977 views
843 views

Published on

Slides from the talk on metrics standardization from the March 2014 Wikimedia Research & Data Showcase.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
977
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
1
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Metrics standardization. Wikimedia Research & Data Showcase, March 2014

  1. 1. Metrics standardization Dario Taraborelli • Aaron Halfaker Wikimedia Research and Data showcase March 2014
  2. 2. summer 2013
  3. 3. cohort-level metrics
  4. 4. cohort-level metrics project-level metrics
  5. 5. project-level metrics
  6. 6. project-level metrics
  7. 7. ENWIKI New Editors / day 1D: 21% 30D: 18% YTD: 20% Editor engagement vital signs key performance indicators for user engagement, community and content growth aggregated daily / weekly / monthly for every single Wikimedia project https://www.mediawiki.org/wiki/Analytics/Epics/Editor_Engagement_Vital_Signs 02/01: 1240 • summer 2014
  8. 8. Key metrics New users Community Content Curation Newly registered users New editors Productive new editors Surviving new editors ... Editors Active editors Very active editors IP editors Bots Page creators ... Edits Bot edits Uploads Pages ... Page deletions Reverts ... https://meta.wikimedia.org/wiki/Research:Metrics_standardization
  9. 9. Relevant Measure quantities that describe important phenomena Replicable Make research easily reproducible and verifiable Transparent Provide formal specifications, remove ambiguity Consistent Replace proprietary, ad-hoc metric definitions; compare apples to apples Robust Make metrics replicable via multiple data sources at any point in time Granular Computable at different time scales Rationale
  10. 10. Anatomy of a metric 1. specification
  11. 11. Anatomy of a metric 2. visualizations registration time Activation Trial Survival
  12. 12. Anatomy of a metric 2. visualizations New editor Productive new editor Surviving new editor
  13. 13. Anatomy of a metric 3. discussion
  14. 14. Anatomy of a metric 4. sensitivity analysis
  15. 15. Sensitivity analysis https://meta.wikimedia.org/wiki/Research:Productive_new_editor Does new editor productivity vary when we measure it over the first day or the first week?
  16. 16. Sensitivity analysis https://meta.wikimedia.org/wiki/Research:New_editor Does it really matter to limit new editor activation to main namespace edits only?
  17. 17. Sensitivity analysis https://meta.wikimedia.org/wiki/Research:Surviving_new_editor Does the length of the trial and survival period affect the measurement of new editor survival?
  18. 18. Why does this matter at all? 1. Data exploration “Newly registered users on German and Dutch Wikipedia have a higher activation rate than newbies on English Wikipedia” “Spanish Wikipedia adds every day twice as many new editors than German Wikipedia, despite having only half its new user activation rate” 2. Natural experiments “A change in abuse filter rules on the Italian Wikipedia significantly increased new editor survival”
  19. 19. Metric specification Sensitivity analysis Parameter recommendation Release Evaluation Evaluation
  20. 20. Metric specification Sensitivity analysis Parameter recommendation Release Evaluation Evaluation feedback
  21. 21. Questions? dario@wikimedia.org ahalfaker@wikimedia.org Read more https://meta.wikimedia.org/wiki/Research:Metrics_standardization Image credits W.Wood (1839) Index Entomologicus http://dx.doi.org/10.5962/bhl.title.12503

×