Heartbeat: Measuring Active User Base and Potential User Interest


Published on

Presentation for OSS 2009 in Skövde, Sweden.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Heartbeat: Measuring Active User Base and Potential User Interest

  1. 1. Heartbeat Measuring Active User Base and Potential User Interest in FLOSS Projects Andrea Wiggins, James Howison & Kevin Crowston 4 June, 2009
  2. 2. Introduction <ul><li>Success measures for FLOSS </li></ul><ul><ul><li>Internal versus external - success according to whom? </li></ul></ul><ul><li>Software usage is a desirable success measure, but difficult to obtain </li></ul><ul><li>Goal: Develop an algorithm to estimate active user base and general interest based on download counts </li></ul>
  3. 3. Measuring Software Use <ul><li>Many ways to measure usage </li></ul><ul><ul><li>Surveys </li></ul></ul><ul><ul><li>Usage reporting agents </li></ul></ul><ul><ul><li>Mining online data (downloads) </li></ul></ul><ul><li>Downloads provide a proxy for usage </li></ul><ul><ul><li>Must get software before you can use it </li></ul></ul><ul><ul><li>Usually FLOSS software is downloaded, which can be counted </li></ul></ul>
  4. 4. Problems with Downloads <ul><li>Downloads often used as direct proxy for usage, but… </li></ul><ul><ul><li>Cannot indicate how many downloads “convert” to actual use </li></ul></ul><ul><ul><li>Regular users are counted multiple times due to release updates </li></ul></ul><ul><ul><li>Measures inflated by user experimentation </li></ul></ul><ul><ul><li>Only counts one distribution channel </li></ul></ul><ul><ul><li>Release rates vary, hard to compare </li></ul></ul>
  5. 5. Hypothesis Development <ul><li>Experience-based theory </li></ul><ul><ul><li>What is the experience of adopting FLOSS for end-user applications? </li></ul></ul><ul><ul><li>Try it out, adopt it, update it when notified </li></ul></ul><ul><li>H1: There is a relatively constant level of downloads by new users trying out the software </li></ul><ul><li>H2: Regular users respond relatively quickly to new releases </li></ul>
  6. 6. Idealized Release/Download Grey area: potential user downloads White areas: active user downloads Ideally, we would expect that… - experimentation rate is nearly constant, growing over time - active user base updates after release, growing over time
  7. 7. Data & Analysis <ul><li>Daily time series data on package downloads </li></ul><ul><ul><li>FLOSSmole </li></ul></ul><ul><ul><li>http://ossmole.sourceforge.net </li></ul></ul><ul><li>Release data for each package </li></ul><ul><ul><li>SRDA </li></ul></ul><ul><ul><li>http://zerlot.cse.nd.edu </li></ul></ul><ul><li>Analysis with Taverna </li></ul><ul><ul><li>http://taverna.sourceforge.net </li></ul></ul>
  8. 8. Descriptive Results - BibDesk <ul><li>Spikes following new releases </li></ul><ul><li>Cyclic weekly effects </li></ul><ul><li>“ Flat” periods between releases </li></ul><ul><li>Growth over time in both baseline and spikes </li></ul>
  9. 9. Descriptive Results - SkimApp <ul><li>Similar overall patterns </li></ul><ul><li>Recently founded, less data </li></ul><ul><li>More rapid release cycle than BibDesk </li></ul><ul><li>In both projects, occasional non-release spikes appear - one-time marketing? </li></ul>
  10. 10. Quantifying User Base <ul><li>Calculations based on daily downloads for two one-week observation periods centered around release date </li></ul><ul><li>Potential user base : sum of daily downloads before release </li></ul><ul><li>Active user base : sum of daily downloads after release, less the baseline average download rate </li></ul>
  11. 11. Numerical Results - BibDesk <ul><li>Consistent baseline experimentation rate </li></ul><ul><li>Large variance for installed user base </li></ul><ul><ul><li>Further smoothing might help </li></ul></ul><ul><li>User base may be declining in BibDesk, due to small target audience and competition </li></ul>
  12. 12. Numerical Results - Skim-app <ul><li>Stable baseline, but substantial variance in calculated installed base </li></ul><ul><ul><li>Big spike in April 2008: first release in 3 months </li></ul></ul><ul><li>Overall trends toward growth in both user base and baseline </li></ul>
  13. 13. Discussion - Limitations <ul><li>Download data are problematic for a number of reasons </li></ul><ul><li>Calibrating the measures </li></ul><ul><ul><li>Varying the duration of time periods leads to substantial changes </li></ul></ul><ul><ul><li>User response rate varies by project </li></ul></ul><ul><ul><li>Very sensitive to release date accuracy </li></ul></ul><ul><ul><li>Also difficult to sample releases with sufficient time in between for baselines </li></ul></ul>
  14. 14. Discussion - Uses <ul><li>Generalizability </li></ul><ul><ul><li>Assumes swift user response </li></ul></ul><ul><ul><li>Different cases for end user versus enterprise software, varying market sizes </li></ul></ul><ul><li>Use with caution </li></ul><ul><ul><li>Examine data for consistent release response patterns </li></ul></ul><ul><ul><li>Either measure can serve as a dependent variable for project popularity </li></ul></ul>
  15. 15. Future Work <ul><li>Compare these findings against more dynamically selected time ranges </li></ul><ul><ul><li>e.g. time required to return to a rate close to the pre-release baseline </li></ul></ul><ul><li>Application to more projects, and comparison against other measures </li></ul><ul><li>Statistical fitting for growth estimates </li></ul><ul><li>May apply to other non-FLOSS downloaded software, e.g. iPhone apps </li></ul>
  16. 16. Conclusions <ul><li>Introduced a measure for estimating baseline user interest, and one for active user base in FLOSS projects </li></ul><ul><li>Baseline measure shows good face validity in longitudinal time series </li></ul><ul><li>Active user base measure shows surprising variance </li></ul>
  17. 17. Thanks! <ul><li>Questions? </li></ul><ul><li>{awiggins|crowston}@syr.edu , [email_address] </li></ul><ul><li>floss.syr.edu </li></ul><ul><li>flosshub.org </li></ul><ul><li>Background image derived from photo by Vincent Kaczmarek, http://www.flickr.com/photos/kaczmarekvincent/3263200507/ </li></ul>