Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Free QA!

14 views

Published on

Did you know that CPAN comes with a free QA team? CPAN Testers is a distributed, grass-roots project with over 6.5 million test reports. This talk describes how the project benefits Perl developers and offers four important practices for any large-scale, volunteer QA effort

Published in: Software
  • Be the first to comment

  • Be the first to like this

Free QA!

  1. 1. Free QA! David Golden dagolden@cpan.org www.dagolden.com OSCON ▪ July 22, 2010
  2. 2. Free as in beer Free as in speech
  3. 3. Quality + Assurance
  4. 4. Quality is an ideal
  5. 5. Quality Assurance is what you can measure
  6. 6. CPAN Testers is Free QA for CPAN The objective of the group is to test as many of the distributions on CPAN as possible, on as many platforms as possible. The ultimate goal is to improve the portability of the distributions on CPAN, and provide good feedback to the authors.
  7. 7. CPAN is 15 years young ~135,000 uploads from ~5000 authors ~23,000 unique libraries uploaded ~21,000 active libraries available ~1,000 uploads every month ~200 new libraries every month
  8. 8. CPAN Testers is 11 years young 7.8 million reports 1000+ CPAN Testers Peak month: 400,000+ reports
  9. 9. Learning from Crisis
  10. 10. Four ideas for success
  11. 11. Culture Fun Mashups Scale
  12. 12. Values Tools Culture
  13. 13. $ perl Makefile.PL $ make $ make test $ make install
  14. 14. $ perl Makefile.PL $ make $ make test $ make install
  15. 15. CPAN (1995) CPAN.pm (1996) $ cpan File::Spec (runs 'make test')
  16. 16. We built tests into our tools
  17. 17. Thousands of people running CPAN tests ... and throwing them away
  18. 18. CPAN Testers (1999) cpan-testers@perl.org Reports submitted "by hand"
  19. 19. CPANPLUS (2002) Can submit reports "automatically" when you test or install
  20. 20. A step change in participation 0 5 10 15 20 25 30 35 40 45 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 Reports/ Upload A. Tang, “Becoming a CPAN Tester with CPANPLUS” (2002)
  21. 21. We built tests and test reporting into our tools
  22. 22. But still just ~5 reports per upload 0 5 10 15 20 25 30 35 40 45 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 Reports/ Upload
  23. 23. Crisis #1: Lack of coverage CPAN Testers no longer a toy but not yet useful for much
  24. 24. Fun Community Competition
  25. 25. Step 1: Make it easy to join
  26. 26. CPAN::YACSmoke (2005) POE::Component::CPAN::YACSmoke (2006) CPAN::Reporter (2006)
  27. 27. Step 2: Build a community and give feedback
  28. 28. http://stats.cpantesters.org/ (2006) cpan-testers-discuss@perl.org (2007) Barbie starts a monthly blog (2007)
  29. 29. Competition
  30. 30. September, 2007 "The heat is on for Chris Williams, he only thought Dave Cantrell was chasing his heels, but Andreas Koenig has also stepped up his game too. It has been a staggering month for the sheer volume of reports..." - Barbie
  31. 31. Climb the leaderboard
  32. 32. Test more or weirder Perls and platforms
  33. 33. Perl version and OS coverage
  34. 34. The result of -Ofun 0 5 10 15 20 25 30 35 40 45 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 Reports/ Upload
  35. 35. Zooming out 0 50 100 150 200 250 300 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Reports/ Upload
  36. 36. … rise of the machines …
  37. 37. CPAN::Reporter::Smoker (2008) POE::Component::SmokeBox (2008) CPANPLUS::YACSmoke (2009) minismokebox (2009)
  38. 38. Reports by #1 tester each month 1 10 100 1,000 10,000 100,000 1,000,000 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Manual Reporting Automated Reporting Dedicated Smoking Smoke Clusters
  39. 39. More testers & reports per tester 0 50 100 150 200 250 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Number of Testers Reports per Tester Manual Reporting Automated Reporting Dedicated Smoking Smoke Clusters
  40. 40. Crisis #2: Spam ~15,000 reports a month (2006) ~250,000 reports a month (2008) By email
  41. 41. Cc: AUTHOR@cpan.orgJoe Tester To: cpan-testers@perl.org 2006: ~5 reports/upload
  42. 42. "Stop SPAMMING me!" 2008: ~120 reports/upload WTF?!?
  43. 43. NOT fun for CPAN authors Disabled author CC's Notification through a central service Added author contact preferences
  44. 44. Mashups Innovation Insight
  45. 45. Open data allows innovation
  46. 46. Test matrix matrix.cpantesters.org
  47. 47. Dependency tree deps.cpantesters.org
  48. 48. "Volatile 100" "FAIL 100"
  49. 49. Regression analysis and failure correlations o OS, architecture, perl version, perl configuration o test output, e.g. “failed at line …” o tester, test framework analysis.cpantesters.org – but offline! 
  50. 50. Crisis #3: Email overload Reports sent by email Reports re-distributed via NNTP
  51. 51. Success? Failure? 455,875 2,080 1,163 953 555 525 206 117 95 89 1 10 100 1000 10000 100000 1000000 perl.cpan.testers perl.cpan.uploads perl.perl5.porters perl.daily-build.reports perl.beginners perl.modules perl.perl5.changes perl.moose perl.module-authors perl.perl6.language November 2009 perl.org mailing list monthly volume
  52. 52. The way perl.org sees email perl.cpan.testers everything else 99%
  53. 53. November 2009: throttling back
  54. 54. Scale Anticipation Leverage
  55. 55. Scale is not always a technical problem!
  56. 56. Visualize wild success
  57. 57. But don't build for it right away
  58. 58. Email worked! NNTP worked! For 10 years
  59. 59. 1. Anticipate the breakpoint 2. Do something about it
  60. 60. FAIL We should have seen it coming sooner
  61. 61. We built 2.0 by designing an API for server components and implementing a version of it on AWS
  62. 62. Clouds give leverage
  63. 63. Technical scale Administrative scale
  64. 64. Volunteer time is more scarce than nominal cloud costs
  65. 65. CT 2.0 running in beta since April
  66. 66. Culture within values and tools Fun from community and competition Mashups spur innovation and insight Scale needs anticipation and leverage
  67. 67. Thank you for coming! Your QA task: Rate this talk, leave comments and tell me what could be better

×