2010 GTAC Crowd Source Testing Mozilla Style

2,124 views
1,947 views

Published on

Presented at GTAC 2010 conference, Hyderabad, India

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,124
On SlideShare
0
From Embeds
0
Number of Embeds
21
Actions
Shares
0
Downloads
42
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

2010 GTAC Crowd Source Testing Mozilla Style

  1. 1. Crowd Source Testing Mozilla Community Style Matt Evans Mozilla QA Director Wednesday, November 10, 2010
  2. 2. about:talk • Background and Scale of Mozilla • Crowdsourcing: Crowd vs. Community • History of Mozilla Community testing • How Mozilla succeeds with Community testing Wednesday, November 10, 2010
  3. 3. • I have been in the testing biz for most my career • 9 mos as QA Director @ Mozilla • Manage a team of ~30 Global Test Engineers about:me Wednesday, November 10, 2010
  4. 4. about:mozilla Wednesday, November 10, 2010
  5. 5. about:mozilla Mozilla’s Mission: is to promote openness, innovation, and opportunity on the web Wednesday, November 10, 2010
  6. 6. I’m still getting my head around scale at Mozilla • The scale of information flow • The scale of the user base Wednesday, November 10, 2010
  7. 7. Mozilla Firehose Wednesday, November 10, 2010
  8. 8. Mozilla Firehose • Many sources of information • bugzilla (200+bugs/day) • forums (50+) • blogs (50+) • tweets (100s/day) • wikis (1000s pages) • irc (50+ channels) Wednesday, November 10, 2010
  9. 9. Mozilla Scale 400+ Developers 10,000+ Nightly Testers 2,500,000 Beta Testers 400,000,000 Users 400,000,000 350,000,000 300,000,000 250,000,000 200,000,000 150,000,000 100,000,000 50,000,0000 Wednesday, November 10, 2010
  10. 10. Story Time Wednesday, November 10, 2010
  11. 11. Big decision: Ship or No-Ship to Beta A Recent Firefox product team meeting Wednesday, November 10, 2010
  12. 12. Protector of the tester community Wednesday, November 10, 2010
  13. 13. Wednesday, November 10, 2010
  14. 14. Wednesday, November 10, 2010
  15. 15. Get feedback from tester community Wednesday, November 10, 2010
  16. 16. Wednesday, November 10, 2010
  17. 17. We were both wrong right! Wednesday, November 10, 2010
  18. 18. It was a big change for my perspective Wednesday, November 10, 2010
  19. 19. Respecter of the Tester Community as QA Team Partner Wednesday, November 10, 2010
  20. 20. We now work much better together Wednesday, November 10, 2010
  21. 21. Crowd Sourcing: It involves a crowd working towards a common goal Wednesday, November 10, 2010
  22. 22. What is new, the virtual crowd is enabled by internet technologies Wednesday, November 10, 2010
  23. 23. Wednesday, November 10, 2010
  24. 24. • Building a global community of testers ~30k+ • Stewardship of the test community is their secret sauce Community approach to software testing Wednesday, November 10, 2010
  25. 25. Mozilla Labs Crowdsource Crowdsourcing Project Wednesday, November 10, 2010
  26. 26. Key Findings • there are two types users: heavy users and casual users. • 90-9-1 Principle • crowd motivation is separated into two forms: intrinsic and extrinsic • crowdsourcing relies on people, and thus works when participants are treated as people. Mozilla Labs Crowdsource Crowdsourcing Project Wednesday, November 10, 2010
  27. 27. VS Community Crowd Wednesday, November 10, 2010
  28. 28. Mozilla uses a hybrid of crowd based feedback mechanisms and community management “Crowd sourced testing Mozilla Style” Wednesday, November 10, 2010
  29. 29. The History of Mozilla Testing Community Wednesday, November 10, 2010
  30. 30. Mozilla.org is launched by Netscape 1998 2002 Mozilla 1.0 is released Phoenix 0.1 released 2004 Firefox 1.0 released 2005-2009 2010 Firefox 1.5-3.5 released Firefox 4.0 expected release Pioneer Community Testers Community Testers, The Next Generation 1 2 3 Wednesday, November 10, 2010
  31. 31. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 1998 Netscape open sourced the Netscape browser source code. Wednesday, November 10, 2010
  32. 32. History of Mozilla’s Tester Community •A lot of passionate people without a development background wanted to get involved. Asa Dotzler First Mozilla Community Tester Wednesday, November 10, 2010
  33. 33. The Rise of the Nightly Build History of Mozilla’s Tester Community Wednesday, November 10, 2010
  34. 34. History of Mozilla’s Tester Community •A great success •The community of testers grew to the 1000s Wednesday, November 10, 2010
  35. 35. History of Mozilla’s Tester Community Too much of good thing Wednesday, November 10, 2010
  36. 36. History of Mozilla’s Tester Community •As tester community members increased so did the noise Wednesday, November 10, 2010
  37. 37. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 • 2002+ • The rise of Firefox code base • Split from AOL/Netscape • 2003 Mozilla Foundation is formed Wednesday, November 10, 2010
  38. 38. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 2 Wednesday, November 10, 2010
  39. 39. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 • 2007+ • The rise of Continuous Integration and unit tests Wednesday, November 10, 2010
  40. 40. Continuous Integration ~3.3 million automated tests executed for every committed code change •The largest CI system built on Buildbot and Tinderbox technologies •For every code change •11 full builds of the browser across supported platforms •~300k unit/perf tests per build Wednesday, November 10, 2010
  41. 41. Minefield is the result of Continuous Integration Continuous Integration and Unit Testing is the Foundation of Today’s Mozilla Community Testing Success Wednesday, November 10, 2010
  42. 42. Community Feedback Wednesday, November 10, 2010
  43. 43. Mozilla Runs On Bugzilla Wednesday, November 10, 2010
  44. 44. Crash Data Analysis Wednesday, November 10, 2010
  45. 45. Crash Data Analysis Wednesday, November 10, 2010
  46. 46. Crash Data Analysis 5 years of crash data FF1.5 FF2.0 FF3.0 FF3.5 Filed Fixed Dups Wednesday, November 10, 2010
  47. 47. Firefox 4 Beta Feedback Wednesday, November 10, 2010
  48. 48. Firefox 4 Beta Feedback LCD Screens of Feedback and Twitter mentions are displayed throughout the Mozilla Office Wednesday, November 10, 2010
  49. 49. Engaging the Mozilla Tester Community Wednesday, November 10, 2010
  50. 50. Manual Testing Litmus - Test Case Management Wednesday, November 10, 2010
  51. 51. Test and Bug Days Wednesday, November 10, 2010
  52. 52. Mozmill Test Automation Framework Community Focused Automated Testing • Javascript framework for testing gecko based applications • Includes an IDE • Installs as an Firefox extension • Better suited for community contribution Wednesday, November 10, 2010
  53. 53. Mozilla Community Based Automated Testing Wednesday, November 10, 2010
  54. 54. Mozilla Community Based Automated Testing Wednesday, November 10, 2010
  55. 55. Overview of How we succeed with Community Testing • Continuous Integration is the foundation of today’s community testing • Utilizing Community Feedback • Engage in Manual Testing efforts • Engage Community focused automated testing in the near future Wednesday, November 10, 2010
  56. 56. Community Stewardship Wednesday, November 10, 2010
  57. 57. Community Stewardship Wednesday, November 10, 2010
  58. 58. Community Stewardship quality.mozilla.org Wednesday, November 10, 2010
  59. 59. Reaching Out to the Community quality.mozilla.org quality.mozilla.org Wednesday, November 10, 2010
  60. 60. A call out to you! Wednesday, November 10, 2010
  61. 61. Thank you Questions? Wednesday, November 10, 2010

×