Your SlideShare is downloading. ×
2010 GTAC Crowd Source Testing Mozilla Style
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

2010 GTAC Crowd Source Testing Mozilla Style

1,669
views

Published on

Presented at GTAC 2010 conference, Hyderabad, India

Presented at GTAC 2010 conference, Hyderabad, India


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,669
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Crowd Source Testing Mozilla Community Style Matt Evans Mozilla QA Director Wednesday, November 10, 2010
  • 2. about:talk • Background and Scale of Mozilla • Crowdsourcing: Crowd vs. Community • History of Mozilla Community testing • How Mozilla succeeds with Community testing Wednesday, November 10, 2010
  • 3. • I have been in the testing biz for most my career • 9 mos as QA Director @ Mozilla • Manage a team of ~30 Global Test Engineers about:me Wednesday, November 10, 2010
  • 4. about:mozilla Wednesday, November 10, 2010
  • 5. about:mozilla Mozilla’s Mission: is to promote openness, innovation, and opportunity on the web Wednesday, November 10, 2010
  • 6. I’m still getting my head around scale at Mozilla • The scale of information flow • The scale of the user base Wednesday, November 10, 2010
  • 7. Mozilla Firehose Wednesday, November 10, 2010
  • 8. Mozilla Firehose • Many sources of information • bugzilla (200+bugs/day) • forums (50+) • blogs (50+) • tweets (100s/day) • wikis (1000s pages) • irc (50+ channels) Wednesday, November 10, 2010
  • 9. Mozilla Scale 400+ Developers 10,000+ Nightly Testers 2,500,000 Beta Testers 400,000,000 Users 400,000,000 350,000,000 300,000,000 250,000,000 200,000,000 150,000,000 100,000,000 50,000,0000 Wednesday, November 10, 2010
  • 10. Story Time Wednesday, November 10, 2010
  • 11. Big decision: Ship or No-Ship to Beta A Recent Firefox product team meeting Wednesday, November 10, 2010
  • 12. Protector of the tester community Wednesday, November 10, 2010
  • 13. Wednesday, November 10, 2010
  • 14. Wednesday, November 10, 2010
  • 15. Get feedback from tester community Wednesday, November 10, 2010
  • 16. Wednesday, November 10, 2010
  • 17. We were both wrong right! Wednesday, November 10, 2010
  • 18. It was a big change for my perspective Wednesday, November 10, 2010
  • 19. Respecter of the Tester Community as QA Team Partner Wednesday, November 10, 2010
  • 20. We now work much better together Wednesday, November 10, 2010
  • 21. Crowd Sourcing: It involves a crowd working towards a common goal Wednesday, November 10, 2010
  • 22. What is new, the virtual crowd is enabled by internet technologies Wednesday, November 10, 2010
  • 23. Wednesday, November 10, 2010
  • 24. • Building a global community of testers ~30k+ • Stewardship of the test community is their secret sauce Community approach to software testing Wednesday, November 10, 2010
  • 25. Mozilla Labs Crowdsource Crowdsourcing Project Wednesday, November 10, 2010
  • 26. Key Findings • there are two types users: heavy users and casual users. • 90-9-1 Principle • crowd motivation is separated into two forms: intrinsic and extrinsic • crowdsourcing relies on people, and thus works when participants are treated as people. Mozilla Labs Crowdsource Crowdsourcing Project Wednesday, November 10, 2010
  • 27. VS Community Crowd Wednesday, November 10, 2010
  • 28. Mozilla uses a hybrid of crowd based feedback mechanisms and community management “Crowd sourced testing Mozilla Style” Wednesday, November 10, 2010
  • 29. The History of Mozilla Testing Community Wednesday, November 10, 2010
  • 30. Mozilla.org is launched by Netscape 1998 2002 Mozilla 1.0 is released Phoenix 0.1 released 2004 Firefox 1.0 released 2005-2009 2010 Firefox 1.5-3.5 released Firefox 4.0 expected release Pioneer Community Testers Community Testers, The Next Generation 1 2 3 Wednesday, November 10, 2010
  • 31. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 1998 Netscape open sourced the Netscape browser source code. Wednesday, November 10, 2010
  • 32. History of Mozilla’s Tester Community •A lot of passionate people without a development background wanted to get involved. Asa Dotzler First Mozilla Community Tester Wednesday, November 10, 2010
  • 33. The Rise of the Nightly Build History of Mozilla’s Tester Community Wednesday, November 10, 2010
  • 34. History of Mozilla’s Tester Community •A great success •The community of testers grew to the 1000s Wednesday, November 10, 2010
  • 35. History of Mozilla’s Tester Community Too much of good thing Wednesday, November 10, 2010
  • 36. History of Mozilla’s Tester Community •As tester community members increased so did the noise Wednesday, November 10, 2010
  • 37. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 • 2002+ • The rise of Firefox code base • Split from AOL/Netscape • 2003 Mozilla Foundation is formed Wednesday, November 10, 2010
  • 38. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 2 Wednesday, November 10, 2010
  • 39. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 • 2007+ • The rise of Continuous Integration and unit tests Wednesday, November 10, 2010
  • 40. Continuous Integration ~3.3 million automated tests executed for every committed code change •The largest CI system built on Buildbot and Tinderbox technologies •For every code change •11 full builds of the browser across supported platforms •~300k unit/perf tests per build Wednesday, November 10, 2010
  • 41. Minefield is the result of Continuous Integration Continuous Integration and Unit Testing is the Foundation of Today’s Mozilla Community Testing Success Wednesday, November 10, 2010
  • 42. Community Feedback Wednesday, November 10, 2010
  • 43. Mozilla Runs On Bugzilla Wednesday, November 10, 2010
  • 44. Crash Data Analysis Wednesday, November 10, 2010
  • 45. Crash Data Analysis Wednesday, November 10, 2010
  • 46. Crash Data Analysis 5 years of crash data FF1.5 FF2.0 FF3.0 FF3.5 Filed Fixed Dups Wednesday, November 10, 2010
  • 47. Firefox 4 Beta Feedback Wednesday, November 10, 2010
  • 48. Firefox 4 Beta Feedback LCD Screens of Feedback and Twitter mentions are displayed throughout the Mozilla Office Wednesday, November 10, 2010
  • 49. Engaging the Mozilla Tester Community Wednesday, November 10, 2010
  • 50. Manual Testing Litmus - Test Case Management Wednesday, November 10, 2010
  • 51. Test and Bug Days Wednesday, November 10, 2010
  • 52. Mozmill Test Automation Framework Community Focused Automated Testing • Javascript framework for testing gecko based applications • Includes an IDE • Installs as an Firefox extension • Better suited for community contribution Wednesday, November 10, 2010
  • 53. Mozilla Community Based Automated Testing Wednesday, November 10, 2010
  • 54. Mozilla Community Based Automated Testing Wednesday, November 10, 2010
  • 55. Overview of How we succeed with Community Testing • Continuous Integration is the foundation of today’s community testing • Utilizing Community Feedback • Engage in Manual Testing efforts • Engage Community focused automated testing in the near future Wednesday, November 10, 2010
  • 56. Community Stewardship Wednesday, November 10, 2010
  • 57. Community Stewardship Wednesday, November 10, 2010
  • 58. Community Stewardship quality.mozilla.org Wednesday, November 10, 2010
  • 59. Reaching Out to the Community quality.mozilla.org quality.mozilla.org Wednesday, November 10, 2010
  • 60. A call out to you! Wednesday, November 10, 2010
  • 61. Thank you Questions? Wednesday, November 10, 2010