Crowd Source Testing
Mozilla Community Style
Matt Evans
Mozilla QA Director
Wednesday, November 10, 2010
about:talk
• Background and Scale of Mozilla
• Crowdsourcing: Crowd vs. Community
• History of Mozilla Community testing
•...
• I have been in the testing biz for most my career
• 9 mos as QA Director @ Mozilla
• Manage a team of ~30 Global Test En...
about:mozilla
Wednesday, November 10, 2010
about:mozilla
Mozilla’s Mission:
is to promote openness,
innovation, and opportunity
on the web
Wednesday, November 10, 20...
I’m still getting my head around
scale at Mozilla
• The scale of information flow
• The scale of the user base
Wednesday, N...
Mozilla Firehose
Wednesday, November 10, 2010
Mozilla Firehose
• Many sources of information
• bugzilla (200+bugs/day)
• forums (50+)
• blogs (50+)
• tweets (100s/day)
...
Mozilla Scale
400+
Developers
10,000+
Nightly Testers
2,500,000
Beta Testers
400,000,000
Users
400,000,000
350,000,000
300...
Story Time
Wednesday, November 10, 2010
Big decision: Ship or
No-Ship to Beta
A Recent Firefox product
team meeting
Wednesday, November 10, 2010
Protector of the tester
community
Wednesday, November 10, 2010
Wednesday, November 10, 2010
Wednesday, November 10, 2010
Get feedback from
tester community
Wednesday, November 10, 2010
Wednesday, November 10, 2010
We were both wrong
right!
Wednesday, November 10, 2010
It was a big change for
my perspective
Wednesday, November 10, 2010
Respecter of the Tester
Community as QA
Team Partner
Wednesday, November 10, 2010
We now work much
better together
Wednesday, November 10, 2010
Crowd Sourcing: It involves a crowd
working towards a common goal
Wednesday, November 10, 2010
What is new, the virtual crowd is
enabled by internet technologies
Wednesday, November 10, 2010
Wednesday, November 10, 2010
• Building a global community of testers ~30k+
• Stewardship of the test community is their
secret sauce
Community approac...
Mozilla Labs Crowdsource
Crowdsourcing Project
Wednesday, November 10, 2010
Key Findings
• there are two types users: heavy
users and casual users.
• 90-9-1 Principle
• crowd motivation is separated...
VS
Community Crowd
Wednesday, November 10, 2010
Mozilla uses a hybrid of crowd based
feedback mechanisms and community
management
“Crowd sourced
testing Mozilla Style”
We...
The History of Mozilla
Testing Community
Wednesday, November 10, 2010
Mozilla.org is
launched
by Netscape
1998 2002
Mozilla 1.0
is released
Phoenix 0.1
released
2004
Firefox 1.0
released
2005-...
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3
1998 Netscape open so...
History of Mozilla’s Tester Community
•A lot of passionate people without a development
background wanted to get involved....
The Rise of the
Nightly Build
History of Mozilla’s Tester Community
Wednesday, November 10, 2010
History of Mozilla’s Tester Community
•A great success
•The community of testers
grew to the 1000s
Wednesday, November 10,...
History of Mozilla’s Tester Community
Too much of good thing
Wednesday, November 10, 2010
History of Mozilla’s Tester Community
•As tester community
members increased so did
the noise
Wednesday, November 10, 2010
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3
• 2002+
• The rise of...
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
2
Wednesday, November 10, 2...
1998 2002 2004 2005-2009 2010
Pioneer Community Testers
Community Testers, The Next Generation
1 2 3
• 2007+
• The rise of...
Continuous Integration
~3.3 million automated tests
executed for every committed
code change
•The largest CI system built
...
Minefield is the result of
Continuous Integration
Continuous Integration and Unit Testing
is the Foundation of Today’s Mozi...
Community Feedback
Wednesday, November 10, 2010
Mozilla Runs On
Bugzilla
Wednesday, November 10, 2010
Crash Data Analysis
Wednesday, November 10, 2010
Crash Data Analysis
Wednesday, November 10, 2010
Crash Data Analysis
5 years of crash data
FF1.5 FF2.0 FF3.0 FF3.5
Filed
Fixed
Dups
Wednesday, November 10, 2010
Firefox 4 Beta
Feedback
Wednesday, November 10, 2010
Firefox 4 Beta
Feedback
LCD Screens of Feedback and Twitter mentions are
displayed throughout the Mozilla Office
Wednesday,...
Engaging the Mozilla
Tester Community
Wednesday, November 10, 2010
Manual Testing
Litmus - Test Case Management
Wednesday, November 10, 2010
Test and Bug Days
Wednesday, November 10, 2010
Mozmill Test Automation Framework
Community Focused
Automated Testing
• Javascript framework for
testing gecko based
appli...
Mozilla Community Based Automated Testing
Wednesday, November 10, 2010
Mozilla Community Based Automated Testing
Wednesday, November 10, 2010
Overview of How we succeed
with Community Testing
• Continuous Integration is the foundation of
today’s community testing
...
Community
Stewardship
Wednesday, November 10, 2010
Community Stewardship
Wednesday, November 10, 2010
Community Stewardship
quality.mozilla.org
Wednesday, November 10, 2010
Reaching Out to the Community
quality.mozilla.org
quality.mozilla.org
Wednesday, November 10, 2010
A call out to you!
Wednesday, November 10, 2010
Thank you
Questions?
Wednesday, November 10, 2010
Upcoming SlideShare
Loading in...5
×

2010 GTAC Crowd Source Testing Mozilla Style

1,742

Published on

Presented at GTAC 2010 conference, Hyderabad, India

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,742
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

2010 GTAC Crowd Source Testing Mozilla Style

  1. 1. Crowd Source Testing Mozilla Community Style Matt Evans Mozilla QA Director Wednesday, November 10, 2010
  2. 2. about:talk • Background and Scale of Mozilla • Crowdsourcing: Crowd vs. Community • History of Mozilla Community testing • How Mozilla succeeds with Community testing Wednesday, November 10, 2010
  3. 3. • I have been in the testing biz for most my career • 9 mos as QA Director @ Mozilla • Manage a team of ~30 Global Test Engineers about:me Wednesday, November 10, 2010
  4. 4. about:mozilla Wednesday, November 10, 2010
  5. 5. about:mozilla Mozilla’s Mission: is to promote openness, innovation, and opportunity on the web Wednesday, November 10, 2010
  6. 6. I’m still getting my head around scale at Mozilla • The scale of information flow • The scale of the user base Wednesday, November 10, 2010
  7. 7. Mozilla Firehose Wednesday, November 10, 2010
  8. 8. Mozilla Firehose • Many sources of information • bugzilla (200+bugs/day) • forums (50+) • blogs (50+) • tweets (100s/day) • wikis (1000s pages) • irc (50+ channels) Wednesday, November 10, 2010
  9. 9. Mozilla Scale 400+ Developers 10,000+ Nightly Testers 2,500,000 Beta Testers 400,000,000 Users 400,000,000 350,000,000 300,000,000 250,000,000 200,000,000 150,000,000 100,000,000 50,000,0000 Wednesday, November 10, 2010
  10. 10. Story Time Wednesday, November 10, 2010
  11. 11. Big decision: Ship or No-Ship to Beta A Recent Firefox product team meeting Wednesday, November 10, 2010
  12. 12. Protector of the tester community Wednesday, November 10, 2010
  13. 13. Wednesday, November 10, 2010
  14. 14. Wednesday, November 10, 2010
  15. 15. Get feedback from tester community Wednesday, November 10, 2010
  16. 16. Wednesday, November 10, 2010
  17. 17. We were both wrong right! Wednesday, November 10, 2010
  18. 18. It was a big change for my perspective Wednesday, November 10, 2010
  19. 19. Respecter of the Tester Community as QA Team Partner Wednesday, November 10, 2010
  20. 20. We now work much better together Wednesday, November 10, 2010
  21. 21. Crowd Sourcing: It involves a crowd working towards a common goal Wednesday, November 10, 2010
  22. 22. What is new, the virtual crowd is enabled by internet technologies Wednesday, November 10, 2010
  23. 23. Wednesday, November 10, 2010
  24. 24. • Building a global community of testers ~30k+ • Stewardship of the test community is their secret sauce Community approach to software testing Wednesday, November 10, 2010
  25. 25. Mozilla Labs Crowdsource Crowdsourcing Project Wednesday, November 10, 2010
  26. 26. Key Findings • there are two types users: heavy users and casual users. • 90-9-1 Principle • crowd motivation is separated into two forms: intrinsic and extrinsic • crowdsourcing relies on people, and thus works when participants are treated as people. Mozilla Labs Crowdsource Crowdsourcing Project Wednesday, November 10, 2010
  27. 27. VS Community Crowd Wednesday, November 10, 2010
  28. 28. Mozilla uses a hybrid of crowd based feedback mechanisms and community management “Crowd sourced testing Mozilla Style” Wednesday, November 10, 2010
  29. 29. The History of Mozilla Testing Community Wednesday, November 10, 2010
  30. 30. Mozilla.org is launched by Netscape 1998 2002 Mozilla 1.0 is released Phoenix 0.1 released 2004 Firefox 1.0 released 2005-2009 2010 Firefox 1.5-3.5 released Firefox 4.0 expected release Pioneer Community Testers Community Testers, The Next Generation 1 2 3 Wednesday, November 10, 2010
  31. 31. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 1998 Netscape open sourced the Netscape browser source code. Wednesday, November 10, 2010
  32. 32. History of Mozilla’s Tester Community •A lot of passionate people without a development background wanted to get involved. Asa Dotzler First Mozilla Community Tester Wednesday, November 10, 2010
  33. 33. The Rise of the Nightly Build History of Mozilla’s Tester Community Wednesday, November 10, 2010
  34. 34. History of Mozilla’s Tester Community •A great success •The community of testers grew to the 1000s Wednesday, November 10, 2010
  35. 35. History of Mozilla’s Tester Community Too much of good thing Wednesday, November 10, 2010
  36. 36. History of Mozilla’s Tester Community •As tester community members increased so did the noise Wednesday, November 10, 2010
  37. 37. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 • 2002+ • The rise of Firefox code base • Split from AOL/Netscape • 2003 Mozilla Foundation is formed Wednesday, November 10, 2010
  38. 38. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 2 Wednesday, November 10, 2010
  39. 39. 1998 2002 2004 2005-2009 2010 Pioneer Community Testers Community Testers, The Next Generation 1 2 3 • 2007+ • The rise of Continuous Integration and unit tests Wednesday, November 10, 2010
  40. 40. Continuous Integration ~3.3 million automated tests executed for every committed code change •The largest CI system built on Buildbot and Tinderbox technologies •For every code change •11 full builds of the browser across supported platforms •~300k unit/perf tests per build Wednesday, November 10, 2010
  41. 41. Minefield is the result of Continuous Integration Continuous Integration and Unit Testing is the Foundation of Today’s Mozilla Community Testing Success Wednesday, November 10, 2010
  42. 42. Community Feedback Wednesday, November 10, 2010
  43. 43. Mozilla Runs On Bugzilla Wednesday, November 10, 2010
  44. 44. Crash Data Analysis Wednesday, November 10, 2010
  45. 45. Crash Data Analysis Wednesday, November 10, 2010
  46. 46. Crash Data Analysis 5 years of crash data FF1.5 FF2.0 FF3.0 FF3.5 Filed Fixed Dups Wednesday, November 10, 2010
  47. 47. Firefox 4 Beta Feedback Wednesday, November 10, 2010
  48. 48. Firefox 4 Beta Feedback LCD Screens of Feedback and Twitter mentions are displayed throughout the Mozilla Office Wednesday, November 10, 2010
  49. 49. Engaging the Mozilla Tester Community Wednesday, November 10, 2010
  50. 50. Manual Testing Litmus - Test Case Management Wednesday, November 10, 2010
  51. 51. Test and Bug Days Wednesday, November 10, 2010
  52. 52. Mozmill Test Automation Framework Community Focused Automated Testing • Javascript framework for testing gecko based applications • Includes an IDE • Installs as an Firefox extension • Better suited for community contribution Wednesday, November 10, 2010
  53. 53. Mozilla Community Based Automated Testing Wednesday, November 10, 2010
  54. 54. Mozilla Community Based Automated Testing Wednesday, November 10, 2010
  55. 55. Overview of How we succeed with Community Testing • Continuous Integration is the foundation of today’s community testing • Utilizing Community Feedback • Engage in Manual Testing efforts • Engage Community focused automated testing in the near future Wednesday, November 10, 2010
  56. 56. Community Stewardship Wednesday, November 10, 2010
  57. 57. Community Stewardship Wednesday, November 10, 2010
  58. 58. Community Stewardship quality.mozilla.org Wednesday, November 10, 2010
  59. 59. Reaching Out to the Community quality.mozilla.org quality.mozilla.org Wednesday, November 10, 2010
  60. 60. A call out to you! Wednesday, November 10, 2010
  61. 61. Thank you Questions? Wednesday, November 10, 2010
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×