Mozmill Crowd

quality.mozilla.org qa automation Mozmill CrowdA change in how we test Firefox Henrik Skupin <hskupin@mozilla.com> Lead of QA Test Automation

Overview• Release schedule for Firefox• Testing Firefox releases• How the Automation helps QA• Who contributes to QA?• Crowd-sourced Testing (Mozmill Crowd)• Demo

FIREFOX RELEASES

FIREFOX RELEASES Firefox 4.0 Beta • Lately approx. 2 weeks beta cycles • Various bug ﬁxes and improvements Firefox 3.5 and 3.6 • Security releases each 4 to 6 weeks • Both releases happen simultaneously • About 50 - 80 ﬁxed bugs per release

FIREFOX RELEASES

FIREFOX RELEASES Testing Firefox Releases • Exploratory and scripted testing of affected areas • Execution of functional tests, including: • approx. 40 Smoketests • approx. 330 BFT tests (Basic Functional Tests) • Time to ship to beta audience - only a few days • Time to ship the release - about 1 week

FIREFOX RELEASES

FIREFOX RELEASES Testing Firefox Releases • Software Update • Tests from previous versions (partial / complete) • Testing ‘betatest’ and ‘beta’ channels before beta • Testing ‘releasetest’ and ‘release’ channels before release

FIREFOX RELEASES

FIREFOX RELEASES How Automation Helps QA • Usage of Mozmill to run automated functional tests • approx. 110 (out of 330) functional tests automated • Execution of localization speciﬁc tests for all locales • (Nearly) fully automated update tests for all locales • Simultaneously execution on all supported platforms

CONTRIBUTING

CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches

CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?

CONTRIBUTING How can we outreach testing?

CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?

CROWD SOURCED TESTING Crowdsourcing relies on people, and thus works when participants are treated as people

CROWD SOURCED TESTING Crowd-sourced Testing Crowdsourcing relies on people, and thus works when participants are treated as people

CROWD SOURCED TESTING

CROWD SOURCED TESTING What do we expect? • Improved quality of localized builds • Knowledge how Firefox behaves in different locations • Results from machines with unknown conﬁgurations • Performance related information and memory usage • ...

CROWD SOURCED TESTING What’s necessary? • No global installation of additional software • Automatic setup of the Mozmill-CLI environment • Wide range of different test scenarios (BFT, L10n, ...) • Reporting and analysis of collected test results • Promoting tests even outside of our community

CROWD SOURCED TESTING How it works • Mozmill-Crowd add-on for Firefox 4.0 and 3.6 • Automatic setup of the prepared test environment • Simpliﬁed UI for the conﬁguration of the test-run • Automatic download of the latest tests • Test execution and reporting to our database

CROWD SOURCED TESTING User Proﬁle Testing Proﬁle 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment

CROWD SOURCED TESTING How it works User Proﬁle Testing Proﬁle 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment

CROWD SOURCED TESTING Analyzing test results L10n test results • Multiple access keys • Cropped elements • Screenshots

CROWD SOURCED TESTING What’s left • Proof reliability of our Mozmill tests • Fix focus problems with Mozmill (native events) • Offer more options in the extension for contribution • Add more test-runs (i.e. add-ons and endurance tests) • We need your feedback!

REFERENCES Demo

REFERENCES Mozmill Crowd: https://addons.mozilla.org/ﬁrefox/addon/mozmill-crowd QA Test Automation team page: http://quality.mozilla.org/teams/automation/ Mozmill Code: https://github.com/mozautomation/mozmill Mozmill Tests: https://developer.mozilla.org/en/Mozmill_Tests

QUESTIONS & ANSWERS Any Questions?

Thanks

http://www.hskupin.info	200
https://www.hskupin.info	14

Mozmill Crowd

by Henrik Skupin on Feb 07, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 214

Statistics

Mozmill Crowd — Presentation Transcript