quality.mozilla.org   qa automation     Mozmill CrowdA change in how we test Firefox   Henrik Skupin <hskupin@mozilla.com>...
Overview• Release   schedule for Firefox• Testing   Firefox releases• How   the Automation helps QA• Who   contributes to ...
FIREFOX RELEASES
FIREFOX RELEASES                       Firefox 4.0 Beta   • Lately   approx. 2 weeks beta cycles   • Various    bug fixes a...
FIREFOX RELEASES
FIREFOX RELEASES                   Testing Firefox Releases   • Exploratory    and scripted testing of affected areas   • ...
FIREFOX RELEASES
FIREFOX RELEASES                   Testing Firefox Releases   • Software    Update     • Tests   from previous versions (p...
FIREFOX RELEASES
FIREFOX RELEASES                How Automation Helps QA   • Usage   of Mozmill to run automated functional tests   • appro...
CONTRIBUTING
CONTRIBUTING                 Who contributes to QA?   • Small   QA team (~30) and contractors for Releases   • Community  ...
CONTRIBUTING                 Who contributes to QA?   • Small   QA team (~30) and contractors for Releases   • Community  ...
CONTRIBUTING               How can we outreach testing?
CONTRIBUTING                 Who contributes to QA?   • Small   QA team (~30) and contractors for Releases   • Community  ...
CROWD SOURCED TESTING    Crowdsourcing relies on people, and thus works when            participants are treated as people
CROWD SOURCED TESTING               Crowd-sourced Testing    Crowdsourcing relies on people, and thus works when          ...
CROWD SOURCED TESTING
CROWD SOURCED TESTING                    What do we expect?   • Improved    quality of localized builds   • Knowledge    h...
CROWD SOURCED TESTING
CROWD SOURCED TESTING                     What’s necessary?   • No   global installation of additional software   • Automa...
CROWD SOURCED TESTING
CROWD SOURCED TESTING                        How it works   • Mozmill-Crowd     add-on for Firefox 4.0 and 3.6   • Automat...
CROWD SOURCED TESTING                      User Profile        Testing Profile                                    3.        ...
CROWD SOURCED TESTING                              How it works                      User Profile        Testing Profile    ...
CROWD SOURCED TESTING
CROWD SOURCED TESTING                  Analyzing test results   L10n test results    • Multiple   access keys    • Cropped...
CROWD SOURCED TESTING
CROWD SOURCED TESTING                           What’s left   • Proof   reliability of our Mozmill tests   • Fix   focus p...
REFERENCES              Demo
REFERENCES  Mozmill Crowd:  https://addons.mozilla.org/firefox/addon/mozmill-crowd  QA Test Automation team page:  http://q...
QUESTIONS & ANSWERS                      Any Questions?
Thanks
Mozmill Crowd
Upcoming SlideShare
Loading in...5
×

Mozmill Crowd

2,109

Published on

Mozilla QA has created a new feature called the Mozmill Crowd Extension that allows any Firefox user to automatically and effortlessly run our set of basic Mozmill functional test cases against their installed Firefox browser and report the results to our central reporting server. Any user in any locale can now participate in giving us valuable feedback on how their browser is operating in their environment. Mainly targeting the thousands of Firefox nightly testers, this extension will give Mozilla unprecedented insights on how a particular nightly build is working across the globe. In this session we will describe and demonstrate how the extension works, what data is collected from the tests and how the results are reported and analyzed.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,109
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Mozmill Crowd

    1. 1. quality.mozilla.org qa automation Mozmill CrowdA change in how we test Firefox Henrik Skupin <hskupin@mozilla.com> Lead of QA Test Automation
    2. 2. Overview• Release schedule for Firefox• Testing Firefox releases• How the Automation helps QA• Who contributes to QA?• Crowd-sourced Testing (Mozmill Crowd)• Demo
    3. 3. FIREFOX RELEASES
    4. 4. FIREFOX RELEASES Firefox 4.0 Beta • Lately approx. 2 weeks beta cycles • Various bug fixes and improvements Firefox 3.5 and 3.6 • Security releases each 4 to 6 weeks • Both releases happen simultaneously • About 50 - 80 fixed bugs per release
    5. 5. FIREFOX RELEASES
    6. 6. FIREFOX RELEASES Testing Firefox Releases • Exploratory and scripted testing of affected areas • Execution of functional tests, including: • approx. 40 Smoketests • approx. 330 BFT tests (Basic Functional Tests) • Time to ship to beta audience - only a few days • Time to ship the release - about 1 week
    7. 7. FIREFOX RELEASES
    8. 8. FIREFOX RELEASES Testing Firefox Releases • Software Update • Tests from previous versions (partial / complete) • Testing ‘betatest’ and ‘beta’ channels before beta • Testing ‘releasetest’ and ‘release’ channels before release
    9. 9. FIREFOX RELEASES
    10. 10. FIREFOX RELEASES How Automation Helps QA • Usage of Mozmill to run automated functional tests • approx. 110 (out of 330) functional tests automated • Execution of localization specific tests for all locales • (Nearly) fully automated update tests for all locales • Simultaneously execution on all supported platforms
    11. 11. CONTRIBUTING
    12. 12. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches
    13. 13. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?
    14. 14. CONTRIBUTING How can we outreach testing?
    15. 15. CONTRIBUTING Who contributes to QA? • Small QA team (~30) and contractors for Releases • Community members execute manual tests on Litmus • 80.000 nightly users for Firefox 4.0 builds • 15.000 nightly users for older Firefox branches How can we outreach testing?
    16. 16. CROWD SOURCED TESTING Crowdsourcing relies on people, and thus works when participants are treated as people
    17. 17. CROWD SOURCED TESTING Crowd-sourced Testing Crowdsourcing relies on people, and thus works when participants are treated as people
    18. 18. CROWD SOURCED TESTING
    19. 19. CROWD SOURCED TESTING What do we expect? • Improved quality of localized builds • Knowledge how Firefox behaves in different locations • Results from machines with unknown configurations • Performance related information and memory usage • ...
    20. 20. CROWD SOURCED TESTING
    21. 21. CROWD SOURCED TESTING What’s necessary? • No global installation of additional software • Automatic setup of the Mozmill-CLI environment • Wide range of different test scenarios (BFT, L10n, ...) • Reporting and analysis of collected test results • Promoting tests even outside of our community
    22. 22. CROWD SOURCED TESTING
    23. 23. CROWD SOURCED TESTING How it works • Mozmill-Crowd add-on for Firefox 4.0 and 3.6 • Automatic setup of the prepared test environment • Simplified UI for the configuration of the test-run • Automatic download of the latest tests • Test execution and reporting to our database
    24. 24. CROWD SOURCED TESTING User Profile Testing Profile 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment
    25. 25. CROWD SOURCED TESTING How it works User Profile Testing Profile 3. 1. Repositories 2. (Tests and Scripts) MozMill CLI 4. Database Virtual Test Environment
    26. 26. CROWD SOURCED TESTING
    27. 27. CROWD SOURCED TESTING Analyzing test results L10n test results • Multiple access keys • Cropped elements • Screenshots
    28. 28. CROWD SOURCED TESTING
    29. 29. CROWD SOURCED TESTING What’s left • Proof reliability of our Mozmill tests • Fix focus problems with Mozmill (native events) • Offer more options in the extension for contribution • Add more test-runs (i.e. add-ons and endurance tests) • We need your feedback!
    30. 30. REFERENCES Demo
    31. 31. REFERENCES Mozmill Crowd: https://addons.mozilla.org/firefox/addon/mozmill-crowd QA Test Automation team page: http://quality.mozilla.org/teams/automation/ Mozmill Code: https://github.com/mozautomation/mozmill Mozmill Tests: https://developer.mozilla.org/en/Mozmill_Tests
    32. 32. QUESTIONS & ANSWERS Any Questions?
    33. 33. Thanks
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×