Web QA Gaia/B2G/Firefox OS front-end automation


Published on

Quick overview of our infrastructure for, project genesis, and coverage of Firefox OS (Gaia) front-end UI automation using Marionette (Selenium-backed WebDriver) in Python

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Web QA Gaia/B2G/Firefox OS front-end automation

  1. 1. [B2G | Gaia | Firefox OS] Front-endAutomation in Python (Marionette) Stephen Donner Web QA March 20, 2013
  2. 2. Overview• Genesis / Introduction• Infrastructure (hardware)• Software stack• Builds in Jenkins / Continuous Integration• Test coverage / Capabilities• By the numbers• Will it blen^H^H run? (demo!)• Contribute!• Questions?
  3. 3. Genesis / IntroductionInitially, “just a B2G QA thing”http://www.flickr.com/photos/daniel_gies/5041121388/ sizes/z/in/photostream/
  4. 4. Genesis / IntroductionHard to keep good secrets for too long! ...so we expanded coverage + infra http://memegenerator.net/instance/27635541
  5. 5. Infrastructure (hardware)7 Unagis, tethered via USB to 7 Mac Minis, running Ubuntu 12.04 LTS Fun facts: 1) Must have earbuds plugged in for radio tests to work 2) For the photo/video tests to work, lens/aperture must not be blocked or in the dark
  6. 6. Software Stack• Marionette (https://developer.mozilla.org/en-US/docs/Marionette) -testing server that sits in B2G • Selenium-backed WebDriver • marionette_client – Python abstraction layer for Marionette• Gaiatest (https://pypi.python.org/pypi/gaiatest) - wrapper aroundPython unnitests that runs the tests• Gaia UI Tests – test framework comprising setUp, tearDown, APIwrappers, and the tests themselves
  7. 7. Builds in Jenkins/Continuous Integration Web QA helps set up, scale, and provide upkeep for more than just our UI testshttp://qa-selenium.mv.mozilla.com:8080/view/B2G/
  8. 8. By the Numbers ● Started on November 11, 2012 ● 25 individual contributors [1] ● From Taipei, Romania, UK, Canada, USA, Beijing ● 89 [2] tests, consisting of: ● ~ 19 unit tests ● 80 UI tests ● ~1hour per build, including initial flash [1] https://github.com/mozilla/gaia-ui-tests/graphs[2] http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.unagi.gaia.nightly.ui/lastBuild/
  9. 9. Test coverage / Capabilities Sampling of coverage:• Dialer (make/receive calls, test IMEI #)• Contacts (call, delete, create, add photo to, send SMS to)• Send/receive SMS• Camera (add, delete photo/videos)• Clock (add/delete alarms)• Calendar (add event, assert todays date)• Browser (load pages over EDGE, Wi-Fi, bookmark, navigate back)• Play a YouTube video• Marketplace (search, install, load, delete apps)• Everything.me (browse to and load/install an app)• Changing wallpaper• Radio (adding/remove station, dial frequency up/down)• Keyboard• Notification bar• and more...
  10. 10. Test coverage / CapabilitiesCapabilities:• we push to device (media files – photos, contacts, etc.)• we also capture the app HTML, take screenshots on failure for post-test debugging, andstore the latest runs logcat (full, debug)• we test the default/empty states • in-between tests, we use the devicemanager API to iterate through and delete media files, delete all contacts, clear Wi-Fi networks, restore modified settings to default, set volume to 0• manifest.ini • gives us the ability to tell which tests to run, where • attributes: antenna, b2g, wifi, lan, camera, xfail, panda, sdcard, qemu, bluetooth • •
  11. 11. Challenges• Two moving targets: B2G (Gaia/Gonk) + Marionette• Marionette-enabled builds are not identical to non-engineering builds • different online/offline handling (patch pending)• changes to how/when adb / remote debugging work in Gaia• memory leak(s)• feature work• performance refactoring• Event-model changes: • .click() → .tap()• async JavaScript – teams expertise is (still) largely Python• very dynamic, heavy HTML5 apps, with CSS3 usage – rotates, transforms, SVG – hard tofind and test location and state of the app• infrastructure set up • acquiring Unagi phones (and keeping them healthy) • SIMs (unlimited plans) • figuring out and maintaining a distributed, consistent format to store each phones info
  12. 12. Will it blen^H^H^ (demo!)https://www.youtube.com/watch?v=CC2ywYeP-4s
  13. 13. Contribute!• Repository: • https://github.com/mozilla/gaia-ui-tests• How to contribute: • https://github.com/mozilla/gaia-ui-tests/blob/master/CONTRIBUTING.md• Open issues up for grabs! • https://github.com/mozilla/gaia-ui-tests/issues? direction=desc&labels=difficulty+beginner&sort=created&state=openIRC:• • #mozwebqa (core Gaia UI Tests) • #ateam (perf tests)
  14. 14. Questions?