[B2G | Gaia | Firefox OS] Front-endAutomation in Python (Marionette) Stephen Donner Web QA March 20, 2013
Overview• Genesis / Introduction• Infrastructure (hardware)• Software stack• Builds in Jenkins / Continuous Integration• Test coverage / Capabilities• By the numbers• Will it blen^H^H run? (demo!)• Contribute!• Questions?
Genesis / IntroductionInitially, “just a B2G QA thing”http://www.flickr.com/photos/daniel_gies/5041121388/ sizes/z/in/photostream/
Genesis / IntroductionHard to keep good secrets for too long! ...so we expanded coverage + infra http://memegenerator.net/instance/27635541
Infrastructure (hardware)7 Unagis, tethered via USB to 7 Mac Minis, running Ubuntu 12.04 LTS Fun facts: 1) Must have earbuds plugged in for radio tests to work 2) For the photo/video tests to work, lens/aperture must not be blocked or in the dark
Software Stack• Marionette (https://developer.mozilla.org/en-US/docs/Marionette) -testing server that sits in B2G • Selenium-backed WebDriver • marionette_client – Python abstraction layer for Marionette• Gaiatest (https://pypi.python.org/pypi/gaiatest) - wrapper aroundPython unnitests that runs the tests• Gaia UI Tests – test framework comprising setUp, tearDown, APIwrappers, and the tests themselves
Builds in Jenkins/Continuous Integration Web QA helps set up, scale, and provide upkeep for more than just our UI testshttp://qa-selenium.mv.mozilla.com:8080/view/B2G/
By the Numbers ● Started on November 11, 2012 ● 25 individual contributors  ● From Taipei, Romania, UK, Canada, USA, Beijing ● 89  tests, consisting of: ● ~ 19 unit tests ● 80 UI tests ● ~1hour per build, including initial flash  https://github.com/mozilla/gaia-ui-tests/graphs http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.unagi.gaia.nightly.ui/lastBuild/
Test coverage / Capabilities Sampling of coverage:• Dialer (make/receive calls, test IMEI #)• Contacts (call, delete, create, add photo to, send SMS to)• Send/receive SMS• Camera (add, delete photo/videos)• Clock (add/delete alarms)• Calendar (add event, assert todays date)• Browser (load pages over EDGE, Wi-Fi, bookmark, navigate back)• Play a YouTube video• Marketplace (search, install, load, delete apps)• Everything.me (browse to and load/install an app)• Changing wallpaper• Radio (adding/remove station, dial frequency up/down)• Keyboard• Notification bar• and more...
Test coverage / CapabilitiesCapabilities:• we push to device (media files – photos, contacts, etc.)• we also capture the app HTML, take screenshots on failure for post-test debugging, andstore the latest runs logcat (full, debug)• we test the default/empty states • in-between tests, we use the devicemanager API to iterate through and delete media files, delete all contacts, clear Wi-Fi networks, restore modified settings to default, set volume to 0• manifest.ini • gives us the ability to tell which tests to run, where • attributes: antenna, b2g, wifi, lan, camera, xfail, panda, sdcard, qemu, bluetooth • •
Will it blen^H^H^ (demo!)https://www.youtube.com/watch?v=CC2ywYeP-4s
Contribute!• Repository: • https://github.com/mozilla/gaia-ui-tests• How to contribute: • https://github.com/mozilla/gaia-ui-tests/blob/master/CONTRIBUTING.md• Open issues up for grabs! • https://github.com/mozilla/gaia-ui-tests/issues? direction=desc&labels=difficulty+beginner&sort=created&state=openIRC:• • #mozwebqa (core Gaia UI Tests) • #ateam (perf tests)