Your SlideShare is downloading. ×
Web QA Gaia/B2G/Firefox OS front-end automation
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Web QA Gaia/B2G/Firefox OS front-end automation


Published on

Quick overview of our infrastructure for, project genesis, and coverage of Firefox OS (Gaia) front-end UI automation using Marionette (Selenium-backed WebDriver) in Python

Quick overview of our infrastructure for, project genesis, and coverage of Firefox OS (Gaia) front-end UI automation using Marionette (Selenium-backed WebDriver) in Python

Published in: Technology

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. [B2G | Gaia | Firefox OS] Front-endAutomation in Python (Marionette) Stephen Donner Web QA March 20, 2013
  • 2. Overview• Genesis / Introduction• Infrastructure (hardware)• Software stack• Builds in Jenkins / Continuous Integration• Test coverage / Capabilities• By the numbers• Will it blen^H^H run? (demo!)• Contribute!• Questions?
  • 3. Genesis / IntroductionInitially, “just a B2G QA thing” sizes/z/in/photostream/
  • 4. Genesis / IntroductionHard to keep good secrets for too long! we expanded coverage + infra
  • 5. Infrastructure (hardware)7 Unagis, tethered via USB to 7 Mac Minis, running Ubuntu 12.04 LTS Fun facts: 1) Must have earbuds plugged in for radio tests to work 2) For the photo/video tests to work, lens/aperture must not be blocked or in the dark
  • 6. Software Stack• Marionette ( -testing server that sits in B2G • Selenium-backed WebDriver • marionette_client – Python abstraction layer for Marionette• Gaiatest ( - wrapper aroundPython unnitests that runs the tests• Gaia UI Tests – test framework comprising setUp, tearDown, APIwrappers, and the tests themselves
  • 7. Builds in Jenkins/Continuous Integration Web QA helps set up, scale, and provide upkeep for more than just our UI tests
  • 8. By the Numbers ● Started on November 11, 2012 ● 25 individual contributors [1] ● From Taipei, Romania, UK, Canada, USA, Beijing ● 89 [2] tests, consisting of: ● ~ 19 unit tests ● 80 UI tests ● ~1hour per build, including initial flash [1][2]
  • 9. Test coverage / Capabilities Sampling of coverage:• Dialer (make/receive calls, test IMEI #)• Contacts (call, delete, create, add photo to, send SMS to)• Send/receive SMS• Camera (add, delete photo/videos)• Clock (add/delete alarms)• Calendar (add event, assert todays date)• Browser (load pages over EDGE, Wi-Fi, bookmark, navigate back)• Play a YouTube video• Marketplace (search, install, load, delete apps)• (browse to and load/install an app)• Changing wallpaper• Radio (adding/remove station, dial frequency up/down)• Keyboard• Notification bar• and more...
  • 10. Test coverage / CapabilitiesCapabilities:• we push to device (media files – photos, contacts, etc.)• we also capture the app HTML, take screenshots on failure for post-test debugging, andstore the latest runs logcat (full, debug)• we test the default/empty states • in-between tests, we use the devicemanager API to iterate through and delete media files, delete all contacts, clear Wi-Fi networks, restore modified settings to default, set volume to 0• manifest.ini • gives us the ability to tell which tests to run, where • attributes: antenna, b2g, wifi, lan, camera, xfail, panda, sdcard, qemu, bluetooth • •
  • 11. Challenges• Two moving targets: B2G (Gaia/Gonk) + Marionette• Marionette-enabled builds are not identical to non-engineering builds • different online/offline handling (patch pending)• changes to how/when adb / remote debugging work in Gaia• memory leak(s)• feature work• performance refactoring• Event-model changes: • .click() → .tap()• async JavaScript – teams expertise is (still) largely Python• very dynamic, heavy HTML5 apps, with CSS3 usage – rotates, transforms, SVG – hard tofind and test location and state of the app• infrastructure set up • acquiring Unagi phones (and keeping them healthy) • SIMs (unlimited plans) • figuring out and maintaining a distributed, consistent format to store each phones info
  • 12. Will it blen^H^H^ (demo!)
  • 13. Contribute!• Repository: •• How to contribute: •• Open issues up for grabs! • direction=desc&labels=difficulty+beginner&sort=created&state=openIRC:• • #mozwebqa (core Gaia UI Tests) • #ateam (perf tests)
  • 14. Questions?