This document discusses using Selenium 2 and Jenkins together for automated UI testing. It summarizes Selenium's capabilities for cross-browser testing using WebDriver APIs. It then describes how the author set up Selenium Grid for running tests across different OSes and browsers on a single machine. Finally, it explains how Jenkins was used to automate and report on test runs for each browser/OS combination on the Grid.

1. Selenium 2 + Jenkins = (Almost) Pain-Free UI Test Automation

2. Selenium Background Open source (Apache 2.0 License) V1 = record-playback V2 (aka WebDriver) = write tests with code Vital Statistics: seleniumhq.org code.google.com/p/selenium irc.freenode.net #selenium WebDriver API's for: java, ruby, python, .Net, Firefox, IE, Chrome, Opera, HTMLUnit, iPhone, Android

3. Selenium Grid Run Selenium tests on a heterogeneous set of nodes: Different OS Different browser Different browser version (do these still matter?) Grid is super-easy to build: Run a hub instance Run n number of node instances Just about everything is customizable for each leaf node: Host OS/Version Browser type/version Number of browsers to run simultaneously

4. Selenium Grid: The Picture

5. Selenium Grid: Our Journey Started with QTP (formerly Mercury QuickTest Professional) Breadth of coverage with QTP suite. A lot of pages covered, but with only one or two permutations. Zero support from development in creating/maintaining tests. Single-point-of-failure from a personnel view.

6. Selenium vs QTP QTP: Pros: Already owned it, had tests running and infrastructure setup. Cons: Windows/IE only. Data-driven tests rely on VBScript. Heavy license and training expenses. The big win for us was using existing developer knowledge to write tests in a language ecosystem we are already familiar with.

7. Selenium: Our Goals Replace QTP as the de-facto UI testing tool in our organization. Increase OS/browser coverage. Completely data-driven tests. Most test failures should be fixed in Excel, not code. Add new test cases using Excel, not code. Better reporting of failures than QTP. Ability to run subsets of tests on-demand instead of waiting for scheduled runs.

8. Selenium: Our Approach Build out data files to drive tests Show example Build out thin wrapper around WebDriver: Exception handling Handling option lists, frames, etc. Build out library of helper classes: Handle Excel test data (Apache POI) Custom error logging Custom scraping code for specific UI element types

9. Selenium Grid: Our Approach Entire grid runs on a single machine: Ubuntu 11.04 host runs the grid hub. Multiple VM's run the grid nodes (Ubuntu and WinXP). Expanding into external VM's to run Win7. Completely incremental Ability to run same test inside the grid and outside the grid using different test parameters. selenium.grid = {true | false} selenium.platform = {WINDOWS | LINUX | ...} selenium.browser.name = {firefox | 'Internet Explorer' | ..} selenium.grid.hub.url = ....

10. Jenkins: The Automators Automator Created a series of 'infrastructure' builds to get the tests ready for testing: Full uninstall and reinstall of the application including full dataset migration. Build the Selenium plumbing code and run all tests. Use static HTML of custom widgets for time and sanity reasons. Determined early on that we wanted to report each browser/OS combination separately to make managing failures easier.

11. Jenkins + Selenium Create builds for each OS/Browser/Version combo: Creating the first one: PITA Creating 2...n: EASY

12. Caveats Use a common profile for browsers that have them (Firefox) If your test runs slow off the grid, it'll be REALLY slow on the grid. Using XPath selectors is slow (painfully slow for IE). Use either named selectors or CSS selectors for good performance. WebDriver throws exceptions for elements that it can't find. Nothing's perfect: Understand UI testing is still part engineering and part magic and that unexplained failures will occur.

13. Who the heck am I? Mike Reedell [email_address] @mikereedell mikereedell.com Selenium Contributor*