Selenium2 and Jenkins: Almost pain-free UI Testing


Published on

Slides from the Philly.rb talk given on 11 Oct 2011.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Originally a ThoughtWorks project (hence the old docs are really awful). Really active community.  Lots of folks in the IRC channel willing to help. Most core devs are in Europe, so get on IRC early. Constantly, if not consistently, dropping new releases. Project is quite large, but well divided along functional and language lines.
  • Selenium server (hub and nodes) are java applications. They communicate using a JSON based wire protocol that's very well documented on the selenium project page. Some browsers require native clients in addition to the selenium grid. (Chrome, Android). There are defaults for setting up the grid but plenty of options for specifying the binary location of browsers, how many to run, etc. Can run multiple versions of the same browser on the same node with the same Selenium client.  
  • The WebDriver will talk to the grid hub to determine which node(s) to offer up. I *believe* the WebDriver then simply talks directly to the nodes without going through the hub.
  • Our existing QTP tests were mostly record-and-playback style tests.  We wanted more data-driven tests to alleviate the need to manually QA parts of the application. We were going to have to re-write the QTP tests anyway even if we stuck with it.
  • Show the project structure, highlight the fact that each part is distinct.
  • Show the Eclipse test configs to highlight switching between grid mode and standalone mode.
  • Once these builds are done: - Application is installed from the latest green build from our build automation server - Dataset is dropped, recreated, and migrated to latest version. - Selenium test code has itself been built and tested.
  • Use the Jenkins Xvnc plugin to get headless tests running.
  • Selenium2 and Jenkins: Almost pain-free UI Testing

    1. 1. Selenium 2 + Jenkins = (Almost) Pain-Free UI Test Automation
    2. 2. Selenium Background <ul><ul><li>Open source (Apache 2.0 License) </li></ul></ul><ul><ul><li>V1 = record-playback </li></ul></ul><ul><ul><li>V2 (aka WebDriver) = write tests with code </li></ul></ul><ul><ul><li>Vital Statistics: </li></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><ul><li> </li></ul></ul></ul><ul><ul><ul><li> #selenium </li></ul></ul></ul><ul><ul><ul><li>WebDriver API's for: java, ruby, python, .Net,  </li></ul></ul></ul><ul><ul><ul><li>Firefox, IE, Chrome, Opera, HTMLUnit, iPhone, Android </li></ul></ul></ul>
    3. 3. Selenium Grid <ul><ul><li>Run Selenium tests on a heterogeneous set of nodes: </li></ul></ul><ul><ul><ul><li>Different OS </li></ul></ul></ul><ul><ul><ul><li>Different browser </li></ul></ul></ul><ul><ul><ul><li>Different browser version (do these still matter?) </li></ul></ul></ul><ul><ul><li>Grid is super-easy to build: </li></ul></ul><ul><ul><ul><li>Run a hub instance </li></ul></ul></ul><ul><ul><ul><li>Run n number of node instances </li></ul></ul></ul><ul><ul><li>Just about everything is customizable for each leaf node: </li></ul></ul><ul><ul><ul><li>Host OS/Version </li></ul></ul></ul><ul><ul><ul><li>Browser type/version </li></ul></ul></ul><ul><ul><ul><li>Number of browsers to run simultaneously </li></ul></ul></ul>
    4. 4. Selenium Grid: The Picture
    5. 5. Selenium Grid: Our Journey <ul><ul><li>Started with QTP (formerly Mercury QuickTest Professional) </li></ul></ul><ul><ul><li>Breadth of coverage with QTP suite. </li></ul></ul><ul><ul><ul><li>  A lot of pages covered, but with only one or two permutations. </li></ul></ul></ul><ul><ul><li>Zero support from development in creating/maintaining tests. </li></ul></ul><ul><ul><li>Single-point-of-failure from a personnel view. </li></ul></ul>
    6. 6. Selenium vs QTP <ul><li>QTP: </li></ul><ul><ul><li>Pros: </li></ul></ul><ul><ul><ul><li>Already owned it, had tests running and infrastructure setup. </li></ul></ul></ul><ul><ul><li>Cons: </li></ul></ul><ul><ul><ul><li>Windows/IE only. </li></ul></ul></ul><ul><ul><ul><li>Data-driven tests rely on VBScript. </li></ul></ul></ul><ul><ul><ul><li>Heavy license and training expenses. </li></ul></ul></ul><ul><li>The big win for us was using existing developer knowledge to write tests in a language ecosystem we are already familiar with. </li></ul>
    7. 7. Selenium: Our Goals <ul><ul><li>Replace QTP as the de-facto UI testing tool in our organization. </li></ul></ul><ul><ul><li>Increase OS/browser coverage. </li></ul></ul><ul><ul><li>Completely data-driven tests. </li></ul></ul><ul><ul><ul><li>Most test failures should be fixed in Excel, not code. </li></ul></ul></ul><ul><ul><ul><li>Add new test cases using Excel, not code. </li></ul></ul></ul><ul><ul><li>Better reporting of failures than QTP. </li></ul></ul><ul><ul><li>Ability to run subsets of tests on-demand instead of waiting for scheduled runs. </li></ul></ul>
    8. 8. Selenium: Our Approach <ul><ul><li>Build out data files to drive tests </li></ul></ul><ul><ul><ul><li>Show example </li></ul></ul></ul><ul><ul><li>  Build out thin wrapper around WebDriver: </li></ul></ul><ul><ul><ul><li>Exception handling </li></ul></ul></ul><ul><ul><ul><li>Handling option lists, frames, etc. </li></ul></ul></ul><ul><ul><li>Build out library of helper classes: </li></ul></ul><ul><ul><ul><li>Handle Excel test data (Apache POI) </li></ul></ul></ul><ul><ul><ul><li>Custom error logging </li></ul></ul></ul><ul><ul><ul><li>Custom scraping code for specific UI element types </li></ul></ul></ul>
    9. 9. Selenium Grid: Our Approach <ul><ul><li>Entire grid runs on a single machine: </li></ul></ul><ul><ul><ul><li>Ubuntu 11.04 host runs the grid hub. </li></ul></ul></ul><ul><ul><ul><li>Multiple VM's run the grid nodes (Ubuntu and WinXP). </li></ul></ul></ul><ul><ul><ul><li>Expanding into external VM's to run Win7. </li></ul></ul></ul><ul><ul><li>Completely incremental </li></ul></ul><ul><ul><li>Ability to run same test inside the grid and outside the grid using different test parameters. </li></ul></ul><ul><ul><ul><li>selenium.grid = {true | false} </li></ul></ul></ul><ul><ul><ul><li>selenium.platform = {WINDOWS | LINUX | ...} </li></ul></ul></ul><ul><ul><ul><li> = {firefox | 'Internet Explorer' | ..} </li></ul></ul></ul><ul><ul><ul><li>selenium.grid.hub.url = .... </li></ul></ul></ul>
    10. 10. Jenkins: The Automators Automator <ul><ul><li>Created a series of 'infrastructure' builds to get the tests ready for testing: </li></ul></ul><ul><ul><ul><li>Full uninstall and reinstall of the application including full dataset migration. </li></ul></ul></ul><ul><ul><ul><li>Build the Selenium plumbing code and run all tests. </li></ul></ul></ul><ul><ul><ul><ul><li>Use static HTML of custom widgets for time and sanity reasons. </li></ul></ul></ul></ul><ul><ul><li>Determined early on that we wanted to report each browser/OS combination separately to make managing failures easier. </li></ul></ul>
    11. 11. Jenkins + Selenium <ul><ul><li>Create builds for each OS/Browser/Version combo: </li></ul></ul><ul><ul><ul><li>Creating the first one: PITA </li></ul></ul></ul><ul><ul><ul><li>Creating 2...n: EASY </li></ul></ul></ul>
    12. 12. Caveats <ul><ul><li>Use a common profile for browsers that have them (Firefox) </li></ul></ul><ul><ul><li>If your test runs slow off the grid, it'll be REALLY slow on the grid. </li></ul></ul><ul><ul><li>Using XPath selectors is slow (painfully slow for IE). </li></ul></ul><ul><ul><ul><li>Use either named selectors or CSS selectors for good performance. </li></ul></ul></ul><ul><ul><li>WebDriver throws exceptions for elements that it can't find. </li></ul></ul><ul><ul><li>Nothing's perfect: Understand UI testing is still part engineering and part magic and that unexplained failures will occur. </li></ul></ul>
    13. 13. Who the heck am I? <ul><li>Mike Reedell </li></ul><ul><li>[email_address] </li></ul><ul><li>@mikereedell </li></ul><ul><li> </li></ul><ul><li>Selenium Contributor* </li></ul>