This document discusses improving automated web application testing. It begins with an introduction and agenda. Some key points:
- Testing at Google expects test code quality to match production code quality. Tests must be maintainable and meet engineering standards.
- The present focuses on Selenium/WebDriver automation best practices and things to avoid. Base classes and locator organization can help improve test robustness and maintainability.
- The future may include more exploratory testing tools like WebTestingExplorer for automated exploratory testing.
Overall the document provides guidance on using Selenium/WebDriver for web application testing in a best practices way, focusing on test code quality, maintainability, and anticipating future changes. It emphasizes organizing tests through
This talk was given at "The KISS Method and PageObjects" meetup (https://goo.gl/2DBuxj).
The talk is about KISS (Keep It Simple Stupid) principle applied to PageObjects design with a comparison to other more classic approaches.
The slides contain both Selenium Webdriver and Selenide examples.
src code for the talk: https://github.com/yashaka/talks/tree/master/kiss-pageobjects
Three Simple Chords of Alternative PageObjects and Hardcore of LoadableCompon...Iakiv Kramarenko
***VIDEO***: view in SD at http://youtu.be/HPHKeBakulQ or download in HD at http://bit.ly/1nyvA67
Often we have lack of automation resources. If we just would involve less experienced juniors to implement test model and even Manual QA to write DSL like tests…
In this talk I want to present the simplified approach to write PageObjects for your test model as it would be like playing “three chords” song on a guitar. And also share the experience of pacifying the LoadableComponent pattern, rather hard in implementation but making your tests much more DRY and easy to use in context of loading pages.
Automation Abstractions: Page Objects and BeyondTechWell
When you start writing automation for your projects, you quickly realize that you need to organize and design the code. You will write far more than “test” code; you also will write abstraction code because you want to make tests easier to read and maintain. But how do you design all this code? How do you organize and structure it? Should you use a domain specific language? Should you go keyword driven or use Gherkin? Should you use page objects with POJO or Factories? Do you create DOM level abstractions? Where do domain models fit in? Alan Richardson provides an overview of options available to you when modeling abstraction layers. Based on his experience with many approaches on real-world commercial projects, Alan helps you understand how to think about the modeling of abstraction layers. Illustrated with a number of code examples, Alan shows you a variety of approaches and discusses the pros and cons associated with each.
A journey beyond the page object patternRiverGlide
A story of one organisation taking an idea, formed by Antony Marcano and evolved by many, to solve some of the challenges that encountered with the growth of Selenium/WebDriver PageObjects in their Automated Tests. Talk co-presented with two of the people who have been a part of the journey – including Kostas Mamalis & Jan Molak.
BDD (Behaviour Driven Development) is a synthesis and refinement of practices stemming from TDD (Test Driven Development) and ATDD (Acceptance Test Driven Development). Although BDD is principally an idea about how software development should be managed by both business interests and technical insight, the practice of BDD does assume the use of specialized software tools to support the development process.
This talk was given at "The KISS Method and PageObjects" meetup (https://goo.gl/2DBuxj).
The talk is about KISS (Keep It Simple Stupid) principle applied to PageObjects design with a comparison to other more classic approaches.
The slides contain both Selenium Webdriver and Selenide examples.
src code for the talk: https://github.com/yashaka/talks/tree/master/kiss-pageobjects
Three Simple Chords of Alternative PageObjects and Hardcore of LoadableCompon...Iakiv Kramarenko
***VIDEO***: view in SD at http://youtu.be/HPHKeBakulQ or download in HD at http://bit.ly/1nyvA67
Often we have lack of automation resources. If we just would involve less experienced juniors to implement test model and even Manual QA to write DSL like tests…
In this talk I want to present the simplified approach to write PageObjects for your test model as it would be like playing “three chords” song on a guitar. And also share the experience of pacifying the LoadableComponent pattern, rather hard in implementation but making your tests much more DRY and easy to use in context of loading pages.
Automation Abstractions: Page Objects and BeyondTechWell
When you start writing automation for your projects, you quickly realize that you need to organize and design the code. You will write far more than “test” code; you also will write abstraction code because you want to make tests easier to read and maintain. But how do you design all this code? How do you organize and structure it? Should you use a domain specific language? Should you go keyword driven or use Gherkin? Should you use page objects with POJO or Factories? Do you create DOM level abstractions? Where do domain models fit in? Alan Richardson provides an overview of options available to you when modeling abstraction layers. Based on his experience with many approaches on real-world commercial projects, Alan helps you understand how to think about the modeling of abstraction layers. Illustrated with a number of code examples, Alan shows you a variety of approaches and discusses the pros and cons associated with each.
A journey beyond the page object patternRiverGlide
A story of one organisation taking an idea, formed by Antony Marcano and evolved by many, to solve some of the challenges that encountered with the growth of Selenium/WebDriver PageObjects in their Automated Tests. Talk co-presented with two of the people who have been a part of the journey – including Kostas Mamalis & Jan Molak.
BDD (Behaviour Driven Development) is a synthesis and refinement of practices stemming from TDD (Test Driven Development) and ATDD (Acceptance Test Driven Development). Although BDD is principally an idea about how software development should be managed by both business interests and technical insight, the practice of BDD does assume the use of specialized software tools to support the development process.
Selenide is simple and powerful in use wrapper-library over Selenium. But what the point just of shorter lines of code? In this talk we will see how to tame your webui mustang with Selenide and put it into fence of simple BDD stories with Easyb. We also consider pros and cons of the approach and compare to available alternatives.
Automation Abstraction Layers: Page Objects and BeyondAlan Richardson
An overview of different approaches to Page Objects and Domain Objects for GUI Automation. Examples use WebDriver and Java.
Full source code is available to download, For more details see the associated blog post.
A set of slides that you can use to 'cheatsheet' your way to Selenium 2 WebDriver mastery. Formatted as the handouts I use on my Selenium WebDriver training.
Selenide Alternative in Practice - Implementation & Lessons learned [Selenium...Iakiv Kramarenko
Talk given on SeleniumCamp 2016 about:
- Main Selenide Features from Implementation point of view
- Examples of porting these features to python (using simplified Selene sources)
This talk can be considered as a HOWTO on porting Selenide to any language, if the following programming paradigms are considered as tools: Procedural, Modular, OOP.
On May 14, 2015, Jeff Winkler gave a talk at Harvard University's Lamont Library titled "Intro to ReactJS."
Description
Created by Facebook and Instagram, React has recently been embraced by companies and organizations including Airbnb, Khan Academy, Reddit, the BBC, and Code Academy. This presentation will be especially interesting for those using or planning to use javascript libraries such as angularJS, backbone.js, ember.js, and others.
For this talk, Jeff Winkler will present:
- An introduction to React, mixed with demos.
- An examination of how React implements Computer Science principles from Functional and Object-Oriented. The discussion will consider the impact on maintainable large-scale systems.
Biography
Jeff Winkler, is a professional developer* in the Boston area and organizer of the Boston ReactJS Meetup. In addition to the ReactJS Boston Meetup, Jeff works with React professionally at TapJoy and runs http://react.rocks.
(* Full-stack guy. ReactJS, Rails, TDD. Best OODA loop wins)
This talk was given at "Easy Automation in Python" meetup (https://www.meetup.com/Test-Fanatic-MeetUp/events/236481850/).
The talk is about some easy approaches in building web ui automation in python.
As an "easy tool" the selene library is uses.
src code for the talk: https://github.com/yashaka/talks/tree/master/easy-automation-py
Geb is a browser automation solution.
Geb uses the power of Selenium WebDriver and more.
Geb + Spock API simplifies test automation using Given-When-Then Concept.
ForwardJS 2017 - Fullstack end-to-end Test Automation with node.jsMek Srunyu Stittri
Slide deck for ForwardJS 2017 in San Francisco - March 1st 2017
https://forwardjs.com/schedule#lecture-224
Airware builds hardware, software and cloud for commercial drones. We have transitioned to Node.js for cloud functional test automation in 2015. The purpose of this is to unite Fullstack developers and Automation engineers to speak in the same language which is JavaScript. With a year worth of lessons learnt, we will share the challenges involved with building a full-stack test automation framework with Node.js while using the latest and greatest in JavaScript tools.
Page Object Model and Implementation in Selenium Zoe Gilbert
There are various frameworks that have been used by the IT industry for robust, easy maintenance, and to automate their applications to make them flexible and reusable. Page Object Model is one of the frameworks that have been covered here in this blog step by step clearly to help you understand.
WinAppDriver - Windows Store Apps Test AutomationJeremy Kao
Introduce the background and initial implementation of WinAppDriver - WebDriver for Windows Applications, including desktop applications, store apps (formerly known as Metro-style apps).
Selenide is simple and powerful in use wrapper-library over Selenium. But what the point just of shorter lines of code? In this talk we will see how to tame your webui mustang with Selenide and put it into fence of simple BDD stories with Easyb. We also consider pros and cons of the approach and compare to available alternatives.
Automation Abstraction Layers: Page Objects and BeyondAlan Richardson
An overview of different approaches to Page Objects and Domain Objects for GUI Automation. Examples use WebDriver and Java.
Full source code is available to download, For more details see the associated blog post.
A set of slides that you can use to 'cheatsheet' your way to Selenium 2 WebDriver mastery. Formatted as the handouts I use on my Selenium WebDriver training.
Selenide Alternative in Practice - Implementation & Lessons learned [Selenium...Iakiv Kramarenko
Talk given on SeleniumCamp 2016 about:
- Main Selenide Features from Implementation point of view
- Examples of porting these features to python (using simplified Selene sources)
This talk can be considered as a HOWTO on porting Selenide to any language, if the following programming paradigms are considered as tools: Procedural, Modular, OOP.
On May 14, 2015, Jeff Winkler gave a talk at Harvard University's Lamont Library titled "Intro to ReactJS."
Description
Created by Facebook and Instagram, React has recently been embraced by companies and organizations including Airbnb, Khan Academy, Reddit, the BBC, and Code Academy. This presentation will be especially interesting for those using or planning to use javascript libraries such as angularJS, backbone.js, ember.js, and others.
For this talk, Jeff Winkler will present:
- An introduction to React, mixed with demos.
- An examination of how React implements Computer Science principles from Functional and Object-Oriented. The discussion will consider the impact on maintainable large-scale systems.
Biography
Jeff Winkler, is a professional developer* in the Boston area and organizer of the Boston ReactJS Meetup. In addition to the ReactJS Boston Meetup, Jeff works with React professionally at TapJoy and runs http://react.rocks.
(* Full-stack guy. ReactJS, Rails, TDD. Best OODA loop wins)
This talk was given at "Easy Automation in Python" meetup (https://www.meetup.com/Test-Fanatic-MeetUp/events/236481850/).
The talk is about some easy approaches in building web ui automation in python.
As an "easy tool" the selene library is uses.
src code for the talk: https://github.com/yashaka/talks/tree/master/easy-automation-py
Geb is a browser automation solution.
Geb uses the power of Selenium WebDriver and more.
Geb + Spock API simplifies test automation using Given-When-Then Concept.
ForwardJS 2017 - Fullstack end-to-end Test Automation with node.jsMek Srunyu Stittri
Slide deck for ForwardJS 2017 in San Francisco - March 1st 2017
https://forwardjs.com/schedule#lecture-224
Airware builds hardware, software and cloud for commercial drones. We have transitioned to Node.js for cloud functional test automation in 2015. The purpose of this is to unite Fullstack developers and Automation engineers to speak in the same language which is JavaScript. With a year worth of lessons learnt, we will share the challenges involved with building a full-stack test automation framework with Node.js while using the latest and greatest in JavaScript tools.
Page Object Model and Implementation in Selenium Zoe Gilbert
There are various frameworks that have been used by the IT industry for robust, easy maintenance, and to automate their applications to make them flexible and reusable. Page Object Model is one of the frameworks that have been covered here in this blog step by step clearly to help you understand.
WinAppDriver - Windows Store Apps Test AutomationJeremy Kao
Introduce the background and initial implementation of WinAppDriver - WebDriver for Windows Applications, including desktop applications, store apps (formerly known as Metro-style apps).
Mastering Test Automation: How To Use Selenium SuccessfullySpringPeople
In this slide, identify what to test and choose the best language for automation. Learn to write maintainable and reusable Selenium tests and add UI layout test as part of automation using Galen framework. This slide will also guide you in reporting structure by using external plugin's, an illustration covering cross browser testing (Running selenium grid with Docker) and explain Code repository (Git) and Jenkins CI tool.
Split into two parts, the first one is basics in web user interfaces, the second one - protractor/jasmine testings (structure, set configs, spec and test example).
Javascript and first-class citizenry: require.js & node.js
Javascript on web pages is ubiquitous and its problems are legendary. Javascript, seen as a second-class code citizen, is usually hacked together even by seasoned developers. New libraries (jQuery, prototype, backbone, knockout, underscore) and runtime tools (firebug, jasmine) look like they solve many problems - and they do. But they still leave poorly written code as just that. One key problem is that all javascript code lives globally and this results in poorly managed, tested and delivered code.
In this session, I will illustrate that we can treat javascript as a first-class citizen using with require.js and node.js: it can be modular, encapsulated and easily unit tested and added to continuous integration cycle. The dependencies between javascript modules can also be managed and packaged just like in C# and Java. In the end, we can resolve many javascript difficulties at compile time rather than waiting until runtime.
Main instruments used for Web UI test automation.
Drivers: Selenium WebDriver, Sikuli X.
Frameworks: PageFactory, TestNG.
Other tools: Apache Ant, Jenkins CI.
This is a presentation about mobile apps testing using Appium. The presentation conrtains tips on comprehensive testing of apps written for various devices and platforms.
Presentation by Yaroslav Pernerovskyy (Lead Test Engineer, GlobalLogic, Kyiv), delivered at UA Mobile, November 22, 2014.
More details - http://uamobile.org
How to generate customized java 8 code from your databaseSpeedment, Inc.
Did you know that database classes, that require many lines of Java and SQL code, may be replaced with a single line of Java 8 code? In this tutorial session you will learn how to use standard Java 8 Streams as an alternative to traditional Object Relational Mappers (ORM). We will use the open-source tool Speedment to show how development speed can be increased and how the application code can be more concise and run faster.
Silicon Valley JUG - How to generate customized java 8 code from your databaseSpeedment, Inc.
The best code is the one you never need to write. Using code generation and automated builds you can minimize the risk of human error when developing software, but how do you maintain control over code when large parts of it is handed over to a machine? In this tutorial, you will learn how to use open-source software to create and control code automation. You will see how you can generate a completely object-oriented domain model by automatically analyzing your database schemas. Every aspect of the process is transparent and configurable, giving you as a developer 100% control of the generated code. This will not only increase your productivity, but also help you build safer and more maintainable Java applications.
Top100summit 谷歌-scott-improve your automated web application testing
1. Improve Your Automated Web
Application Testing
Scott McMaster
Google
Kirkland, Washington USA
2. About Me
• Software Design Engineer @ Google.
– Building high-traffic web frontends and services in Java.
– AdWords, Google Code
• Ph.D. in Computer Science, U. of Maryland.
• Formerly of Amazon.com (2 years), Lockheed Martin (2 years), Microsoft
(7 years), and some small startups.
• Frequent adjunct professor @ Seattle University, teaching software
design, architecture, and OO programming.
• Author of technical blog at http://www.scottmcmaster365.com.
3. Testing and Me
• Doing automated testing since 1995.
• Ph.D. work in test coverage and test suite maintenance.
• Champion of numerous unit, system, and performance
testing tools and techniques.
• Co-founder of WebTestingExplorer open-source automated
web testing framework (www.webtestingexplorer.org).
4. Agenda
• Web Testing Challenges
• The Present: Selenium/WebDriver Automation.
– Best Practices
– Things to Avoid
• The Future? Automated Exploratory Testing with
WebTestingExplorer
5. A Few Words About Testing @
Google
• At Google, test engineers are developers. They develop:
– Test code.
– Test frameworks and productivity tools that are used by software design engineers.
• Test code quality == development code quality.
– Test code must be maintainable, robust, and meet other general software engineering
guidelines.
• For more on “How Google Tests Software”, see the series of blog posts by James
Whittaker (starting at http://googletesting.blogspot.com/2011/01/how-google-
tests-software.html).
6. Assumptions in This Talk
• You have at least some background in:
– Web application development and/or testing.
– Basic Selenium and/or WebDriver usage.
• And you would like to improve your testing
through the user interface!
8. Some Things on That Page
• Dynamic content and hundreds of HTML elements.
• User content (reviews, likes).
• Multiple dropdown menus.
• Search.
• Security (login).
• State (shopping cart).
• Embedded video.
9. It’s Even Worse
• After we write the tests, customers /
designers will want changes.
– Our tests will probably break.
– Managers don’t want to wait for testing to fix
their broken tests before releasing the software.
13. Use Cases
• Enter the name of a country and load cities.
– Error message if not found.
• Select a city and load the current weather.
• The weather appears in a table.
• Signed-in user gets weather for their “favorite”
city.
14. Architecture
Rich Browser UI (GWT)
Frontend Server (GWT RPC
servlet)
Remote Web
Service (XML-over- User Database
HTTP)
15. What Testing Should We Automate?
• Unit?
– Yes!
• System?
– Yes!
• End-to-end?
– Yes! This will be the focus of this talk.
16. Web UI System Test Automation
• What tools do we use?
– Selenium WebDriver
– http://seleniumhq.org/projects/webdriver/
17. Selenium WebDriver
“Driving a browser natively as a user would either locally or on a remote machine…”
• Selenium IDE
• Browser plugin for play/record.
• Mostly for non-programmers – don’t use this.
• Selenium RC / Selenium 1
• Deprecated – do not start using this!
• Selenium 2 (WebDriver)
• More concise API.
• Uses browser’s native automation support, works in all
major browsers (but Firefox tends to work “best”).
• Specially designed for rich (AJAX) web UIs.
• (For example, performs style calculations to help
you only work with displayed elements.)
18. Selenium WebDriver Language
Bindings
• Java
• Python
• C#
• Ruby
• PHP
• Perl
• Javascript
My examples will use Java.
19. “Simple” WebDriver Test: Spec
• Load the homepage.
• Find the “country” text box.
• Type in the name of a country that does not exist.
• Find the “load cities” button.
• Click it.
• Wait (important!)
• Verify that the “cities” UI is not displayed.
• Verify that the “country not found” error is displayed.
21. Bad News / Good News
• This code is verbose and hard to maintain.
• But we can make it better!
22. Tip#1: Organize Your Locators
• Shortens the code and gives you one easy-to-find
place to change when the UI changes.
private static interface WeatherHomeLocators {
By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID);
By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID);
By noCities = By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID);
By cityRow = By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID);
By city = By.id(GWT_DEBUG + DebugIds.CITY_ID);
By loadWeather = By.id(GWT_DEBUG + DebugIds.LOAD_WEATHER_ID);
By weatherCity = By.id(GWT_DEBUG + DebugIds.WEATHER_CITY);
By weatherTemp = By.id(GWT_DEBUG + DebugIds.WEATHER_TEMP);
By userName = By.id(GWT_DEBUG + DebugIds.USER_NAME);
By login = By.id(GWT_DEBUG + DebugIds.LOGIN);
By weather = By.id(GWT_DEBUG + DebugIds.WEATHER);
By unknownUser = By.id(GWT_DEBUG + DebugIds.UNKNOWN_USER);
}
…
WebElement countryElement = driver.findElement(WeatherHomeLocators.
countryName);
23. Bonus Tip #1a: How to Locate
• In priority order, locate elements by:
1. id (ask developers to add these, make them stable, and
available in all builds)
2. name
3. css class
4. link text (but watch for dynamic content and localization)
5. Xpath expression
24. Tip #2: Use a Base Class for Tests
• Some things you can do in a base class:
1. Create the browser profile and WebDriver.
2. Safely close the WebDriver.
3. Improve on the WebDriver API.
25. Base Class Example
public class AbstractWebDriverTest {
protected static final String GWT_DEBUG = "gwt-debug-";
protected WebDriver driver;
@Before
public void setUp() throws IOException {
FirefoxProfile firefoxProfile = new FirefoxProfile();
driver = new FirefoxDriver(firefoxProfile);
}
@After
public void tearDown() {
driver.close();
}
protected WebElement findElementOrFail(By by) {
WebElement element = driver.findElement(by);
if (element == null) {
org.junit.Assert.fail("Element " + by + " not found");
}
return element;
}
}
26. Other Good Base Class Methods
• Action methods:
protected void click(By by) {
WebElement element = findElementOrFail(by);
element.click();
}
API improvements:
protected void setText(By by, String text) {
WebElement element = findElementOrFail(by);
element.clear();
element.sendKeys(text);
}
Rich assertions:
protected void setText(By by, String text) {
WebElement element = findElementOrFail(by);
assertTrue(element.isDisplayed());
}
28. Now It Looks Like…
@Test
public void testUnknownCountryName() {
setText(WeatherHomeLocators.countryName, "Does not exist");
click(WeatherHomeLocators.loadCities);
WebDriverWait wait = createDefaultWait();
wait.until(ExpectedConditions.visibilityOfElementLocated(WeatherHomeLocators.noCities));
assertIsDisplayed(WeatherHomeLocators.noCities);
assertIsNotDisplayed(WeatherHomeLocators.cityRow);
}
Much Better!!!
29. Any More Problems?
• Today you select a country by:
– Typing a country name.
– Clicking the “Load Cities” button.
• What if we change it to:
– Select the country from a dropdown list.
• Our tests all break!
30. Tip #3: Use Page Objects
• Create higher-level action abstractions on a
per-page basis.
• Write tests that use those abstractions.
• Then, if the underlying UI changes, we can
update a single Page Object instead of many
individual tests.
31. Page Object Design Considerations
• Put your Locator definitions there.
• Give it access to your WebDriver API helpers.
• Put your page-specific action and (more
optionally) assertion methods there.
32. Page Object Example
public class WeatherHomePageObject {
public static interface WeatherHomeLocators {
public static final String GWT_DEBUG = "gwt-debug-";
By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID);
By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID);
// etc.
}
private WebDriverWrapper driverWrapper;
public WeatherHomePageObject(WebDriverWrapper driverWrapper) {
this.driverWrapper = driverWrapper;
}
public void loadCitiesForCountry(String countryName) {
driverWrapper.setText(WeatherHomeLocators. countryName, "Does not exist");
driverWrapper.click(WeatherHomeLocators. loadCities);
}
public void assertCitiesNotLoadedDisplay() {
driverWrapper.assertIsDisplayed(WeatherHomeLocators.noCities);
driverWrapper.assertIsNotDisplayed(WeatherHomeLocators.cityRow);
}
}
33. Now It Looks Like…
@Test
public void testUnknownCountryName() {
pageObject.loadCitiesForCountry("Does not exist");
WebDriverWait wait = createDefaultWait();
wait.until(ExpectedConditions.visibilityOfElementLocated(
WeatherHomeLocators.noCities));
pageObject.assertCitiesNotLoadedDisplay();
}
This change is subtle…but important!
It makes your tests maintainable and
robust to change in the UI!
We are pretty happy, but…
34. That Seems Like a Lot of Work!
• It is! Selenium WebDriver tests are:
– Time-consuming to develop.
– Somewhat susceptible to being “flaky” (example: use of “wait”s).
– Highly susceptible to breaking when the UI changes.
– Sometimes hard to maintain.
• Consequently, we often focus more on unit testing, less on
end-to-end tests.
• Usually this is pretty successful, but can we do better?
35. Big Idea
What if we could generate and regenerate interesting tests automatically,
and find bugs in the process?
36. WebTestingExplorer
• Supports Automated Exploratory Testing.
– Automated: Easy to replay (and regenerate) interesting
test cases.
– Exploratory: Systematically discovers new application
states.
– Testing: Includes test case generation, oracle generation,
real-time bug detection, and test case replay.
37. Advantages
• Efficiently test through the user interface.
• Test like a real user.
• Don't miss integration issues and things that
are hard to observe.
– Behind the scenes in the browser.
– Or on other application tiers.
38. WebTestingExplorer Design
• Assumes a testable application.
– Example: We can ask developers to add id attributes.
• Built on top of WebDriver.
– Easy to integrate with existing WebDriver code.
• Highly configurable and extensible.
• Easy to skip/ignore uninteresting and/or unstable regions of
the web page.
– Examples: Ads, “Like” buttons, common headers/footers, etc.
39. WebTestingExplorer Workflow
•Two Complementary Phases:
1. Exploratory
– Incrementally searches for new application states.
– Writes out new test cases and oracles.
– Can search for bugs.
– Tester iterates on the configuration to guide the tool into new areas of the application
(the feedback-driven, exploratory approach to testing).
2. Replay
– Executes tests generated by the Exploratory phase.
– For regression testing.
– A/B, comparison of actual and expected states.
40. State
• An abstraction of the current state of the web application, customizable by
the tester.
• Looking at “state” is key to how WebTestingExplorer generates test cases.
• State can be any combination of:
– Web browser state (HTML, DOM, cookies, ...)
– Web server state (session data, variables, ...)
– Database state (persistent application data)
– User-defined (but some state implementations are included and ready-to-
use!)
41. Exploratory Workflow
• Iteratively and systematically moves through the web
application via the user interface.
• Monitors for different types of bugs.
• Generates increasingly-long and sophisticated test cases
(and oracles) for later replay.
– Driven by feedback derived from state.
• Tester iterates on the configuration to guide the tool into interesting new
areas of the application and away from uninteresting ones (the feedback-
driven, exploratory approach to testing).
42. Automated Exploration
• Changes in observed state between actions move the exploration forward into
longer test cases.
• High-level algorithm:
– Pick a candidate test case (a series of actions – click, set-text, select, etc.).
– Take a state snapshot.
– Do something.
– Take another state snapshot.
– Compare the “before” and “after” states.
– Different? Then:
• Generate a new test case.
• Enqueue new, longer candidate test cases to execute.
– Repeat until out of time, or all test cases of a specified length are executed.
43. Automated Error Detection (Oracles)
• HttpResponseOracle: Checks for allowed and disallowed
HTTP response codes.
• JSErrorCollectorOracle: Looks for Javascript runtime errors in
Firefox.
• FinalStateCheckOracle: Uses expected and observed final
state for regression testing.
• Testers can plug in custom oracle implementations.
– Example: If your application displays an error bar when an error
occurs, you can write an oracle that looks for the error bar!
44. Regression Testing
• Exploration creates a test case.
– Test case includes saved final state for the configured state checkers.
• Replayer is configured with the FinalStateCheckOracle.
• Replayer replays the test case.
• At the end, a state snapshot is taken and compared to the
final state saved in the test case.
• If they are different, a failure is recorded.
45. Configuration
• Selectors for Actionable and Stateful web elements.
• Pre-packaged and custom Oracles.
• Action sequence filters to eliminate redundant or uninteresting
exploration paths.
– Example: MaxRepeatedActionSequenceFilter eliminates action sequences
that repeat the same action more than a set number of times.
• Action generators to create custom actions on specific elements.
• Action sequence prioritizer for test case prioritization during exploration.
• Custom wait conditions to increase stability while executing an action
sequence.
47. Custom Action Generator
• If you have links to external web sites (e.g. www.google.com) , you
probably don’t want to write tests that navigate there.
new TagActionGeneratorConfig("a") {
@Override
public Set<Action> generateActions(WebElementWithIdentifier elementWithId) {
Set<Action> actions = Sets.newHashSet();
String href = elementWithId.getElement().getAttribute("href");
if (href != null && href.startsWith("javascript:")) {
actions.add(new ClickAction(elementWithId.getIdentifier()));
}
return actions;
}
}
• This will disallow test cases for all <a>’s that do NOT execute Javascript.
• This is sufficiently useful that it is included as a library class:
• JavascriptAnchorActionGeneratorConfig
48. Another Example
• If you have a textbox that takes login names, you want to try both valid
and invalid logins.
new IdentifierActionGeneratorConfig(
new IdWebElementIdentifier(GWT_DEBUG + DebugIds.USER_NAME)) {
@Override
public Set<Action> generateActions(WebElementWithIdentifier elementWithId) {
Set<Action> actions = Sets.newHashSet();
actions.add(new SetTextAction(elementWithId.getIdentifier(), "scott@example.com"));
actions.add(new SetTextAction(elementWithId.getIdentifier(), "not a real user"));
return actions;
}
})
This will generate test cases with combinations of valid and invalid login names.
49. A True Story
• I built the WeatherApp just for this conference.
• The first time I ran WebTestingExplorer against it, I
found a bug!
– An empty country name tries to load all the cities in the
world that have weather data!
– Not what I intended, not a good idea.
50. Test Case (XML Format)
<?xml version="1.0" ?>
<TestCase>
<url>http://127.0.0.1:8888/WeatherAppV1.html</url>
<actionSequence>
<actions>
<SetTextAction>
<keysToSend>Not a country</keysToSend>
<identifier class= "IdWebElementIdentifier">
<id>gwt-debug-CountryName</id>
</identifier>
<initial>false</initial>
</SetTextAction>
<ClickAction>
<identifier class= "IdWebElementIdentifier">
<id>gwt-debug-LoadCities</id>
</identifier>
<initial>false</initial>
</ClickAction>
</actions>
</actionSequence>
<finalState>
<VisibleElementsState>
<elementProperties>
<!-- Lots more XML, properties and values of page elements -->
51. Test Case (Pretty Format)
1. Click the element with id gwt-debug-LoadCities
2. Click the element with id gwt-debug-LoadCities
3. Set the text in the element with id gwt-debug-CountryName
to 'China'
4. Set the text in the element with id gwt-debug-UserName to
'scott@example.com'
5. Refresh the page
PASSED
52. WebTestingExplorer is Open Source
Software
• http://www.webtestingexplorer.org
• http://code.google.com/p/webtestingexplorer/
• Contributors welcome!
53. Planned Features
• Parallel exploration.
• Code coverage integration.
• Different state abstractions.
– And advice and data on which ones work best.
54. Conclusions
• End-to-end testing of web applications is hard. But we can do
it!
• We can organize our Selenium WebDriver code to support
rapid development of concise, robust, maintainable tests.
– Organize Your Locators.
– Use a Base Class for Tests
– Use Page Objects
• And for automated exploratory testing:
– Try WebTestingExplorer.
55. Questions?
Scott McMaster
Google
Kirkland, Washington USA