Top100summit  谷歌-scott-improve your automated web application testing
Upcoming SlideShare
Loading in...5
×
 

Top100summit 谷歌-scott-improve your automated web application testing

on

  • 534 views

 

Statistics

Views

Total Views
534
Views on SlideShare
534
Embed Views
0

Actions

Likes
0
Downloads
5
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Top100summit  谷歌-scott-improve your automated web application testing Top100summit 谷歌-scott-improve your automated web application testing Presentation Transcript

  • Improve Your Automated Web Application Testing Scott McMaster Google Kirkland, Washington USA
  • About Me• Software Design Engineer @ Google. – Building high-traffic web frontends and services in Java. – AdWords, Google Code• Ph.D. in Computer Science, U. of Maryland.• Formerly of Amazon.com (2 years), Lockheed Martin (2 years), Microsoft (7 years), and some small startups.• Frequent adjunct professor @ Seattle University, teaching software design, architecture, and OO programming.• Author of technical blog at http://www.scottmcmaster365.com.
  • Testing and Me• Doing automated testing since 1995.• Ph.D. work in test coverage and test suite maintenance.• Champion of numerous unit, system, and performance testing tools and techniques.• Co-founder of WebTestingExplorer open-source automated web testing framework (www.webtestingexplorer.org).
  • Agenda• Web Testing Challenges• The Present: Selenium/WebDriver Automation. – Best Practices – Things to Avoid• The Future? Automated Exploratory Testing with WebTestingExplorer
  • A Few Words About Testing @ Google• At Google, test engineers are developers. They develop: – Test code. – Test frameworks and productivity tools that are used by software design engineers.• Test code quality == development code quality. – Test code must be maintainable, robust, and meet other general software engineering guidelines.• For more on “How Google Tests Software”, see the series of blog posts by James Whittaker (starting at http://googletesting.blogspot.com/2011/01/how-google- tests-software.html).
  • Assumptions in This Talk• You have at least some background in: – Web application development and/or testing. – Basic Selenium and/or WebDriver usage.• And you would like to improve your testing through the user interface!
  • How Would You Test This?
  • Some Things on That Page• Dynamic content and hundreds of HTML elements.• User content (reviews, likes).• Multiple dropdown menus.• Search.• Security (login).• State (shopping cart).• Embedded video.
  • It’s Even Worse• After we write the tests, customers / designers will want changes. – Our tests will probably break. – Managers don’t want to wait for testing to fix their broken tests before releasing the software.
  • !!!
  • Calm Down. What Can We Do?• Write more robust, more maintainable tests!
  • Let’s Use a Slightly Simpler Example
  • Use Cases• Enter the name of a country and load cities. – Error message if not found.• Select a city and load the current weather.• The weather appears in a table.• Signed-in user gets weather for their “favorite” city.
  • ArchitectureRich Browser UI (GWT) Frontend Server (GWT RPC servlet) Remote WebService (XML-over- User Database HTTP)
  • What Testing Should We Automate?• Unit? – Yes!• System? – Yes!• End-to-end? – Yes! This will be the focus of this talk.
  • Web UI System Test Automation• What tools do we use? – Selenium WebDriver – http://seleniumhq.org/projects/webdriver/
  • Selenium WebDriver “Driving a browser natively as a user would either locally or on a remote machine…”• Selenium IDE • Browser plugin for play/record. • Mostly for non-programmers – don’t use this.• Selenium RC / Selenium 1 • Deprecated – do not start using this!• Selenium 2 (WebDriver) • More concise API. • Uses browser’s native automation support, works in all major browsers (but Firefox tends to work “best”). • Specially designed for rich (AJAX) web UIs. • (For example, performs style calculations to help you only work with displayed elements.)
  • Selenium WebDriver Language Bindings• Java• Python• C#• Ruby• PHP• Perl• Javascript My examples will use Java.
  • “Simple” WebDriver Test: Spec• Load the homepage.• Find the “country” text box.• Type in the name of a country that does not exist.• Find the “load cities” button.• Click it.• Wait (important!)• Verify that the “cities” UI is not displayed.• Verify that the “country not found” error is displayed.
  • “Simple” WebDriver Test: Code@Testpublic void testUnknownCountryName() { FirefoxProfile firefoxProfile = new FirefoxProfile(); WebDriver driver = new FirefoxDriver(firefoxProfile); WebElement countryElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID)); countryElement.sendKeys("Does not exist"); WebElement loadCitiesButton = driver.findElement(By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID)); loadCitiesButton.click(); WebDriverWait wait = new WebDriverWait(driver, 5000); wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID))); WebElement noCitiesElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID)); assertTrue(noCitiesElement.isDisplayed()); WebElement cityRowElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID)); assertFalse(cityRowElement.isDisplayed());}
  • Bad News / Good News• This code is verbose and hard to maintain.• But we can make it better!
  • Tip#1: Organize Your Locators • Shortens the code and gives you one easy-to-find place to change when the UI changes. private static interface WeatherHomeLocators { By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID); By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID); By noCities = By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID); By cityRow = By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID); By city = By.id(GWT_DEBUG + DebugIds.CITY_ID); By loadWeather = By.id(GWT_DEBUG + DebugIds.LOAD_WEATHER_ID); By weatherCity = By.id(GWT_DEBUG + DebugIds.WEATHER_CITY); By weatherTemp = By.id(GWT_DEBUG + DebugIds.WEATHER_TEMP); By userName = By.id(GWT_DEBUG + DebugIds.USER_NAME); By login = By.id(GWT_DEBUG + DebugIds.LOGIN); By weather = By.id(GWT_DEBUG + DebugIds.WEATHER); By unknownUser = By.id(GWT_DEBUG + DebugIds.UNKNOWN_USER); } …WebElement countryElement = driver.findElement(WeatherHomeLocators. countryName);
  • Bonus Tip #1a: How to Locate• In priority order, locate elements by: 1. id (ask developers to add these, make them stable, and available in all builds) 2. name 3. css class 4. link text (but watch for dynamic content and localization) 5. Xpath expression
  • Tip #2: Use a Base Class for Tests• Some things you can do in a base class: 1. Create the browser profile and WebDriver. 2. Safely close the WebDriver. 3. Improve on the WebDriver API.
  • Base Class Examplepublic class AbstractWebDriverTest { protected static final String GWT_DEBUG = "gwt-debug-"; protected WebDriver driver; @Before public void setUp() throws IOException { FirefoxProfile firefoxProfile = new FirefoxProfile(); driver = new FirefoxDriver(firefoxProfile); } @After public void tearDown() { driver.close(); } protected WebElement findElementOrFail(By by) { WebElement element = driver.findElement(by); if (element == null) { org.junit.Assert.fail("Element " + by + " not found"); } return element; }}
  • Other Good Base Class Methods• Action methods: protected void click(By by) { WebElement element = findElementOrFail(by); element.click(); }API improvements: protected void setText(By by, String text) { WebElement element = findElementOrFail(by); element.clear(); element.sendKeys(text); }Rich assertions: protected void setText(By by, String text) { WebElement element = findElementOrFail(by); assertTrue(element.isDisplayed()); }
  • Remember Our “Simple” Test?@Testpublic void testUnknownCountryName() { FirefoxProfile firefoxProfile = new FirefoxProfile(); WebDriver driver = new FirefoxDriver(firefoxProfile); WebElement countryElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID)); countryElement.sendKeys("Does not exist"); WebElement loadCitiesButton = driver.findElement(By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID)); loadCitiesButton.click(); WebDriverWait wait = new WebDriverWait(driver, 5000); wait.until(ExpectedConditions.visibilityOfElementLocated(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID))); WebElement noCitiesElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.NO_CITIES_ID)); assertTrue(noCitiesElement.isDisplayed()); WebElement cityRowElement = driver.findElement(By.id(GWT_DEBUG + DebugIds.CITY_ROW_ID)); assertFalse(cityRowElement.isDisplayed());}
  • Now It Looks Like…@Testpublic void testUnknownCountryName() { setText(WeatherHomeLocators.countryName, "Does not exist"); click(WeatherHomeLocators.loadCities); WebDriverWait wait = createDefaultWait(); wait.until(ExpectedConditions.visibilityOfElementLocated(WeatherHomeLocators.noCities)); assertIsDisplayed(WeatherHomeLocators.noCities); assertIsNotDisplayed(WeatherHomeLocators.cityRow);} Much Better!!!
  • Any More Problems?• Today you select a country by: – Typing a country name. – Clicking the “Load Cities” button.• What if we change it to: – Select the country from a dropdown list.• Our tests all break!
  • Tip #3: Use Page Objects• Create higher-level action abstractions on a per-page basis.• Write tests that use those abstractions.• Then, if the underlying UI changes, we can update a single Page Object instead of many individual tests.
  • Page Object Design Considerations• Put your Locator definitions there.• Give it access to your WebDriver API helpers.• Put your page-specific action and (more optionally) assertion methods there.
  • Page Object Examplepublic class WeatherHomePageObject { public static interface WeatherHomeLocators { public static final String GWT_DEBUG = "gwt-debug-"; By countryName = By.id(GWT_DEBUG + DebugIds.COUNTRY_NAME_ID); By loadCities = By.id(GWT_DEBUG + DebugIds.LOAD_CITIES_ID); // etc. } private WebDriverWrapper driverWrapper; public WeatherHomePageObject(WebDriverWrapper driverWrapper) { this.driverWrapper = driverWrapper; } public void loadCitiesForCountry(String countryName) { driverWrapper.setText(WeatherHomeLocators. countryName, "Does not exist"); driverWrapper.click(WeatherHomeLocators. loadCities); } public void assertCitiesNotLoadedDisplay() { driverWrapper.assertIsDisplayed(WeatherHomeLocators.noCities); driverWrapper.assertIsNotDisplayed(WeatherHomeLocators.cityRow); }}
  • Now It Looks Like…@Testpublic void testUnknownCountryName() { pageObject.loadCitiesForCountry("Does not exist"); WebDriverWait wait = createDefaultWait(); wait.until(ExpectedConditions.visibilityOfElementLocated( WeatherHomeLocators.noCities)); pageObject.assertCitiesNotLoadedDisplay();} This change is subtle…but important! It makes your tests maintainable and robust to change in the UI! We are pretty happy, but…
  • That Seems Like a Lot of Work!• It is! Selenium WebDriver tests are: – Time-consuming to develop. – Somewhat susceptible to being “flaky” (example: use of “wait”s). – Highly susceptible to breaking when the UI changes. – Sometimes hard to maintain.• Consequently, we often focus more on unit testing, less on end-to-end tests.• Usually this is pretty successful, but can we do better?
  • Big IdeaWhat if we could generate and regenerate interesting tests automatically, and find bugs in the process?
  • WebTestingExplorer• Supports Automated Exploratory Testing. – Automated: Easy to replay (and regenerate) interesting test cases. – Exploratory: Systematically discovers new application states. – Testing: Includes test case generation, oracle generation, real-time bug detection, and test case replay.
  • Advantages• Efficiently test through the user interface.• Test like a real user.• Dont miss integration issues and things that are hard to observe. – Behind the scenes in the browser. – Or on other application tiers.
  • WebTestingExplorer Design• Assumes a testable application. – Example: We can ask developers to add id attributes.• Built on top of WebDriver. – Easy to integrate with existing WebDriver code.• Highly configurable and extensible.• Easy to skip/ignore uninteresting and/or unstable regions of the web page. – Examples: Ads, “Like” buttons, common headers/footers, etc.
  • WebTestingExplorer Workflow•Two Complementary Phases:1. Exploratory – Incrementally searches for new application states. – Writes out new test cases and oracles. – Can search for bugs. – Tester iterates on the configuration to guide the tool into new areas of the application (the feedback-driven, exploratory approach to testing).2. Replay – Executes tests generated by the Exploratory phase. – For regression testing. – A/B, comparison of actual and expected states.
  • State• An abstraction of the current state of the web application, customizable by the tester.• Looking at “state” is key to how WebTestingExplorer generates test cases.• State can be any combination of: – Web browser state (HTML, DOM, cookies, ...) – Web server state (session data, variables, ...) – Database state (persistent application data) – User-defined (but some state implementations are included and ready-to- use!)
  • Exploratory Workflow• Iteratively and systematically moves through the web application via the user interface.• Monitors for different types of bugs.• Generates increasingly-long and sophisticated test cases (and oracles) for later replay.– Driven by feedback derived from state.• Tester iterates on the configuration to guide the tool into interesting new areas of the application and away from uninteresting ones (the feedback- driven, exploratory approach to testing).
  • Automated Exploration• Changes in observed state between actions move the exploration forward into longer test cases.• High-level algorithm: – Pick a candidate test case (a series of actions – click, set-text, select, etc.). – Take a state snapshot. – Do something. – Take another state snapshot. – Compare the “before” and “after” states. – Different? Then: • Generate a new test case. • Enqueue new, longer candidate test cases to execute. – Repeat until out of time, or all test cases of a specified length are executed.
  • Automated Error Detection (Oracles)• HttpResponseOracle: Checks for allowed and disallowed HTTP response codes.• JSErrorCollectorOracle: Looks for Javascript runtime errors in Firefox.• FinalStateCheckOracle: Uses expected and observed final state for regression testing.• Testers can plug in custom oracle implementations. – Example: If your application displays an error bar when an error occurs, you can write an oracle that looks for the error bar!
  • Regression Testing• Exploration creates a test case. – Test case includes saved final state for the configured state checkers.• Replayer is configured with the FinalStateCheckOracle.• Replayer replays the test case.• At the end, a state snapshot is taken and compared to the final state saved in the test case.• If they are different, a failure is recorded.
  • Configuration• Selectors for Actionable and Stateful web elements.• Pre-packaged and custom Oracles.• Action sequence filters to eliminate redundant or uninteresting exploration paths. – Example: MaxRepeatedActionSequenceFilter eliminates action sequences that repeat the same action more than a set number of times.• Action generators to create custom actions on specific elements.• Action sequence prioritizer for test case prioritization during exploration.• Custom wait conditions to increase stability while executing an action sequence.
  • Let’s Look at Some Code!
  • Custom Action Generator• If you have links to external web sites (e.g. www.google.com) , you probably don’t want to write tests that navigate there. new TagActionGeneratorConfig("a") { @Override public Set<Action> generateActions(WebElementWithIdentifier elementWithId) { Set<Action> actions = Sets.newHashSet(); String href = elementWithId.getElement().getAttribute("href"); if (href != null && href.startsWith("javascript:")) { actions.add(new ClickAction(elementWithId.getIdentifier())); } return actions; } } • This will disallow test cases for all <a>’s that do NOT execute Javascript. • This is sufficiently useful that it is included as a library class: • JavascriptAnchorActionGeneratorConfig
  • Another Example• If you have a textbox that takes login names, you want to try both valid and invalid logins. new IdentifierActionGeneratorConfig( new IdWebElementIdentifier(GWT_DEBUG + DebugIds.USER_NAME)) { @Override public Set<Action> generateActions(WebElementWithIdentifier elementWithId) { Set<Action> actions = Sets.newHashSet(); actions.add(new SetTextAction(elementWithId.getIdentifier(), "scott@example.com")); actions.add(new SetTextAction(elementWithId.getIdentifier(), "not a real user")); return actions; } }) This will generate test cases with combinations of valid and invalid login names.
  • A True Story• I built the WeatherApp just for this conference.• The first time I ran WebTestingExplorer against it, I found a bug! – An empty country name tries to load all the cities in the world that have weather data! – Not what I intended, not a good idea.
  • Test Case (XML Format)<?xml version="1.0" ?><TestCase> <url>http://127.0.0.1:8888/WeatherAppV1.html</url> <actionSequence> <actions> <SetTextAction> <keysToSend>Not a country</keysToSend> <identifier class= "IdWebElementIdentifier"> <id>gwt-debug-CountryName</id> </identifier> <initial>false</initial> </SetTextAction> <ClickAction> <identifier class= "IdWebElementIdentifier"> <id>gwt-debug-LoadCities</id> </identifier> <initial>false</initial> </ClickAction> </actions> </actionSequence> <finalState> <VisibleElementsState> <elementProperties> <!-- Lots more XML, properties and values of page elements -->
  • Test Case (Pretty Format)1. Click the element with id gwt-debug-LoadCities2. Click the element with id gwt-debug-LoadCities3. Set the text in the element with id gwt-debug-CountryNameto China4. Set the text in the element with id gwt-debug-UserName toscott@example.com5. Refresh the pagePASSED
  • WebTestingExplorer is Open Source Software• http://www.webtestingexplorer.org• http://code.google.com/p/webtestingexplorer/• Contributors welcome!
  • Planned Features• Parallel exploration.• Code coverage integration.• Different state abstractions. – And advice and data on which ones work best.
  • Conclusions• End-to-end testing of web applications is hard. But we can do it!• We can organize our Selenium WebDriver code to support rapid development of concise, robust, maintainable tests. – Organize Your Locators. – Use a Base Class for Tests – Use Page Objects• And for automated exploratory testing: – Try WebTestingExplorer.
  • Questions? Scott McMaster GoogleKirkland, Washington USA
  • 2012-12-20