Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Test Automation Framework Designs


Published on

See how Martin Lienhard of Williams-Sanoma builds a test automation framework using Selenium, data-driven tests, and much more!

Published in: Technology

Test Automation Framework Designs

  1. Test Automation Framework Designs Designing standardized frameworks instead of simple scripting By Martin Lienhard
  2. The Great Divide <ul><li>There has been a divide separating QA and Dev due in large part by 3 rd party tool vendors and their proprietary test tools </li></ul><ul><li>Open source tools have helped significantly to bridge that divide, so that we can work together to create better software </li></ul><ul><li>Most QAs have the bigger picture of what they want to test, and how to do it, but have trouble building robust automation frameworks, because they lack the technical skills. </li></ul><ul><li>Most Developers focus on testing code in isolation, but lack the testing experience from a system-wide or end-to-end perspective, so they have trouble building a test automation framework that suits the needs of QA. </li></ul>
  3. 1 st Phase of Test Progression <ul><li>Most QA start out creating test scripts by using record & playback tools </li></ul><ul><li>Lack conditional logic and looping </li></ul><ul><li>Contain static data, UI locators, URLs, etc. </li></ul><ul><li>Cannot be executed iteratively </li></ul><ul><li>Cannot be reusable modules </li></ul><ul><li>Should not be chained together as dependencies </li></ul><ul><li>Lack configuration and global properties </li></ul><ul><li>Cannot abstract away from the test tool or test runner </li></ul>
  4. Record & Playback Test Scripts
  5. 2 nd Phase of Test Progression <ul><li>QA usually improves their tests by adding data-driven techniques </li></ul><ul><li>Create variables and arrays to store data </li></ul><ul><li>Create parameterized functions or methods to store data </li></ul><ul><li>Add looping to recorded scripts </li></ul>
  6. 3 rd Phase of Test Progression <ul><li>QA usually improves their tests by creating a business function library </li></ul><ul><li>Modular, reusable test libraries of functions </li></ul><ul><li>Create file readers that pull in parameters from text files, CSV files, etc. </li></ul><ul><li>Abstract out the data from the tests by externalizing data </li></ul><ul><li>Create parameterized or iterative tests that are data-driven </li></ul><ul><li>Abstract out the UI locators from the tests by externalizing UI maps </li></ul><ul><li>Create global properties and configuration files </li></ul><ul><li>A vast array of disparate data pools and UI maps are created, which usually become unmanageable after a couple of years </li></ul>
  7. 4 th Phase of Test Progression <ul><li>QA usually improves their tests by creating a keyword-driven library: </li></ul><ul><li>Create ODBC functions that read data from spreadsheets and databases </li></ul><ul><li>Create spreadsheets of keywords that represent business domain language </li></ul><ul><ul><li>Low-level keywords describe test operations </li></ul></ul><ul><ul><li>High-level keywords describe business language </li></ul></ul><ul><ul><li>Intermediate-level keywords map business functions to test operations </li></ul></ul><ul><ul><li>Tests become readable by business users and non-technical QA </li></ul></ul>
  8. … continued <ul><li>Create spreadsheets of data that drive the tests </li></ul><ul><li>Query data from the application database as input and verification points for tests </li></ul><ul><li>A hierarchy of abstracted keyword spreadsheets are created </li></ul><ul><li>Now non-technical BAs and QAs are creating automated tests </li></ul><ul><li>A vast array of disparate keyword spreadsheets are created, which usually become unmanageable after a couple of years </li></ul><ul><li>Nearly all 3 rd party vendor test tools live in this area today </li></ul><ul><ul><li>Most of these tools use legacy scripting languages, which are not compatible with today’s technology </li></ul></ul>
  9. 5 th Phase of Test Progression <ul><li>QAs become more technical and improve their tests by using object oriented testing techniques: </li></ul><ul><li>Objects are designed and created to further modularize and reuse code with very low maintenance </li></ul><ul><li>Test code is stored in objects </li></ul><ul><ul><li>The page object design is used </li></ul></ul><ul><ul><li>Component object designs are used </li></ul></ul><ul><li>Test data are stored in objects </li></ul><ul><li>Configuration and properties are stored in objects </li></ul><ul><li>Calls are made to the application and developer written APIs to make tests “smarter” </li></ul>
  10. … continued <ul><li>Continuous integration environments are utilized to run tests </li></ul><ul><li>QA moves into white box testing to drive quality deeper into the application </li></ul><ul><ul><li>UI tests are ran “headless” to make them execute faster </li></ul></ul><ul><li>QA expands the test framework to include multiple test tools and test runners </li></ul><ul><ul><li>Databases, web services, and APIs are created to reuse code and data across multiple test tools </li></ul></ul><ul><ul><li>Test tools and 3 rd party utilities can be upgraded to newer versions with enhanced functionality with minor impact to the test framework </li></ul></ul>
  11. … continued <ul><li>Developers can now run the tests, and contribute to the overall test suites </li></ul><ul><li>More 3 rd party vendors are rewriting their test tools to include object oriented languages to accommodate these technical QAs </li></ul><ul><ul><li>Due to the cost of the tools and their proprietary nature, it is difficult to place these tools into a continuous integration environment to share with developers </li></ul></ul>
  12. What is a test automation framework? <ul><li>A test automation framework is an application that allows you to write a series of tests without worrying about the constraints or limitations of the underlying test tools </li></ul>
  13. Benefits of building a Test Framework <ul><li>Tool agnostic </li></ul><ul><ul><li>Abstracts away low level commands </li></ul></ul><ul><ul><li>Utilize many test tools </li></ul></ul><ul><ul><ul><li>Test tools such as Selenium, WebDriver, etc. </li></ul></ul></ul><ul><ul><ul><li>Test runner such as unit test frameworks </li></ul></ul></ul><ul><ul><ul><li>Other common utilities </li></ul></ul></ul><ul><ul><li>Perform multiple levels and types of testing </li></ul></ul><ul><ul><ul><li>Functional, regression, load, performance, unit, integration, etc. </li></ul></ul></ul>
  14. … continued <ul><li>Generic test API </li></ul><ul><ul><li>Common codebase and database </li></ul></ul><ul><ul><li>Exception and error handling </li></ul></ul><ul><ul><li>Modularized, reusable, and maintainable code and data </li></ul></ul><ul><ul><li>Standardized test idioms </li></ul></ul><ul><li>Test configuration </li></ul><ul><ul><li>Configurable test suites </li></ul></ul><ul><ul><li>Global test properties </li></ul></ul>
  15. … continued <ul><li>Automatic test versioning </li></ul><ul><ul><li>No need to get a specific version from the source control repository </li></ul></ul><ul><ul><ul><li>Always get the latest version </li></ul></ul></ul><ul><ul><li>Eliminates deployment to multiple machines for testing different versions of an application </li></ul></ul><ul><ul><ul><li>Deploy to a single server machine </li></ul></ul></ul>
  16. … continued <ul><li>Multi-threading </li></ul><ul><ul><li>Run tests in parallel </li></ul></ul><ul><ul><li>Runs on test machines in the grid or cloud </li></ul></ul><ul><ul><li>Test across multiple environments and application versions simultaneously </li></ul></ul><ul><li>Continuous integration </li></ul><ul><ul><li>Run in a continuous integration environment </li></ul></ul><ul><ul><li>Share tests and collaborate with other teams </li></ul></ul>
  17. Test Framework Architecture
  18. Test Suite <ul><li>A collection of tests to run </li></ul><ul><li>Have test parameters </li></ul><ul><ul><li>i.e.: application, environment, version, etc. </li></ul></ul><ul><li>Have test groups </li></ul><ul><ul><li>i.e.: regression, build, etc. </li></ul></ul>
  19. Test Suite example
  20. Test Configuration <ul><li>Any configurable test item that is separate from the test suite </li></ul><ul><ul><li>i.e.: web site URLs, database URLs, usernames, passwords, lab machine configurations, etc. </li></ul></ul><ul><li>Separated from the rest of the framework for information security and regulatory compliance </li></ul>
  21. Test Configuration example
  22. Test Properties <ul><li>Are thread specific </li></ul><ul><ul><li>Have global scope and can be accessed by any class method </li></ul></ul>
  23. Test Properties example
  24. Test Data Repository <ul><li>Abstracts away changes made only to test data </li></ul><ul><li>Data can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. </li></ul><ul><li>Data can be retrieved from data sources such as databases, web services, APIs, EDI, etc. </li></ul><ul><li>Metadata can be used to describe additional information about the test data such as versions, application names, required fields, invalid formats, etc. </li></ul><ul><li>Share data across multiple screens, applications, and tools </li></ul>
  25. Test Data example
  26. Parameterized Test example <ul><li>JUnit4 </li></ul><ul><li>TestNG </li></ul>
  27. Test Parameters <ul><li>Uses a hash map of key-value pairs </li></ul><ul><ul><li>Very useful for both functional and load testing </li></ul></ul><ul><li>Hash maps eliminate the need for </li></ul><ul><ul><li>Creating fixed method parameters that change frequently </li></ul></ul><ul><ul><li>Changing data access objects that change frequently </li></ul></ul><ul><li>Hash maps can be easily populated from any data source </li></ul><ul><li>Retrieve the test data and process the format for input to various test tools </li></ul><ul><li>Handles version differences between test data </li></ul>
  28. Test Parameters example
  29. User Interface (UI) Mapping <ul><li>Abstracts away changes made only to UI object locators </li></ul><ul><li>UI object identifiers can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. </li></ul><ul><li>Metadata can be used to describe additional information about the UI objects such as versions, application names, form fields, etc. </li></ul><ul><li>Share UI identifiers across multiple screens, applications, and tools </li></ul>
  30. UI Map example
  31. Test Controls <ul><li>Retrieve the UI locators and process the format for input to various test tools </li></ul><ul><li>Handles version differences between UI locators </li></ul><ul><li>Develop custom controls to mimic UI objects </li></ul><ul><li>Self test verification </li></ul>
  32. Test Control Interface example
  33. Test Control Class example
  34. Test Components <ul><li>A class object that represents a piece of a form, page, or screen </li></ul><ul><li>Groups of common objects that can be found across more than one page </li></ul><ul><li>Standardized component verification tests </li></ul>
  35. Test Components example
  36. Test Forms <ul><li>A class object that represents a piece a form </li></ul><ul><li>Groups of components and controls that mimic the UI form </li></ul><ul><li>Standardized form verification tests </li></ul>
  37. Test Form Interface example
  38. Test Form Class example
  39. … continued
  40. Test Page or Screen <ul><li>A class object that represents a web page or screen – commonly referred to as “page objects” </li></ul><ul><li>Groups of forms, components, and controls that mimic the UI page or screen </li></ul><ul><li>Self test verification </li></ul>
  41. Test Page Class example
  42. Instances when NOT to return a new Page Object for each Page <ul><li>Submitting a page can return one of many different pages, or versions of pages, which would be represented by many different classes </li></ul><ul><ul><li>Page A could return B, C, or D </li></ul></ul><ul><ul><li>Page A could return B v1.0, B v2.0, or Cv1.0, C v2.0, or D v1.0, D v2.0, etc. </li></ul></ul><ul><ul><li>This means that you would need to do the following: </li></ul></ul><ul><ul><ul><li>Create multiple methods with similar names that return different page objects </li></ul></ul></ul><ul><ul><ul><ul><li>Page A.submitB() returns B </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Page A.submitC() returns C </li></ul></ul></ul></ul>
  43. … continued <ul><ul><ul><li>Return an page object of an inherited type, and then cast the type of the page in the script </li></ul></ul></ul><ul><ul><ul><ul><li>Page A.submit() returns page X </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Pages B and C inherit from page X </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Script checks the type of X and casts to B or C </li></ul></ul></ul></ul><ul><ul><li>This can get very ugly in a script… </li></ul></ul><ul><li>Submitting a page may initiate an AJAX call, which can rewrite the same page </li></ul><ul><ul><li>Page A could display an AJAX (modal) dialog </li></ul></ul><ul><ul><li>Page A could display some other AJAX widget </li></ul></ul><ul><ul><li>Page A could be rewritten to look like page B </li></ul></ul>
  44. … continued <ul><li>If the page flow is not what is being tested, then an unexpected page or AJAX call could fail and kill the test </li></ul><ul><ul><li>Test case: </li></ul></ul><ul><ul><ul><li>We are trying to test the functionality of page Z </li></ul></ul></ul><ul><ul><ul><li>The primary page flow is as follows: </li></ul></ul></ul><ul><ul><ul><ul><li>A to B, B to C, C to D, and D to Z </li></ul></ul></ul></ul><ul><ul><ul><li>The alternative page flows could also correctly occur, but are unpredictable base on many unknown variables: </li></ul></ul></ul><ul><ul><ul><ul><li>A to H, H to L, L to M, and M to Z </li></ul></ul></ul></ul><ul><ul><ul><ul><li>A to R, R to S, and S to Z </li></ul></ul></ul></ul><ul><ul><ul><ul><li>A to X, X to Y, and Y to Z </li></ul></ul></ul></ul>
  45. … continued <ul><ul><li>What if page A unexpectedly returned page H, R, or X instead of page B? </li></ul></ul><ul><ul><li>If page H, R, or X verified itself and asserted a failure, then how would the test ever reach page Z? </li></ul></ul>
  46. JavaScript & HTML DOM <ul><li>There are many instances when we need the test framework to mimic a human reading the content of a page and then making a decision on what action to perform next </li></ul><ul><ul><li>Verifying the list of items on a search results page </li></ul></ul><ul><ul><li>Selecting items to purchase from a list </li></ul></ul><ul><ul><li>Selecting elements to edit, update, or delete </li></ul></ul><ul><li>Scenarios like these usually occur as a result of some previous action, and display content generated dynamically to the user </li></ul><ul><li>The test framework may have to query an application database or web service to provide the input to trigger the dynamic page content </li></ul>
  47. … continued <ul><li>The test framework will then need to interrogate the page to determine if the expected elements, attributes, or content are present, and then make a decision on whether to perform an action on some element </li></ul><ul><li>JavaScript can be used to inspect the HTML DOM to find these elements and their attributes and content </li></ul><ul><li>JavaScript expressions can be evaluated by the test tool, and then return some result that the test framework can use to make a decision </li></ul><ul><li>Checking if an element is present or getting a value will not work on a dynamically generated element </li></ul>
  48. JS DOM example <ul><li>HTML </li></ul><ul><li>JavaScript </li></ul>
  49. Test Application Container <ul><li>A class object that represents the application, and contains the major page and component objects by version </li></ul><ul><li>It provides the test services available to the test scripts </li></ul><ul><li>Is thread specific </li></ul>
  50. Container example <ul><li>Interface </li></ul><ul><li>Class </li></ul>
  51. Parallel Environment Tracks with Different Build Versions <ul><li>Parallel development tracks competing to deploy to production </li></ul><ul><li>Run automated build verification tests </li></ul><ul><li>Run automated functional and regression tests </li></ul><ul><li>Run automated user acceptance tests </li></ul><ul><li>Run automated tests for production patches and fixes </li></ul><ul><li>Run automated sanity tests against production post-deployment </li></ul><ul><li>Run automated tests across versions for production rollback strategy </li></ul>
  52. Parallel Environment Tracks with Different Build Versions
  53. Test Scripts <ul><li>Test scripts can be written using standard unit test frameworks </li></ul><ul><li>Classes and methods are written using domain language </li></ul><ul><li>Script details and complexity are abstracted to the framework layers </li></ul><ul><li>Scripts are shorter and easier to read </li></ul>
  54. Test Script Class
  55. Test Reports and Logs <ul><li>Reports can be generated from </li></ul><ul><ul><li>The unit test framework </li></ul></ul><ul><ul><li>External reporting tools </li></ul></ul><ul><ul><li>Custom report utilities can be written </li></ul></ul><ul><li>Test execution activity can be written to standard logging utilities </li></ul><ul><li>Test failures, errors, and exceptions can be written to the reports and logs   </li></ul>
  56. Q&A Thank you for your time! Martin Lienhard [email_address]