Test Automation Framework Designs


Published on

See how Martin Lienhard of Williams-Sanoma builds a test automation framework using Selenium, data-driven tests, and much more!

Published in: Technology
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Test Automation Framework Designs

  1. Test Automation Framework Designs Designing standardized frameworks instead of simple scripting By Martin Lienhard
  2. The Great Divide <ul><li>There has been a divide separating QA and Dev due in large part by 3 rd party tool vendors and their proprietary test tools </li></ul><ul><li>Open source tools have helped significantly to bridge that divide, so that we can work together to create better software </li></ul><ul><li>Most QAs have the bigger picture of what they want to test, and how to do it, but have trouble building robust automation frameworks, because they lack the technical skills. </li></ul><ul><li>Most Developers focus on testing code in isolation, but lack the testing experience from a system-wide or end-to-end perspective, so they have trouble building a test automation framework that suits the needs of QA. </li></ul>
  3. 1 st Phase of Test Progression <ul><li>Most QA start out creating test scripts by using record & playback tools </li></ul><ul><li>Lack conditional logic and looping </li></ul><ul><li>Contain static data, UI locators, URLs, etc. </li></ul><ul><li>Cannot be executed iteratively </li></ul><ul><li>Cannot be reusable modules </li></ul><ul><li>Should not be chained together as dependencies </li></ul><ul><li>Lack configuration and global properties </li></ul><ul><li>Cannot abstract away from the test tool or test runner </li></ul>
  4. Record & Playback Test Scripts
  5. 2 nd Phase of Test Progression <ul><li>QA usually improves their tests by adding data-driven techniques </li></ul><ul><li>Create variables and arrays to store data </li></ul><ul><li>Create parameterized functions or methods to store data </li></ul><ul><li>Add looping to recorded scripts </li></ul>
  6. 3 rd Phase of Test Progression <ul><li>QA usually improves their tests by creating a business function library </li></ul><ul><li>Modular, reusable test libraries of functions </li></ul><ul><li>Create file readers that pull in parameters from text files, CSV files, etc. </li></ul><ul><li>Abstract out the data from the tests by externalizing data </li></ul><ul><li>Create parameterized or iterative tests that are data-driven </li></ul><ul><li>Abstract out the UI locators from the tests by externalizing UI maps </li></ul><ul><li>Create global properties and configuration files </li></ul><ul><li>A vast array of disparate data pools and UI maps are created, which usually become unmanageable after a couple of years </li></ul>
  7. 4 th Phase of Test Progression <ul><li>QA usually improves their tests by creating a keyword-driven library: </li></ul><ul><li>Create ODBC functions that read data from spreadsheets and databases </li></ul><ul><li>Create spreadsheets of keywords that represent business domain language </li></ul><ul><ul><li>Low-level keywords describe test operations </li></ul></ul><ul><ul><li>High-level keywords describe business language </li></ul></ul><ul><ul><li>Intermediate-level keywords map business functions to test operations </li></ul></ul><ul><ul><li>Tests become readable by business users and non-technical QA </li></ul></ul>
  8. … continued <ul><li>Create spreadsheets of data that drive the tests </li></ul><ul><li>Query data from the application database as input and verification points for tests </li></ul><ul><li>A hierarchy of abstracted keyword spreadsheets are created </li></ul><ul><li>Now non-technical BAs and QAs are creating automated tests </li></ul><ul><li>A vast array of disparate keyword spreadsheets are created, which usually become unmanageable after a couple of years </li></ul><ul><li>Nearly all 3 rd party vendor test tools live in this area today </li></ul><ul><ul><li>Most of these tools use legacy scripting languages, which are not compatible with today’s technology </li></ul></ul>
  9. 5 th Phase of Test Progression <ul><li>QAs become more technical and improve their tests by using object oriented testing techniques: </li></ul><ul><li>Objects are designed and created to further modularize and reuse code with very low maintenance </li></ul><ul><li>Test code is stored in objects </li></ul><ul><ul><li>The page object design is used </li></ul></ul><ul><ul><li>Component object designs are used </li></ul></ul><ul><li>Test data are stored in objects </li></ul><ul><li>Configuration and properties are stored in objects </li></ul><ul><li>Calls are made to the application and developer written APIs to make tests “smarter” </li></ul>
  10. … continued <ul><li>Continuous integration environments are utilized to run tests </li></ul><ul><li>QA moves into white box testing to drive quality deeper into the application </li></ul><ul><ul><li>UI tests are ran “headless” to make them execute faster </li></ul></ul><ul><li>QA expands the test framework to include multiple test tools and test runners </li></ul><ul><ul><li>Databases, web services, and APIs are created to reuse code and data across multiple test tools </li></ul></ul><ul><ul><li>Test tools and 3 rd party utilities can be upgraded to newer versions with enhanced functionality with minor impact to the test framework </li></ul></ul>
  11. … continued <ul><li>Developers can now run the tests, and contribute to the overall test suites </li></ul><ul><li>More 3 rd party vendors are rewriting their test tools to include object oriented languages to accommodate these technical QAs </li></ul><ul><ul><li>Due to the cost of the tools and their proprietary nature, it is difficult to place these tools into a continuous integration environment to share with developers </li></ul></ul>
  12. What is a test automation framework? <ul><li>A test automation framework is an application that allows you to write a series of tests without worrying about the constraints or limitations of the underlying test tools </li></ul>
  13. Benefits of building a Test Framework <ul><li>Tool agnostic </li></ul><ul><ul><li>Abstracts away low level commands </li></ul></ul><ul><ul><li>Utilize many test tools </li></ul></ul><ul><ul><ul><li>Test tools such as Selenium, WebDriver, etc. </li></ul></ul></ul><ul><ul><ul><li>Test runner such as unit test frameworks </li></ul></ul></ul><ul><ul><ul><li>Other common utilities </li></ul></ul></ul><ul><ul><li>Perform multiple levels and types of testing </li></ul></ul><ul><ul><ul><li>Functional, regression, load, performance, unit, integration, etc. </li></ul></ul></ul>
  14. … continued <ul><li>Generic test API </li></ul><ul><ul><li>Common codebase and database </li></ul></ul><ul><ul><li>Exception and error handling </li></ul></ul><ul><ul><li>Modularized, reusable, and maintainable code and data </li></ul></ul><ul><ul><li>Standardized test idioms </li></ul></ul><ul><li>Test configuration </li></ul><ul><ul><li>Configurable test suites </li></ul></ul><ul><ul><li>Global test properties </li></ul></ul>
  15. … continued <ul><li>Automatic test versioning </li></ul><ul><ul><li>No need to get a specific version from the source control repository </li></ul></ul><ul><ul><ul><li>Always get the latest version </li></ul></ul></ul><ul><ul><li>Eliminates deployment to multiple machines for testing different versions of an application </li></ul></ul><ul><ul><ul><li>Deploy to a single server machine </li></ul></ul></ul>
  16. … continued <ul><li>Multi-threading </li></ul><ul><ul><li>Run tests in parallel </li></ul></ul><ul><ul><li>Runs on test machines in the grid or cloud </li></ul></ul><ul><ul><li>Test across multiple environments and application versions simultaneously </li></ul></ul><ul><li>Continuous integration </li></ul><ul><ul><li>Run in a continuous integration environment </li></ul></ul><ul><ul><li>Share tests and collaborate with other teams </li></ul></ul>
  17. Test Framework Architecture
  18. Test Suite <ul><li>A collection of tests to run </li></ul><ul><li>Have test parameters </li></ul><ul><ul><li>i.e.: application, environment, version, etc. </li></ul></ul><ul><li>Have test groups </li></ul><ul><ul><li>i.e.: regression, build, etc. </li></ul></ul>
  19. Test Suite example
  20. Test Configuration <ul><li>Any configurable test item that is separate from the test suite </li></ul><ul><ul><li>i.e.: web site URLs, database URLs, usernames, passwords, lab machine configurations, etc. </li></ul></ul><ul><li>Separated from the rest of the framework for information security and regulatory compliance </li></ul>
  21. Test Configuration example
  22. Test Properties <ul><li>Are thread specific </li></ul><ul><ul><li>Have global scope and can be accessed by any class method </li></ul></ul>
  23. Test Properties example
  24. Test Data Repository <ul><li>Abstracts away changes made only to test data </li></ul><ul><li>Data can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. </li></ul><ul><li>Data can be retrieved from data sources such as databases, web services, APIs, EDI, etc. </li></ul><ul><li>Metadata can be used to describe additional information about the test data such as versions, application names, required fields, invalid formats, etc. </li></ul><ul><li>Share data across multiple screens, applications, and tools </li></ul>
  25. Test Data example
  26. Parameterized Test example <ul><li>JUnit4 </li></ul><ul><li>TestNG </li></ul>
  27. Test Parameters <ul><li>Uses a hash map of key-value pairs </li></ul><ul><ul><li>Very useful for both functional and load testing </li></ul></ul><ul><li>Hash maps eliminate the need for </li></ul><ul><ul><li>Creating fixed method parameters that change frequently </li></ul></ul><ul><ul><li>Changing data access objects that change frequently </li></ul></ul><ul><li>Hash maps can be easily populated from any data source </li></ul><ul><li>Retrieve the test data and process the format for input to various test tools </li></ul><ul><li>Handles version differences between test data </li></ul>
  28. Test Parameters example
  29. User Interface (UI) Mapping <ul><li>Abstracts away changes made only to UI object locators </li></ul><ul><li>UI object identifiers can be stored in databases and files such as XML, JSON, XLS, TXT, CSV, etc. </li></ul><ul><li>Metadata can be used to describe additional information about the UI objects such as versions, application names, form fields, etc. </li></ul><ul><li>Share UI identifiers across multiple screens, applications, and tools </li></ul>
  30. UI Map example
  31. Test Controls <ul><li>Retrieve the UI locators and process the format for input to various test tools </li></ul><ul><li>Handles version differences between UI locators </li></ul><ul><li>Develop custom controls to mimic UI objects </li></ul><ul><li>Self test verification </li></ul>
  32. Test Control Interface example
  33. Test Control Class example
  34. Test Components <ul><li>A class object that represents a piece of a form, page, or screen </li></ul><ul><li>Groups of common objects that can be found across more than one page </li></ul><ul><li>Standardized component verification tests </li></ul>
  35. Test Components example
  36. Test Forms <ul><li>A class object that represents a piece a form </li></ul><ul><li>Groups of components and controls that mimic the UI form </li></ul><ul><li>Standardized form verification tests </li></ul>
  37. Test Form Interface example
  38. Test Form Class example
  39. … continued
  40. Test Page or Screen <ul><li>A class object that represents a web page or screen – commonly referred to as “page objects” </li></ul><ul><li>Groups of forms, components, and controls that mimic the UI page or screen </li></ul><ul><li>Self test verification </li></ul>
  41. Test Page Class example
  42. Instances when NOT to return a new Page Object for each Page <ul><li>Submitting a page can return one of many different pages, or versions of pages, which would be represented by many different classes </li></ul><ul><ul><li>Page A could return B, C, or D </li></ul></ul><ul><ul><li>Page A could return B v1.0, B v2.0, or Cv1.0, C v2.0, or D v1.0, D v2.0, etc. </li></ul></ul><ul><ul><li>This means that you would need to do the following: </li></ul></ul><ul><ul><ul><li>Create multiple methods with similar names that return different page objects </li></ul></ul></ul><ul><ul><ul><ul><li>Page A.submitB() returns B </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Page A.submitC() returns C </li></ul></ul></ul></ul>
  43. … continued <ul><ul><ul><li>Return an page object of an inherited type, and then cast the type of the page in the script </li></ul></ul></ul><ul><ul><ul><ul><li>Page A.submit() returns page X </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Pages B and C inherit from page X </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Script checks the type of X and casts to B or C </li></ul></ul></ul></ul><ul><ul><li>This can get very ugly in a script… </li></ul></ul><ul><li>Submitting a page may initiate an AJAX call, which can rewrite the same page </li></ul><ul><ul><li>Page A could display an AJAX (modal) dialog </li></ul></ul><ul><ul><li>Page A could display some other AJAX widget </li></ul></ul><ul><ul><li>Page A could be rewritten to look like page B </li></ul></ul>
  44. … continued <ul><li>If the page flow is not what is being tested, then an unexpected page or AJAX call could fail and kill the test </li></ul><ul><ul><li>Test case: </li></ul></ul><ul><ul><ul><li>We are trying to test the functionality of page Z </li></ul></ul></ul><ul><ul><ul><li>The primary page flow is as follows: </li></ul></ul></ul><ul><ul><ul><ul><li>A to B, B to C, C to D, and D to Z </li></ul></ul></ul></ul><ul><ul><ul><li>The alternative page flows could also correctly occur, but are unpredictable base on many unknown variables: </li></ul></ul></ul><ul><ul><ul><ul><li>A to H, H to L, L to M, and M to Z </li></ul></ul></ul></ul><ul><ul><ul><ul><li>A to R, R to S, and S to Z </li></ul></ul></ul></ul><ul><ul><ul><ul><li>A to X, X to Y, and Y to Z </li></ul></ul></ul></ul>
  45. … continued <ul><ul><li>What if page A unexpectedly returned page H, R, or X instead of page B? </li></ul></ul><ul><ul><li>If page H, R, or X verified itself and asserted a failure, then how would the test ever reach page Z? </li></ul></ul>
  46. JavaScript & HTML DOM <ul><li>There are many instances when we need the test framework to mimic a human reading the content of a page and then making a decision on what action to perform next </li></ul><ul><ul><li>Verifying the list of items on a search results page </li></ul></ul><ul><ul><li>Selecting items to purchase from a list </li></ul></ul><ul><ul><li>Selecting elements to edit, update, or delete </li></ul></ul><ul><li>Scenarios like these usually occur as a result of some previous action, and display content generated dynamically to the user </li></ul><ul><li>The test framework may have to query an application database or web service to provide the input to trigger the dynamic page content </li></ul>
  47. … continued <ul><li>The test framework will then need to interrogate the page to determine if the expected elements, attributes, or content are present, and then make a decision on whether to perform an action on some element </li></ul><ul><li>JavaScript can be used to inspect the HTML DOM to find these elements and their attributes and content </li></ul><ul><li>JavaScript expressions can be evaluated by the test tool, and then return some result that the test framework can use to make a decision </li></ul><ul><li>Checking if an element is present or getting a value will not work on a dynamically generated element </li></ul>
  48. JS DOM example <ul><li>HTML </li></ul><ul><li>JavaScript </li></ul>
  49. Test Application Container <ul><li>A class object that represents the application, and contains the major page and component objects by version </li></ul><ul><li>It provides the test services available to the test scripts </li></ul><ul><li>Is thread specific </li></ul>
  50. Container example <ul><li>Interface </li></ul><ul><li>Class </li></ul>
  51. Parallel Environment Tracks with Different Build Versions <ul><li>Parallel development tracks competing to deploy to production </li></ul><ul><li>Run automated build verification tests </li></ul><ul><li>Run automated functional and regression tests </li></ul><ul><li>Run automated user acceptance tests </li></ul><ul><li>Run automated tests for production patches and fixes </li></ul><ul><li>Run automated sanity tests against production post-deployment </li></ul><ul><li>Run automated tests across versions for production rollback strategy </li></ul>
  52. Parallel Environment Tracks with Different Build Versions
  53. Test Scripts <ul><li>Test scripts can be written using standard unit test frameworks </li></ul><ul><li>Classes and methods are written using domain language </li></ul><ul><li>Script details and complexity are abstracted to the framework layers </li></ul><ul><li>Scripts are shorter and easier to read </li></ul>
  54. Test Script Class
  55. Test Reports and Logs <ul><li>Reports can be generated from </li></ul><ul><ul><li>The unit test framework </li></ul></ul><ul><ul><li>External reporting tools </li></ul></ul><ul><ul><li>Custom report utilities can be written </li></ul></ul><ul><li>Test execution activity can be written to standard logging utilities </li></ul><ul><li>Test failures, errors, and exceptions can be written to the reports and logs   </li></ul>
  56. Q&A Thank you for your time! Martin Lienhard [email_address]