The document outlines the evolution of test automation frameworks, emphasizing the progression from simple script-based testing to more complex object-oriented and keyword-driven approaches. It highlights the challenges faced by QA and development teams in collaboration due to disparate skill sets and tools, and describes the benefits of standardized frameworks that improve modularity, maintainability, and integration with continuous deployment practices. Key phases of test progression are detailed, along with best practices for creating reusable test libraries, effective test configurations, and automated processes to enhance software quality assurance.

1. Test Automation Framework
Designs
Designing standardized frameworks
instead of simple scripting
By Martin Lienhard

2. The Great Divide
• There has been a divide separating QA and Dev due in large
part by 3rd
party tool vendors and their proprietary test tools
• Open source tools have helped significantly to bridge that
divide, so that we can work together to create better
software
• Most QAs have the bigger picture of what they want to test,
and how to do it, but have trouble building robust automation
frameworks, because they lack the technical skills.
• Most Developers focus on testing code in isolation, but lack
the testing experience from a system-wide or end-to-end
perspective, so they have trouble building a test automation
framework that suits the needs of QA.

3. 1st
Phase of Test Progression
• Most QA start out creating test scripts by using record &
playback tools
• Lack conditional logic and looping
• Contain static data, UI locators, URLs, etc.
• Cannot be executed iteratively
• Cannot be reusable modules
• Should not be chained together as dependencies
• Lack configuration and global properties
• Cannot abstract away from the test tool or test runner

4. Record & Playback Test Scripts

5. 2nd
Phase of Test Progression
• QA usually improves their tests by adding data-driven
techniques
• Create variables and arrays to store data
• Create parameterized functions or methods to store data
• Add looping to recorded scripts

6. 3rd
Phase of Test Progression
• QA usually improves their tests by creating a business
function library
• Modular, reusable test libraries of functions
• Create file readers that pull in parameters from text files, CSV
files, etc.
• Abstract out the data from the tests by externalizing data
• Create parameterized or iterative tests that are data-driven
• Abstract out the UI locators from the tests by externalizing UI
maps
• Create global properties and configuration files
• A vast array of disparate data pools and UI maps are created,
which usually become unmanageable after a couple of years

7. 4th
Phase of Test Progression
• QA usually improves their tests by creating a keyword-driven
library:
• Create ODBC functions that read data from spreadsheets and
databases
• Create spreadsheets of keywords that represent business
domain language
– Low-level keywords describe test operations
– High-level keywords describe business language
– Intermediate-level keywords map business functions to
test operations
– Tests become readable by business users and non-
technical QA

8. …continued
• Create spreadsheets of data that drive the tests
• Query data from the application database as input and
verification points for tests
• A hierarchy of abstracted keyword spreadsheets are created
• Now non-technical BAs and QAs are creating automated tests
• A vast array of disparate keyword spreadsheets are created,
which usually become unmanageable after a couple of years
• Nearly all 3rd
party vendor test tools live in this area today
– Most of these tools use legacy scripting languages, which
are not compatible with today’s technology

9. 5th
Phase of Test Progression
• QAs become more technical and improve their tests by using
object oriented testing techniques:
• Objects are designed and created to further modularize and
reuse code with very low maintenance
• Test code is stored in objects
– The page object design is used
– Component object designs are used
• Test data are stored in objects
• Configuration and properties are stored in objects
• Calls are made to the application and developer written APIs
to make tests “smarter”

10. …continued
• Continuous integration environments are utilized to run tests
• QA moves into white box testing to drive quality deeper into
the application
– UI tests are ran “headless” to make them execute faster
• QA expands the test framework to include multiple test tools
and test runners
– Databases, web services, and APIs are created to reuse
code and data across multiple test tools
– Test tools and 3rd
party utilities can be upgraded to newer
versions with enhanced functionality with minor impact to
the test framework

11. …continued
• Developers can now run the tests, and contribute to the
overall test suites
• More 3rd
party vendors are rewriting their test tools to include
object oriented languages to accommodate these technical
QAs
– Due to the cost of the tools and their proprietary nature, it
is difficult to place these tools into a continuous
integration environment to share with developers

12. What is a test automation framework?
• A test automation framework is an application
that allows you to write a series of tests
without worrying about the constraints or
limitations of the underlying test tools

13. Benefits of building a Test Framework
• Tool agnostic
– Abstracts away low level commands
– Utilize many test tools
• Test tools such as Selenium, WebDriver, etc.
• Test runner such as unit test frameworks
• Other common utilities
– Perform multiple levels and types of testing
• Functional, regression, load, performance, unit,
integration, etc.

14. …continued
• Generic test API
– Common codebase and database
– Exception and error handling
– Modularized, reusable, and maintainable code and data
– Standardized test idioms
• Test configuration
– Configurable test suites
– Global test properties

15. …continued
• Automatic test versioning
– No need to get a specific version from the source control
repository
• Always get the latest version
– Eliminates deployment to multiple machines for testing
different versions of an application
• Deploy to a single server machine

16. …continued
• Multi-threading
– Run tests in parallel
– Runs on test machines in the grid or cloud
– Test across multiple environments and application
versions simultaneously
• Continuous integration
– Run in a continuous integration environment
– Share tests and collaborate with other teams

17. Test Framework Architecture

18. Test Suite
• A collection of tests to run
• Have test parameters
– i.e.: application, environment, version, etc.
• Have test groups
– i.e.: regression, build, etc.

19. Test Suite example

20. Test Configuration
• Any configurable test item that is separate from the test suite
– i.e.: web site URLs, database URLs, usernames, passwords, lab
machine configurations, etc.
• Separated from the rest of the framework for information
security and regulatory compliance

21. Test Configuration example

22. Test Properties
• Are thread specific
• Have global scope and can be accessed by any class method

23. Test Properties example

24. Test Data Repository
• Abstracts away changes made only to test data
• Data can be stored in databases and files such as XML, JSON,
XLS, TXT, CSV, etc.
• Data can be retrieved from data sources such as databases,
web services, APIs, EDI, etc.
• Metadata can be used to describe additional information
about the test data such as versions, application names,
required fields, invalid formats, etc.
• Share data across multiple screens, applications, and tools

25. Test Data example

26. Parameterized Test example
JUnit4 TestNG

27. Test Parameters
• Uses a hash map of key-value pairs
– Very useful for both functional and load testing
• Hash maps eliminate the need for
– Creating fixed method parameters that change frequently
– Changing data access objects that change frequently
• Hash maps can be easily populated from any data source
• Retrieve the test data and process the format for input to
various test tools
• Handles version differences between test data

28. Test Parameters example

29. User Interface (UI) Mapping
• Abstracts away changes made only to UI object locators
• UI object identifiers can be stored in databases and files such
as XML, JSON, XLS, TXT, CSV, etc.
• Metadata can be used to describe additional information
about the UI objects such as versions, application names,
form fields, etc.
• Share UI identifiers across multiple screens, applications, and
tools

30. UI Map example

31. Test Controls
• Retrieve the UI locators and process the format for input to
various test tools
• Handles version differences between UI locators
• Develop custom controls to mimic UI objects
• Self test verification

32. Test Control Interface example

33. Test Control Class example

34. Test Components
• A class object that represents a piece of a form, page, or
screen
• Groups of common objects that can be found across more
than one page
• Standardized component verification tests

35. Test Components example

36. Test Forms
• A class object that represents a piece a form
• Groups of components and controls that mimic the UI form
• Standardized form verification tests

37. Test Form Interface example

38. Test Form Class example

39. …continued

40. Test Page or Screen
• A class object that represents a web page or screen –
commonly referred to as “page objects”
• Groups of forms, components, and controls that mimic the UI
page or screen
• Self test verification

41. Test Page Class example

42. Instances when NOT to return a new
Page Object for each Page
• Submitting a page can return one of many different pages, or
versions of pages, which would be represented by many
different classes
– Page A could return B, C, or D
– Page A could return B v1.0, B v2.0, or Cv1.0, C v2.0, or D
v1.0, D v2.0, etc.
– This means that you would need to do the following:
• Create multiple methods with similar names that
return different page objects
– Page A.submitB() returns B
– Page A.submitC() returns C

43. …continued
• Return an page object of an inherited type, and then
cast the type of the page in the script
– Page A.submit() returns page X
– Pages B and C inherit from page X
– Script checks the type of X and casts to B or C
– This can get very ugly in a script…
• Submitting a page may initiate an AJAX call, which can rewrite
the same page
– Page A could display an AJAX (modal) dialog
– Page A could display some other AJAX widget
– Page A could be rewritten to look like page B

44. …continued
• If the page flow is not what is being tested, then an
unexpected page or AJAX call could fail and kill the test
– Test case:
• We are trying to test the functionality of page Z
• The primary page flow is as follows:
– A to B, B to C, C to D, and D to Z
• The alternative page flows could also correctly occur,
but are unpredictable base on many unknown
variables:
– A to H, H to L, L to M, and M to Z
– A to R, R to S, and S to Z
– A to X, X to Y, and Y to Z

45. …continued
– What if page A unexpectedly returned page H, R, or X
instead of page B?
– If page H, R, or X verified itself and asserted a failure, then
how would the test ever reach page Z?

46. JavaScript & HTML DOM
• There are many instances when we need the test framework
to mimic a human reading the content of a page and then
making a decision on what action to perform next
– Verifying the list of items on a search results page
– Selecting items to purchase from a list
– Selecting elements to edit, update, or delete
• Scenarios like these usually occur as a result of some previous
action, and display content generated dynamically to the user
• The test framework may have to query an application
database or web service to provide the input to trigger the
dynamic page content

47. …continued
• The test framework will then need to interrogate the page to
determine if the expected elements, attributes, or content
are present, and then make a decision on whether to perform
an action on some element
• JavaScript can be used to inspect the HTML DOM to find
these elements and their attributes and content
• JavaScript expressions can be evaluated by the test tool, and
then return some result that the test framework can use to
make a decision
• Checking if an element is present or getting a value will not
work on a dynamically generated element

48. JS DOM example
HTML JavaScript

49. Test Application Container
• A class object that represents the application, and contains
the major page and component objects by version
• It provides the test services available to the test scripts
• Is thread specific

50. Container example
Interface Class

51. Parallel Environment Tracks with
Different Build Versions
• Parallel development tracks competing to deploy to
production
• Run automated build verification tests
• Run automated functional and regression tests
• Run automated user acceptance tests
• Run automated tests for production patches and fixes
• Run automated sanity tests against production post-
deployment
• Run automated tests across versions for production rollback
strategy

52. Parallel Environment Tracks with
Different Build Versions

53. Test Scripts
• Test scripts can be written using standard unit test
frameworks
• Classes and methods are written using domain language
• Script details and complexity are abstracted to the framework
layers
• Scripts are shorter and easier to read

54. Test Script Class

55. Test Reports and Logs
• Reports can be generated from
– The unit test framework
– External reporting tools
– Custom report utilities can be written
• Test execution activity can be written to standard logging
utilities
• Test failures, errors, and exceptions can be written to the
reports and logs

56. Q&A
Thank you for your time!
Martin Lienhard
testautomaton1@gmail.com