Automation Framework Design

White Paper on Automation Framework Design
Written by Kunal Saxena 1 | P a g e
Steps to succeed in Automation Framework Design
Automation Framework Design
White Paper
ABSTRACT: This is a white paper on creating a robust automation framework for your application. Good automation
framework will guide your project to success, where as bad framework will cost you money, time and resource wastage.
Whitepaper will highlight good Automation Framework guidelines, considerations and benefits.

INDEX
Table of Contents
Introduction ............................................................................................................................................................................3
Generations of Automation Frameworks ...............................................................................................................................4
What is AUTOMATION............................................................................................................................................................5
Automation Framework..........................................................................................................................................................7
Framework Design Guidelines ................................................................................................................................................9
Framework deign considerations..........................................................................................................................................10
Summary...............................................................................................................................................................................13
About the author ..................................................................................................................................................................14

Introduction
In today’s environment, software quality professionals are expected to do more with high quality and precision. They are
supposed to work equivalent to two resources put together. To increase their speed, accuracy, precision during testing
can be achieved only by automating features, which are complete. So that testers can save their time for new features
which needs manual effort.
Creating Automation Framework means you are mechanizing your redundant task. Now you don’t have to waste time in
doing same thing again, concentrate on new features and check result of older features generated by automation suite.
Creating automation suite, by using one method is different than implementing an automation framework for your test,
which make your testing efficient and you can include many task inside your framework e.g. record and play, scripting,
data driven test, logging, reporting and using capabilities of various tools available.
Good Framework Designing means you decide each task to be done modularly and without hindering other task. Coding
of cases is kept separate and handling logging and reporting is implemented separately. It includes development of
common library for the suite, with good logging and reporting to the stakeholders and removing duplicate functions to
remove redundancy. This will help you to add new feature and new test cases easily to the suite.
A framework should be built to support commercial and open source tools for the development. Different tools provide
different inbuilt features, and a framework must focus to use features of different tools and languages. A test
automation framework allows you to write a series of tests without worrying about the constraints or limitations of the
underlying test tools.
“The most critical aspect of building a good test automation framework is its design. This Whitepaper discusses
the best practices in designing a test automation framework.”

Generations of Automation Frameworks
First Generation:-
 This refers to Record/Replay (Capture/Playback) as publicized by different automation testing tools.
 This our 1st generation, may be applied at beginner learning level.
 Major flaws of this framework are non-reusable code, difficult to maintain and application flow dependent.
Second Generation:-
 Commonly known as Data Driven Framework, it is to run the same test cases with varying test data.
 This data can be stored in Data tables/ Excel / Access. The idea is to separate coding part from the data portion.
 It provides flexibility to tester to modify data easily rather than going to the code.
Third Generation:-
 Keyword Driven / Hybrid frameworks. As the name indicates Hybrid means mix of two or more frameworks.
 This is frequently implemented framework in the testing world.
 It can be modular in approach, separate SET of data and driven by keywords.
Fourth Generation:-
 Known with the name MBT (Model Based Testing).
 Key approach is focusing on building a model, with specified input test data, estimated output results,
 Run the tests and after running compare actual with expected result to decide the future course of action.
Fifth Generation:-
 This is the upcoming framework called as Script less/Codeless Framework.
 Approach is to use UI interface to record the objects and build the tests.
 The key to a 5th gen framework is building the pieces in memory at runtime. Like object building using
Descriptive Programming in UFT.

What is AUTOMATION
Automation is to execute your existing regression test cases without any human intervention. And your script should
keep on updating stakeholders after every milestone. For good automation, manager and resources have to plan, lot of
things in advance, so that your automation suite is robust, maintainable and error free. All points which are taken into
consideration before developing and automation suite is called automation Framework.
Following are basic points you should keep in mind while creating Automation Framework:
 Proper documentation for running Automation Suite
 Easy to learn and easy to execute
 Easy to add new Test Cases
 Easy to add new Features / Product.
 Easy to maintain Input data for different products.
 Good Logging for each test case.
 Reporting to Stakeholder at each milestone.
 Scaling of your automation suite.
 Easy to increase / decrease number of cases in regression.
 Easy to Increase / decrease number of system.
 Can plugin script to check performance stats and Accuracy Stats
 Automation suite setup in new machine
Lifecycle of Automation Framework:

Lifecycle of Automation, starts when, product is in design phase and ends when product is obsolete and moved to rack.
There are five different stages of automation and each phase has its own importance. If you miss one phase than it will
effect next phase and your automation framework as a whole.
First Stage is to understand the business and product. Identify tools to be used in framework. Test tool is decided by
going through design document and requirement of the product. Factors like cost, availability of the tool and ease of
integration with other tools will affect the decision. We can select tools internally used by the company, which can be
embedded in the framework, for various task like logging, reporting, and integration with messaging apps like hipchat,
slack, flowdock etc …
Second Stage is defining scope and coming up with the test cases you can and test cases which you cannot automate.
Here you have to define scope and different milestones to track to your progress. Application Under Test, itself have
different testing types to automate (namely UI, End-to-End, server messages, logging, memory leak, threshold, load,
stress, performance). Finalize scope before starting framework deign. Design should be flexible enough to incorporate
different scenarios.
Third Stage is Planning, Designing and Developing your framework. Which is most time consuming part of the whole
cycle. This phase goes with the framework development. You plan and design your framework and as features are
available you starts automating those features. Separate code for features, logging and reporting. If needed you can
easily added or remove features and test cases without changing framework and code. Config file is very important
while implementing framework, you take necessary values from config files when your framework is running.
Fourth Stage is execution and it goes side by side with third phase. Here you are developing new features and
executing old features. You have to analyze your test cases which are failed and understand the reason of failures, by
going through logs. Reason could be an issue in application or flaw in your automation suite. You can embed a log parser
for the test cases which are failed.
Fifth Stage is maintenance phase, when your product is stable and in the market and no more features are planned.
You maintain your framework and scripts and update results of automation results on daily basis.

Test Automation Framework Objective
A test automation framework can be defined as a set of processes, standards and interactions between the components
in which scripts are designed and executed. Different types of Automation framework can be developed.
It is very important to define the objectives of a Test Automation Framework before starting its design and
development. An organization may choose some or all of the objectives listed below, depending on its business needs.
The design of the test automation framework should be based on its objectives defined by the organization.
 Independent of Platform & OS  Support different testing types (UI,
database, WebService)
Test Automation
Framework are guidelines,
which when followed
gives high quality results.
Guidelines are like coding
standards, test-input data
handling, directory
structure, logging,
reporting etc.
This helps in reducing
code redundancy, higher
portability, reduced script
maintenance and better
quality of framework
 Hybrid Framework is used to
cater multiple needs (tables /
keywords)
 Ability to parameterize (recorded)
scripts
 Implement dynamic conditions
& loop support
 Test framework should be loosely /
tightly coupled with AUT depending
on business needs
 Auto update of test results in a
test management tool
 Less dependence on highly skilled
automation professionals
 Ability to schedule and run
prioritized test cases
 Auto raise defect in defect tracking
tool.
 Data driven capabilities (input
data, output data)
 Test reporting objectives including
email notifications.

 End to End testing capability to
cover business scenarios
 Log defects in a defect management
tool

Framework Methodology:
10 Steps to Design
a Good Automation
Framework
Key parameters to keep in mind for
framework design
 Handle scripts and data separately
 Create libraries
 Follow coding standards
 Offer high extensibility
 Less maintenance
 Script/Framework version control
 Logging
 Reporting to Stakeholders
1. Identification of the Scope: Company oriented, Product
oriented, Project Oriented.
2. Identification of the Needs: Identify Types of testing e.g.
Functional Test, Web Services etc. and application / modules to
be tested, GUI testing.
3. Identification of the Requirements: Find out the nature of
requirements, identify type of actions for each requirement &
identify high priority requirements.
4. Evaluation of the Test Automation Tool: Evaluation checklist,
Identify the candidate tools available in the market, Sample
run, rate & select the tools, Implementation & Training
5. Identification of the Actions to be automated: Actions,
Validations & requirements supported by the Tool, messages
between servers.
6. Design of the Test Automation Framework: Framework
guidelines, validations, Actions Involved, Systems involved, Tool
Extensibility Support, Customs messages & UML
Documentation.
7. Design of the Input Data Bank: Types of Input file. Input files –
Categorization & Design of file prototypes.
8. Development of the Automation Framework: Development of
script based upon framework design, Driver scripts, Worker
Scripts, Record / Playback, Screen / Window / Transaction,
Action / Keyword & Data Driven.
9. Population of Input Data Bank: Different Types of data Input,
Populate data from different data sources, Manual input of
data and Parent – Child data hierarchy.
10. Configuration of the Schedulers: Identify scheduler
requirements & configure the schedulers.

Framework Design Guidelines
As in SDLC to develop software applications, framework design plays a vital role in building a test automation
framework for an organization. Enterprise applications are large, complex, and ever-evolving, and to keep pace with
the organization’s growth, the design of the test automation framework should conform to the guidelines listed
below.
 Should be application-independent and easy to
enhance and maintain
 Identify and abstract common functions used across
multiple test scripts
 The framework should be based on open technologies to limit maintenance costs. Openness might also allow
cross-tool execution when a suitable API exits or can be developed
 The framework should allow easy integration with
other tools and components
 The framework should allow end-to-end testing of
the application suite
 Keep complex scenarios apart from normal scenarios. E.g. Functional test (separate different modules libraries),
Complex scenarios, Load test, Performance test, Stress test.
 Structure scripts with minimal dependencies –
ensuring execution of scripts unattended even on
failures
 Encapsulate testers from complexities of the test
framework
 Decouple test data from the test scripts

Framework Deign considerations
1. Type of framework
Test Framework helps to organize test suite and improve test efficiency. Different framework focuses on
different objectives.
Data Driven: Focuses on data management. If you need to execute your test cases with different data
than this framework is suitable.
Keyword Driven: its runs test cases based on keywords. If you have many different scenarios for each
features, you can implement this framework
Modular: If you want to separate coding of each feature than this framework is ideal. .It reduces
redundancy and dependency of code.
Hybrid: This is used when someone have to implement more than one framework. This is widely used
framework which can cater benefits of all framework.
2. Data Management
Data is an important factor in framework deign. There could be different types of data set needed by
you. User has to decide what to use for which data.
Data for setting up automation suite
Data for the list of test cases
Data for the test cases, should run in automation suite.
Data for input to a script and expected result
Data for validation.
For configuration values, you can use text file
For validation and input/output result, you can use excel or csv are access.
For test cases, you can use excel or test case management tool

3. Generic Modules
Test automation Framework have many reusable subroutines. If designed and used properly than it will
increase the pace of framework development with minimal errors. These subroutines are defined at
high level and based on the input to the subroutine it will do the necessary action.
e.g. written a function to send mail after every milestone. Same function you can call at every milestone
Log parsing: you wrote a function for parsing log based on search criteria and log path and output value.
check(“SUCCESS”,”Feature3_TC18”) OR check(“REQUEST”,”Feature3_TC18”,3)
4. Rules of Framework
This ensure consistency in your framework development. Defining set of rules needed for scripting and
execution automation framework. It includes, coding, reusable function usage, directory structure to be
followed, exception handling, creation of input data test, naming conventions for test cases and test
scripts. It helps future development easy and streamlined.
5. Integration of tools
Framework Design should allow easy integration of various scripts and tools you are planning to use.
Integration with Email script, test case management tool or defect tracking tool. If automation suite
finds any issue, than it should raise defect in defect tracking tool and update test cases management
tool with the defect ID. And all stake holders should be informed about the action taken by the
automation framework. Integration should not be dependent on the code.
6. Execution Configuration
There are important things should be considered before executing your suite.
1. Feature selection
2. Test case selection for each feature.
3. Log level and Reporting settings
4. Milestone in your regression run
5. Stakeholders email ID
7. Reporting Management
Reporting is important part of framework and considered as a face of a framework. Reporting itself has
many parameters to check.
To whom it should go
Customizable and user-friendly report.
Different level of reporting Summary and Detailed
Format needed by stakeholders (html, excel, pdf)
8. Logging Mechanism
Log generation is critical part of automation. It is vital while debugging issue and fixing the code. Logging
can be done at different level, depending on configuration parameter.

Benefits of Test Automation Framework:
A well designed framework increases the speed and accuracy of the testing process, enables a higher return on
investment (ROI), reduces maintenance and systematically minimizes risks. A well designed framework:
 Enhances efficiency in development of
automated test scripts through modularized,
reusable, and maintainable code and data
 Provides a structured development methodology
to ensure uniformity of design across multiple
test scripts to reduce dependency on individual
test-case developers
 Reduces dependence on subject matter experts
by automatically selecting the test to execute
according to test scenarios and dynamically
refining the test scope according to changes in
the test strategy or conditions of the system
under test
 Provides reliable issue detection and efficient
root-cause analysis with minimum human
intervention for the system under test
 Runs tests in parallel, runs on test machines in
the grid or cloud, tests across multiple
environments and application versions
simultaneously
 Performs multiple levels and types of testing –
functional, regression, load, performance, unit,
integration etc
 Eliminates deployment to multiple machines for
testing different versions of an application
 Is tool agnostic
 Runs in a continuous integration environment.

Summary
In today’s software development environment, test automation framework has become an integral component of
a software testing life cycle. By following the above test automation framework design guidelines, software testers
and software testing companies can immensely benefit by executing successful and maintainable test automation
projects.
In order to keep up with the pace of product development and delivery it is essential to implement an effective, reusable
test automation framework. We cannot expect the traditional capture/replay framework to fill this role for us. Past
experience has shown that capturereplay tools alone will never provide the long-term automation successes that other
more robust test automation strategies can.
A test strategy relying on data driven automation tool scripts is definitely the easiest and quickest to implement if you
have and keep the technical staff to handle it. But it is the hardest of the data driven approaches to maintain and
perpetuate and often leads to long-term failure.
A keyword driven automation framework is probably the hardest and potentially most time-consuming data driven
approach to implement initially. However, this investment is mostly a one-shot deal. Once in place, keyword driven
automation is arguably the easiest of the data driven frameworks to maintain and perpetuate providing the greatest
potential for long-term success. There are also a few commercially available framework which can fulfil few needs.
The most effective test strategies allow us to develop our structured test designs in a format and vocabulary suitable for
both manual and automated testing. This will enable testers to focus on effective test designs unencumbered by the
technical details of the framework used to execute them, while technical automation experts implement and maintain a
reusable automation framework independent of any application that will be tested by it.
What we really want is a framework that can be both keyword driven while also providing enhanced functionality for
data driven scripts. When we integrate these two approaches the results can be very impressive!
The essential guiding principles we should follow when developing our overall test strategy (or evaluating the test
strategy of a tool we wish to consider):
 The test design and the test framework are totally separate entities.
 The test framework should be application-independent.
 The test framework must be easy to expand, maintain, and perpetuate.
 The test strategy/design vocabulary should be framework independent.
 The test strategy/design should isolate testers from the complexities of the test framework.

About the author
Kunal Saxena, is a QA Expert in Hewlett Packard and has more than 14 years of experience in the field of QA. For the
past 11 years he is involved in Automation. Kunal have expertise in Design, Development and Implementing Automation
Framework from the scratch.
Implemented different types of framework in his carrier. (Record and Playback, Keyword driven, Data driven, Hybrid).
Also acquainted with functional tools like Winrunner, Testpartner, QTP. Kunal is very good in Perl, Python and shell
scripting also has good experience in the excel macros and google docs. Currently working in Cloud Domain in Hewlett
Packard. He is a person with Problem Solving attitude and resolves issue with his Logical Thinking. He has always kept
himself involved in writing scripts for the redundant task and increasing the efficiency of himself and his team.
Acknowledgements
I would like to thank …
My Brother Amit Saxena, who is continuous source of light, knowledge and power for me.
My Wife and Kids for whom I tries to strive best in the world.
My previous manager Karthik Krishnamurthy for having faith in me and giving opportunity to design framework from
scratch.
My current manager Gary Korhonen for giving me time and all necessary help to write this whitepaper.

Automation Framework Design

More Related Content

What's hot

Similar to Automation Framework Design

Automation Framework Design