Introduction to Software Automation

1. Software Test Automation

2. What is Test Automation
• Developing software to test the software is called test automation.
• Test automation can help address several problems.
1. Automation saves time as software can execute test cases faster than
human do.
2. Test automation can free the test engineers from mundane tasks and
make them focus on more creative tasks.
3. Automated tests can be more reliable.
4. Automation helps in immediate testing
5. Automation can protect organization against attrition of test
engineers.
6. Test automation opens up opportunities for better utilization of global
resources.
7. Certain types of testing cannot be executed without automation
8. Automation means end-to-end, not test execution alone.
• Data generator: Automation should have scripts that produce test dat to
maximize coverage of permutations and combinations of inputs and
expected output for result comparison.

3. Terms Used In Automation
• A test case is a set of sequential steps to execute a test operating
on a set of predefined inputs to produce the certain expected
outputs.
• It always have an expected result associated when executed.
• It can be documented as a set of simple steps or it could be an
assertion statement or set of assertions.
• Example for assertion: “Opening a file which is already opened
should fail”.
• Test case are repeated for several times –build for several times.
• Example Log in process is performed in large number of test cases.
• In these case the automation code can be reused.
• Reusability reduces the effort and time.
• The table show two dimensions:
– What operation have to be tested – product -specific
– How the operations have to be tested – scenarios, framework -specific
• When a set of test cases is combined and associated with a set of
scenarios, they are called “test suite”.

4. Skills Needed For Automation
• The skills needed for automation depends on what generation of automation the
company is in or desires to be the in the near future.
• The automation of testing is broadly classified into three generations:
• First generation : Record and Play back:
– Avoid the repetitive nature of executing tests.
– A test engineer records the sequence of actions by keyboard characters or mouse clicks and
those recorded scripts are played back later.
– Drawbacks:
• Hard coded values so difficult to perform general types of tests.
• Requires a lot of manual intervention to detect and correct error conditions.
• When the application changes, all the scripts have to be re-recorded, thereby increasing the test
maintenance costs.
• Second generation: Data-driven:
– It helps in developing test scripts that generates the set of input conditions and corresponding
expected output.
– Test can be repeated for different input and output conditions.
• Third generation: Action-Driven:
– It enables a layman to create automated tests.
– All the actions that appear on the application are automatically tested, based on a generic set of
controls defined for automation.
– The set of actions are represented as objects and those objects are reused.
– Everything relevant to the action are automatically generated.
• Input , output, scenarios, framework
– It involves two major aspects “ test case automation” and “framework design”.

5. What to Automate, Scope of
Automation
• The specific requirement may vary from product to product, situation to situation and from time to
time.
• Tips to identify the scope of automation:
1. Identifying the Types of Testing amenable to Automation:
1. Stress, reliability and performance testing
2. Regression test
3. Functional tests
Tumb rule : effort for automation does not exceed the effort needed for executing those test case , then it can be
automated.
2. Automating areas less prone to change:
– Automate the requirements where there is less or no change.
– Normally change in requirements cause scenarios and new features to be impacted, not the basic
functionality of the product.
– User interface normally undergoes changes.
– While automating functions involving user –interfaces and non-user interfaces elements ,clear
demarcation and “pluggability” have to be provided.
3. Automate tests that pertain to standards:
– One of the test that the product may undergo is compliance to standards.
– Automating for standards provides a dual advantage:
• Test suites developed for standards can be
– Used for product testing
– sold as test tools for the market.
– Certification testing: the certification suites are executed every time by the supporting organization before
the release of hardware and software.

6. 4. Management Aspects in Automation:
– Prior to starting automation, adequate effort has
to be spent to obtain management commitment.
– Management permission are only given in
phases and part by part.
– Return on Investment.
– Automate the critical and basic functionalities of
the product first.

7. Design and Architecture For
Automation
1. External Modules:
– TCDB and Defect repository.
– The test cases, the steps to execute them and the history of their execution are stored in the
TCDB.
– Defect DB contains the details of all defects that are found in various products that are tested
in a particular organization.
– For automated test cases, the framework can automatically submit the defects to the Defect
DB during execution.
2. Scenario and Configuration File Modules:
– Scenarios- “how to execute a particular tests”
– Configuration files – set of variables used in automation.
– The variable could be for the test framework or for other modules in automation such as tools
and metrics or for the test suite or for a set of test cases or for a particular test case.
– The values of the variables in the configuration file can be changed dynamically to achieve
different execution, input, output and state conditions.
3. Test Cases and Test Framework Modules:
– Test case is an object for execution for other modules in the architecture and does not
represent any interaction by itself.
– Test framework – “what to execute” and “how they are executed”.
– The test framework is considered as the core for the automation design.
– The test framework can be developed by the organization internally or can be bought from the
vendor.

8. 4. Tools and Results Module:
– Build tools.
– When a test framework executes a set of test cases with a set of
scenarios for the different values provided by the configuration file,
the results for each of the test case along with scenarios and variable
values have to be stored for future analysis and action.
– The achieve of results help in executing test cases based on previous
test results.
5. Report Generator and Reports/Metrics Modules:
– Once the results of a test run are available, the next step is to prepare
the test reports and metrics.
– Preparing reports is a complex and time- consuming effort and hence
it should be part of the automation design.
– Customized report – very high level status, technical report –
moderate level of detail of the test run and detailed or debug reports-
which are generated for the developers to debug the failed test cases
and the product.
– The periodicity of the report is different such as daily, weekly, monthly
and milestone reports.
– The module that takes the necessary inputs and prepares a formatted
report is called a report generator.

9. Generic Requirements for Test
Tool/Framework
• Requirement 1: No hard coding in the test suite:
– One of the important requirements for the test suite to keep all
variables separately in a file.
– Variable are called as configuration variables and the file in
which all variable names and their associated values are kept is
called configuration file.
• Requirement 2: Test case/suite expandability:
– In a product scenario involving several releases and several test
cycles and defect fixes, the test cases go through large amount
of changes and additionally there are situations for the new test
cases to be added to the test suite.
• Adding a test cases should not affect other test cases.
• Adding a test case should not result in retesting the complete test
suite.
• Adding a new test suite to the framework should not affect existing
test suites.

10. • Requirement 3: Reuse of code for different types of testing, test cases:
– The test suite should only do what a test is expected to do. The test
framework needs to take care of “how” and
– The test programs need to be modular to encourage reuse of code.
• Requirement 4: Automatic set and cleanup:
– For each test case there could be some perquisite to be met before they
are run.
– Each test program should have a “setup” program that will create the
necessary setup before executing the test cases.
– A setup for one test case may negatively work for another test case. So
“undo” the setup soon after the test execution for the test case.-cleanup
program
• Requirement 5:Independent Test cases:
– Each test case should be executed alone.
– This requirement enables the test engineer to select and execute any
test case at random without worrying about other dependencies.
• Requirement 6: Test case dependencies:
– There may be a need for test cases to depend on others.
– A test tool or framework should help to specify the dynamic
dependencies between test cases.

11. • Requirement 7: Insulating test cases during execution:
– To avoid test cases failing due to some unforeseen events, the
framework should provide an option for users to block some of the
events.
• Requirement 8: Coding standards and Directory Structures:
– Coding standards and proper directory structures for a test suite help
the new engineers in understanding the test suite fast and help in
maintaining the test suites.
• Requirement 9: Selective execution of test cases:
– A framework may have multiple test suites; a test suite may have
multiple test programs and a test program may have multiple test
cases.
– The framework should enable the test engineer to select a particular
test case or set of test cases and execute them.
• Example :
test-program-name 2,4,1,7-10

12. • Requirement 10: Random execution of test cases:
– Test engineer may some times select a test case randomly from a list of test cases.
– Given a set of test cases and expecting a test tool to select the test case is called
random execution of test cases.
– Example:
random random
test-program-name 2,1,5 test-program1(2,1,5)
test-program2
test-program3
• Requirement 11: Parallel execution of test cases:
– Parallel execution simulates the behavior of several machines running the
same test and hence is very useful for performance and load testing.
– Example:
instances,5 instances,5
test-program1(3) test-program1(2,1,5)
test-program2
test-program3

13. • Requirement 12: Looping the test cases:
– There are two types of loops that are available.
• One is the iteration loop which gives the number of iterations of a particular test
case to be executed.
• Other is the timed loop, which keeps executing the test cases in a loop till the
specified time duration is reached.
• Example :
Repeat_loop,50 Time_loop,5 Hours
Test-program1(3) test-program1(2,1,5)
test-program2
test-program3
• Requirement 13: Gourping of test scenarios:
– The group scenarios allow the selected test cases to be executed in order,
random, in a loop all at the same time.
– Example :
group_scenario1
parallel,2 AND repeat,10 @scene1
Scene1
test –program1(2,1,5)
test-program2
test-program3

14. • Requirement 14: Test case execution based on previous
results:
– Rerun all the test cases which were executed previously.
– Resume the test cases from where they were stopped the previous
time.
– Rerun only failed/not run test cases.
– Execute all test cases that were executed on “date”.
• Requirement 15: Remote Execution of Test Cases:
– The central machine that allocates tests to multiple machines and co-
ordinates the execution and result is called test console or test monitor.
– It should be possible to execute/stop the test suite on any machine/set
of machines from the test console.
– The test results and logs can be collected from the test console:
– The progress of testing can be found from the test console.
• Requirement 16 : Automatic archival of test data:
– Archival of test data must include
• What configuration variable are used;
• What scenario was used; and
• What programs were executed and from what path.

15. • Requirement 17 : Reporting Scheme:
– Audit logs are very important to analyze the behavior of a test suite
and a product.
– A reporting scheme should include,
• When the framework , scenario, test suite, test program, and each test
case were started/completed;
• Result of each test case;
• Log messages;
• Category of events and log of events and
• Audit reports
• Requirement 18 : Independent of language:
– The framework should be independent of programming languages and scripts.
– A framework should provide choice of programming languages, scripts and
their combinations.
– A framework or test suite should not force a language/script.
– A framework or test suite should work with different test programs written
using different languages and scripts.
– A framework should have exported interfaces to support all popular, standard
languages and scripts.
– The internal scripts and options used by the framework should allow the
developers of a test suite to migrate to better framework.

16. Process Model for Automation
• Delivery of the automated test should be done before the execution phase so that
the deliverables from automation effort can be utilized for the current release of
the product.
• SDLC model followed for the product development is followed for the automation.
• “Test the test first” – an automated test suite has to be tested first before it can be
used to test the product.
• If test suites report wrong defects, it will severely impact the compatibility of test
automation and will hamper further automation efforts.
• A set of four activities can be performed at each phase of product and automation
development.
• The test suites form one of the deliverables from automation.
• W model includes two parallel set of activities for development and two parallel
set of activities for testing.
• The model is only to ensure the flow of activities, there is no binding on start and
end date because
– The product development and automation are two independent activities.
– In many organizations the same test team is involved in testing the product and in developing
the test suite.