A general slides for test automation principle, terminologies and implementation Also, the slides provide an example - PET, which is a platform written by Perl, but not just for Perl. It provides a general framework to use.

1. Test Automation
Principles, Terminologies & Implementations
Steven Li
Steven004@gmail.com

2. 2
Outline
•Test Automation Principles & Terminologies
 Why Test Automation
 How to do Test Automation
 Terminologies about Test Automation (Sycamore)
•Implement the Terminologies in Automation Scripts
 General Conventions
 An Example - PET

3. Why Test Automation

4. 4
Top 10 Reasons for Automating Tests - What people said
1. Manual testing sucks.
2. Tool vendor said
“Capture replay works.”
3. Fun to watch the
dialogs popping on the
screen.
4. “Afterwards we can fire
all those pesky
testers.”
5. Everybody else is doing
it.
6. “No humans were
harmed in the testing of
this software.”
7. Big bucks already spent
on the test tool.
8. Looks good on the
resume.
9. No Testing for Dummies
book ... yet.
10.Keep the intern busy.

5. 5
Reasons for Automating – What people want
• Speed up testing
• Allow more frequent testing
• Reduce manual labor costs
• Improve test coverage
• Ensure consistency
• Simplify testing
• Define the testing process
• Make testing more interesting and challenging
• Develop programming skills
• Justify cost of the tools
• Of course we’ll have automation!

6. How to do Test Automation

7. 7
Keep it Simple, Sustainable
•Uniform/Common infrastructure
•Appropriate tools
•Manage source code, test data, tools
•Agreed Conventions within the team
•High-quality Scripts
•Gradual Test Automation

8. 8
Quality of Automation Scripts
•Maintainability
 Will the tests still run after product design changes?
 Will tests for 1.0 work with 2.0? And can be easily updated?
•Reviewability
 Can others review and understand what is being covered?
 Are the scripts documented?, can we make sure it matches the original design?
 How about the coverage? How easy another engineer use the scripts for the next
release?
•Reusability
 To what degree the assets can be reused to create more, different tests?
 Can they be reused in different releases, in different environments?
•Dependability
 Integrity
 Reliability
•Repeatability
•Independence
•Performance
•Simplicity

9. Terminologies for
Test Automation

10. 10
Architecture for Automation
Test Planning Test Preparation Test PerfectingTest Performing
Test Suite
Logical TestBed
Test Case
Test Case
…
Test Case
Test Suite
…
Physical TestBed
Test Case
Test Case
…
Test Case
Test Case
Logical TestBed
Test Case
Test Case
…
Test Case
Test Case
Test Plan
…
Test Step 1
Test Step 2
…
Test Step n
Test Task
…
…
Test Report
Specify TestBed & TestCase List
For a release
Set a task to run a plan
Schedule
Runtime Options+
Schedule
Runtime Options+
Detailed Runtime Information
Detailed TestBed Information
Test Case Result
Test Case Result
…
Test Case Result
Detailed Runtime Information
Detailed TestBed Information
Test Case Result
Test Case Result
…
Test Case Result
Revised
Test Suites
Test Plans
Logical TestBed
Test Case
Test Case
…
Test Case
Physical TestBed
Test Case
Test Case
…
Test Case
Execute a task

11. 11
Test Bed
•Briefly, a test bed is the definition of the environment for a
specified test. A test bed include the hardware/software
and configuration/provisioning.
•There are two kinds of test beds:
 Logical TestBed. A logical test bed depicts all the elements and
configuration required for the test, but not indicate which exactly
elements will be used (i.e. the index or addresses are not required.)
 Physical TestBed. A physical test bed is an instantiation of a logical
test bed. A physical test bed exactly indicates which elements are
used in the test, and a test task can be really run on the test bed.
• Base State of a test bed
 The initial state of the test bed, ready for test case (group) start to
run with it.
 A self-cleaning test case (group) should get the test bed back to the
base state when the execution completed.

12. 12
Test Case (group)
•A test case is a sequence of operations against a system
whose results can be used to determine if a requirement
or use case upon an application of the system is partially
or fully satisfied.
 A test case contains a set of test steps, and based on a particular
test bed.
 A test step is an atomic statement of a test case, can have pass/fail
result. Usually it contains one operation and the expected result.
 A set of test cases can be grouped in a test case file.
•Test Case File (test group):
 A test case file defines one or multiple test cases, which is based on
a clean test bed for running, and do self-cleaning after performing
all the cases one by one.
 A test group are a set of test cases saved in one test case file.

13. 13
Test Suite
•A suite of test cases, which share the same test bed to
verify a specified feature in a specified test area. Usually,
there are multiple test cases in one test suite.
•In automation scripts, a test suite includes:
 A test suite file: to describe the basic information of this test suite
 A logical test bed: to define the test bed which the test suite is
based on
 One or more test case files: each test case file is self-cleaning, so
that each file can run independently.
•Usually, test suite is feature oriented. It can be created
once the feature is introduced, and keep being improved
along the product life, or even product family life.

14. 14
Test Plan
•A test plan is an instantiation of a subset of a test suite, in
which a set of test cases selected for a particular test for a
feature in a program; and indicate the physical test bed
for test performing.
 E.g. we have a BLSR_4F_2Node test suite, then we can make a plan
for SN16K R9.0 BLSR 4F 2Node Regression test, targeted run it in
two days.
•In automation scripts, a test plan includes:
 Test suite path/name
 The physical test bed, which is an instantiation of the logical test bed
 A test case file list, selected from the test suite
•Usually, a test plan is feature and release oriented. For
different releases, one can have different test plan for one
feature. And, a test plan should be defined in the test
preparation stage.

15. 15
Test Task
•Test Task is a runtime concept, it defines when and how to
execute a test plan. Once a test task is executed, a test
report should be reproduced
•In automation scripts, a test task could be a command in
CLI, an item in TMS, or a file, which includes:
 The test plan path/name
 A schedule for the test plan running
 Runtime options, e.g. Halt or skip if any failure? Wait or fail if the
test bed is not available? Recycle or one-time running?
•Multiple test tasks can be created for a test plan, that
means, a test plan can be executed multiple times in a
release.
•Usually, a test task is created in test performing stage by
the tester for one-time running, and will not be reused.

16. 16
Test Report
•A test report is a result record of a test task.
•Test reports can also be organized by test plans or test
suites for broadened view to analyze requirements
satisfaction.
•A test report should have the following information
 Runtime information: e.g. who/where/how had run this task with
what options?
 Detailed test bed information: e.g. build no. for each node, serial
numbers of elements, software versions
 Test results for each case and each step.
•A test report can be valuable just with a specified test bed
on which the test was running.

17. 17
Outline
•Test Automation Principles & Terminologies
 Why Test Automation
 How to do Test Automation
 Terminologies about Test Automation (Sycamore)
•Implement the Terminologies in Automation Scripts
 General Conventions
 An Example - PET

18. 18
Classic Reasons for Automation Failure
10. No clear objectives for automation.
9. Minimal or No attention of non GUI automation.
8. Lack of methodology to success of the project.
7. Automation is easy Just record and playback. –
6. Aim for 100% automation.
5. Testing for automation scripts.
4. No clarity of real cost involved in automation.
3. Automation is part time job of testers.
2. Wrong selection of Test cases.
1. Not treating automation as software development.

19. General Conventions

20. 20
A flexible framework
•General framework allows users define format and
language
•User defined languages for:
 Test bed
 Test cases
 Test report
 Test task
 Scheduler ….
•General format
 XML
 Perl, Python
 Simply BDD language,
 Spreadsheet

21. 21
Plug-in technology
•A platform should support a general framework
•All support simple plug in
•

22. 22
An example: PET
•Pet: Peal Engine for Test
 It is not just for Perl testing, but
 Implement a framework to support different test format
 It is independent to run as an application
 Or, to be used as a library for your bigger framework
•Key engine: Test Recorder
•Download from: https://github.com/steven004/Pet

23. An example: PET

24. 24
What is Pet?
•Pet – Perl Engine for Test
 Perl: Pet is based on Perl. It can run on almost any platforms
 Engine: It will speed up your work, and provide lots of benefit
 Test: It is designed for test, not just for test execution, but test
management as well
IQstream
WebUI
SN9K/3K/16K
…
Support different products
Console
HTML
TMS
E-Mail
…
TEA XML Perl …
Support different format/language
Support different outputs
Test Plans/Suites
Pet Engine
Library
Key-Words
Objects
…
Test Cases
Test Beds
TestRecorder

25. 25
Why you need Pet?
•TestRecorder Module embedded in Pet
 You will get test results via different channels
 Console output, HTML files for Web, E-mail notification, TMS database, and more if you request
 All the channels can be configured, you can select any of them as you like/need when running your scripts.
 A bunch of test recorder functions provided
•Multi-format supported for scripts writing
 Pure-perl
 TEA format compliant
 Or mix-programming
 You can use Perl in your TEA case file; or use TEA format in Perl scripts
 Spreadsheet cases
• Scripts running control
 A mechanism provided to run scripts case by case, or step by step
 Runtime control provided, by “Ctrl-c”
 Failure handling, for collect information in your scripts.
 Error handling, If there is some syntax problem in your scripts, Pet
will let you know.

26. 26
How to run Pet – the 1st
step
• Pet looks complicated, but we make it simple
• Let’s start
a. Log into an automation Server using your unix account
• 172.21.7.200 (Shanghai) or 172.31.115.6 (Chicago)
• Run Pet
-bash-3.2$ pet
/usr/bin/pet requires arguments. Try ‘pet –h’ for help.
-bash-3.2$ pet -h
Usage: Pet [-cnhpv] [testplanfile] [switches] [arguments]
-u UserName indicate the who is running the test scripts. Pet will get the user name from OS if no this option
-h print Usage hints
-l OutputDir indicate the output directory of log. tmppetusername will be used for the default output directory
-m the mail address for test result reports
-v print version, subversion <includes VERY IMPORTANT Pet functionality info>
-d [:debugger] run program under debugger (future)
-b BuildNumber indicate the build number of the system/software under test
-n new a test plan file in current directory based on the existing test bed and test case files
-p parse the test plan and print the case list to the standard output
-r ReleaseName indicate the release name/number of the system/software under test
-o [tms/console/html/xml/txt] indicate the environment of the Pet. Local run by default
-c Pet will just check test bed availability, instead running all cases in the test case list
-t TaskID indicate the test task ID for TMS. This option is just required by TMS to run the Pet
-f
[KEEP_RUNNING/STOP_ON_STEP_FAIL/STOP_ON_CASE_FAIL/PROMPT_ON_FAIL/PROMPT_BY_STEP/PROMPT_BY_CASE/T
EST_QUIT] continue the program or stop it when there is a case fail

27. 27
Pet running examples
•Run a Perl file
 $ pet example.pl –r 1.0
•Run a Perl-format test plan
 $ pet example_pl.xml –r 1.2 –b B2.8
•Run a Tea-format test plan
 $ pet example_tea.xml –r 1.2 –b B2.7 –m steven004@gmail.com
•More parameters
 $ pet example_tea.xml –r 1.2 –b B2.7 –u xli –m
steven004@gmail.com –t 53432 –o CONSOLE – l /tmp/xli
•Parse TestPlan
 $pet –p exampletp.xml

28. 28
Outline
•Pet Basic
•Quick Start
 Understand test plan, test suite, test bed and test suite
 Create your own test task
 Testbed modification
 Testcase files selection
 Where you get the results?
 Email, HTML, TMS?
•I want to do some change
•Advanced: How can I use Pet to write scripts
•Dig into Pet

29. 29
Terms – Test Plan/Bed/Suite/Task/Report
•See details in
 Previous slides
•Pet is to run a test plan in which
 Test Bed, Test Suite, Test Case list are specified
•A test plan example
<?xml version="1.0" encoding="UTF-8" ?>
<!-- Test Engineer: Guo, Hugh -->
<!-- Program: Pet Example -->
<!-- Description: Tea-type example test plan -->
<TestPlan
SuiteFile="../exampleSuite.pl"
PhysicalTestBedFile="../PhysicalTestBeds/example_tb.ttb">
<TestCases>
<File>../TestCases/example_tc.ttc</File>
</TestCases>
</TestPlan>
！ Use Relative paths in your test plan, instead of absolute ones ！

30. 30
Create your own test task
•Create/Modify TestBed file
•Create/Modify test plan file
 Indicate the test suite file
 Indicate the test bed file
 TestCase files selection
 Again, Use the relative paths, instead of absolute ones
•Demo …

31. 31
Where is my test results?
•Important Environment Variables
 Get it by running ‘set’ in Linux
-bash-3.2$ set
…
TESTMAILTO=steven004@gmail.com #results will be sent to this mail.
TESTRECOPT='TESTSUMMARY|TESTDETAIL|TESTLOG|ERRORLOG|WARNINGLOG|INFOLOG|DEBUGLOG‘
# All logs will be recorded
TESTRECPATH=/shared1/users/IQstream_Result/auto_pet/output/
#The place to save output results
TESTRECTO='CONSOLE|HTML|EMAIL‘ #You can get results through Console, Email and HTML files.
…
•Demo…

32. 32
Outline
•Pet Basic
•Quick Start
•I want to do some change
 Use default output options
 Set Pet Environment Variables
 TESTRECOPT, TESTRECPATH, TESTRECTO, TESTMAILTO
 Set Pet Options in your scripts
•Advanced: How can I use Pet to write scripts
•Dig into Pet

33. 33
All about TestRecorder
•TestRecorder Manual
 https://github.com/steven004/Pet/blob/master/Documents/TestReco
rder.htm
•TestRecorder Training Slides
 https://github.com/steven004/Pet/blob/master/Documents/Use
%20PET%20step%20by%20step.ppt
•Default TestRecorder Options had been set in automation
servers for all users
 For TestRECTO, TESTRECPATH, TESTRECOPT
•You need to set TESTMAILTO
•You can also set your own Recorder Options in your profile

34. 34
Use different options for a particular test
•Two ways to go
 1) Specify your options as arguments when you run your test
 pet myplan.xml –m notme@abc.com –r 1.2 –b 1 –l /tmp/xli/
 This time, notme@abc.com will get the result, and all results
files will be save in /tmp/xli in the automation server
 pet myplan.xml –u goldman –r 1.2 –b 3.8 –f STOP_ON_CASE_FAIL
 This time, pet will think the goldman is running the scripts, instead of
you, the one log into the server, and scripts will stop once there is a
case fail.
 2) Set in your scripts (not recommended)
 See more in TestRecorder Training Slides

35. 35
Outline
•Pet Basic
•Quick Start
•I want to do some change
•Advanced: How can I use Pet to write scripts
 Call Flow in Pet
 TestSuite file writing
 TestBed & TestCase file
 Pure Perl programming
 TEA format programming
 Mix-mode programming
•Dig into Pet

36. 36
Call Flow in Pet
•Typical call flow
Import TestSuite
(run testsuite file)
Import TestBed
(run PhyTestBed file)
TestBed Setup
Test Start
Test End
TestBed Cleanup
Run CaseFiles #1
#2
…
#n
Case, Step & TestLog
related functions (defined
in TestRecorder) can be
used here, and all activities
will be recorded by Pet.

37. 37
TestSuite file
•TestSuite file has to be a Perl file
•It could be very simple
•The simplest example:
# A Test Suite Example
use IQsteam::Objects; # it means you will test a IQstream system
•The TestSuite.pl template:
## use the necessary packages here
use IQsteam::Objects;
sub TestBedSetup {
#add your scripts here#
1;
}
sub TestBedClearup {
#add your scripts here#
1;
}
1;

38. 38
TestSuite file is serving the test suite
•Anything related to the test suite should be defined in it
•It may contain
 The packages to be used in the test suite
 The attributes of the test suite
 Test Bed base state definition (TestBed Setup)
 How to recover the test bed after test completes
 Any variables/data to be used in the test suite, but not related to a
particular physical test bed
 Any functions definition for test cases & Test bed initiation
 E.g. the testsuite.pl defined for Lab Pcap running

39. 39
TestBed & Test Case files
•Test Bed design
 Define variables/data related to particular test beds, i.e. Any
parameters may change when you change your test bed to do the
same test
•Test Case file design
 Test Case file is just to define test procedures
 Test Case files should not be modified when you run your test on
different test beds
 Case / Step / TestLog related functions defined in TestRecorder will
be used in TestCase Files
 Test Cases defined in one TestCase file should be self-clean for
better test plan making

40. 40
TestBed & TestCase Files – Pure Perl
•TestBed Example (from WebUI test suite)
our $BrowserType='*iexplore';
our $BrowserURL='http://172.21.5.109';
our $CNIPAddress='172.21.5.109';
our $client_IP='172.21.160.154';
our $PortNumber='4444';
 All the definitions are about the particular test bed, and can be easily
changed when you run it in another test bed.
•TestSuite Example
CaseStart(“Just an example");
{
InfoMsg(“All the following steps should pass!");
Step(3*2==6, "3*2==6");
Like(“Shanghai”, “Sh”, “Shanghai =~ Sh");
Is(3*2, 6, "3*2 is 6?");
} CaseEnd();
……
CaseStart(“Other Cases….”); {…} CaseEnd();
More Functions, please
see TestRecorder Manual