Testware Hierarchy for Test Automation

1
Testware
Hierarchy for
Test Automation
Gregory Solovey

The proposed process allows to
perform test automation in parallel
with coding, and can be the base
for continuous integration and agile
testing
2

Developer : unit test
Team: feature test
Project: regression & sanity test
Product: system & load test
Technology: interoperability test
In the traditional approach
automated tests are added
during subsequent test
process phases
In the proposed approach
automated tests are reused
while transitioning to the
subsequent test process phases
Test Automation
3

In the traditional approach test is sequential
(formal relay)
In the proposed approach test is almost parallel
(multi-threaded, with different priorities)
Agile Testing
Requirements Design Coding Unit test
Scenarios
Test plan Testing
Testware Testing
4

Processes
Provide
decomposition/
abstraction
Design Tests
Build Harness
Automate
Testing
There Is Nothing New Under the Sun
5

Processes
Provide
decomposition/
abstraction
Define hierarchy
Present as a
formal model
Design Test
Order error set
Cover all faults
Build Harness
Provide
controllability/
observability
Automate Testing
Write scripts in
test language
Separate business
functions from
implementation
Architecture/
Requirements docs
Test
docs
Design
docs
Testware
Processes
7

Processes
Provide
decomposition/
abstraction
Define hierarchy
Present as a formal
model
Design Test
Order error set
Cover all faults
Build Harness
Provide
controllability/
observability
Automate Testing
Write scripts n test
language
Separate business
functions from
implementation
• Systems hierarchy
• Models hierarchy
• Errors hierarchy
• Tests hierarchy
• Harness elements
hierarchy
• Report hierarchy
• Testware hierarchy
Order is Heaven's first law
9

6/18/2014 10
Process: Provide object decomposition/abstraction
Sub-process: Define object hierarchy
Purpose: Simplify the test design process
• As a rule of thumb, the size of a subsystem to
be tested should be in the order of a hundred
elements (objects, states, messages)
• Represent the system hierarchically, from a
high level description to a less abstract one,
until the desired level of detail is achieved
• Vertical decomposition is achieved by breaking
down an element from black box to white box;
from functional to structural model
• Horizontal decomposition is achieved by
ensuring the independence of the model's
elements.
Object Test Harness Auto

6/18/2014 11
• Provides a natural way to divide the work
• Achieves the fastest way to find most critical
errors by applying tests to the highest level of
object abstraction first
• Defines the boundary between system, feature,
unit test

6/18/2014 12
Nowadays the systems’ architecture provides the right level
of decomposition:
 In the vertical view the direction is to separate the
business and implementation details, allowing for
independent multi-layer development, where the lower
level layers are unaware of the higher layers
 In the horizontal view the direction is towards object
reuse (component based development), with the ability
to assemble the objects in various contexts and access
them via APIs
Hierarchy in SW architecture:
(application (middleware (operating system(firmware (drivers))))
Hierarchy in HW models:
(processors (registers (flip-flops (gates)))
Hierarchy in Protocol models:
(SIP(TCP(IP (PPP(DS)))))

6/18/2014 13
 For systems where this condition is not satisfied
the tester has to provide the decomposition,
document independent objects, review
assumptions with systems engineers, estimate
risk of such assumptions.
 The management has to decide whether to
accept the risk that test is incomplete or request
architectural changes to satisfy the test
requests.

Business
function
Model Test
6/18/2014 14
Sub-process: Present as a formal model
Purpose: Formalize the test design process
• If the architecture/requirements are presented as a set of
business functions, then only subject matter experts can define
tests, based on their expertise. There are no formal rules on how
to build tests from business-oriented requirements; therefore, it
cannot be proved that the tests are complete and consistent.
• If the architecture/requirements are presented as a set of formal
software engineering models, then testers can use well-known
test design approaches to cover all model errors.

Business
function
Model Test
6/18/2014 15
Sub-process: Present as a formal model
Purpose: Formalize the test design process
Data structure
Condition; function
Structural: algorithm; ladder
diagram
Functional: State machine;
Syntax; Object-oriented
Compound: Instruction set
For systems where requirements
are described as business functions
the tester has to transform the plain,
narrative text into formal statements
(models analogies to UML), and
have the result reviewed for
consistency

Business Model Errors Test
6/18/2014 16
• A test cannot prove the correctness of an object, but can identify
its errors. A test is a set of object stimuli that forces the object to
produce different responses in the presence or absence of
errors.
• Errors are possible distortions of the object-to-test’s model: its
elements, structure or behavior.
Process: Design test
Sub-process: Order the error set
Purpose: Increase error resolution

6/18/2014 17
Sub-process: Order the error set
Purpose: Increase error resolution
4
1
3
2
5
4
1
3
2
regression test
each test case attempts to
cover as many errors as
possible (to have fewest
cases)
new feature test
each test case attempts to
cover as few errors as
possible (to reach highest
error resolution)

Model Errors Method Test
6/18/2014 18
Sub-process: Cover all faults
Purpose: Guarantee the coverage with a minimal test set
Model Method
Elementary: Variable; Condition;
Function
Boundary analysis; Graph
coverage; Value changes
Structural: Data/control flow;
Algorithm
Nodes/ Branches/ Path coverage
Functional: State machine;
Syntax; Object-oriented
States and transition functions
coverage; Rules coverage;
Properties, methods coverage
Compound: Instruction set Sensitive Path

6/18/2014 19
a = b or c;
f = a & d;
if (f) then {
} else {
}
D = {a1,a0, b1, b0, c1, c0,d1, d0, f1, f0}
b
c
d
bcd Expected
Result
Errors
101 0 a0 b0 d0 f0
011 0 c0
001 1 a1 b1 c1 f1
110 1 d1
bcd Expected
Result
errors
000 1 f0
111 0 a0 d0 f0
001 1 a1 b1 c1
110 1 d1
011 0 c0
101 0 b0
Regression test
Feature test
f1
a0 d0
f0
a1 b1
c1
c0 b0
d1
Graph model
Error set
Error tree
Program Object Test Harness Auto
19

6/18/2014 20
Process: Build the test harness
Sub-process: Provide controllability/observability
Purpose: Guarantee the test execution
execute
compare
control
observe
Object to test
A test harness is a set of “instruments” that provides the ability to
execute test cases:
• CLI commands to access APIs or test functions
• GUI hidden objects to access database or send transactions
subsystem

21
Recommendation for CLI commands:
• Provide routing to a particular CLI command, using UNIX-like syntax
(“cd”, “ls”, “help”, “show”)
• Print the states and values of the object or group of objects in TLV
format
• Access all object APIs: create, activate, discover, delete, send, etc
• Report errors with the keyword “ERROR”
• Redirect system outcoming messages to CLI interface
• Do not perform any checks other than syntax at the CLI level
• Return “OK/ ERROR” results from the command
Recommendation for GUI objects:
• Use unique names for application objects
• Use naming conventions
• Use automation-friendly controls
Process: Build the test harness
Sub-process: Apply standards
Purpose: Simplify automation

Process: Automate testing
Sub-process: Separate business function from implementation details
Purpose: Provide testware maintenance
Developers and testers have different preferences for test tools:
• Developers prefer to use open source tools (to be able to modify
them) and write test cases in a programming language (C, VB, C#,
PERL, EXPECT)
• Testers feel comfortable using production tools (access to technical
support) and write test cases in specialized test languages (test
framework)
Programming language Test language
variables, arrays, hashes TS,UC,TC,TA w/ parameters
if, case compare
for, while, until loop for TS,UC,TC,TA
function, subroutine, procedure TC {setup, test, compare, reset}
algorithm unconditional sequence 22

Sub-process: Write script in a test language
Purpose: Simplify debugging
Script: capture
& playback
Test Set: subsystem level
Use Case: requirements
level
Test Case: condition level
Test Action: transaction level
Sample framework shows various test object levels:
(TS (UC (TC (TA))))
23

Sub-process: Write script in a test language
Purpose: Guarantee reporting diversity
level verdict report
TS subsystem metrics product management
UC feature full/partial/no functionality project management
TC test case pass/ fail developer/tester
TA transaction ok/ error/ timeout developer/tester
24

25000 20
5000 200
250 250
50 50
SUM UNIQUE Test
TS 2
UC1
TC 1
TA 1 .TA 2
… TC q
TA 1 TA 2
… UC n
TC 1
TA 1 TA 2
… TC w
TA 1 TA 2
… TS n
UC 1
TC 1
TA 1 TA 2
… TC r
TA 1 …. TA 2
… UC m
TS
• XML config file
UC
• Test management system
TC
• Libraries/ Scripts
TA
• Test Language
Sample testware hierarchy for an LTE embedded middleware system:
• XML configuration file contains all test sets , which are invoked
based on changes to subsystems detected during the build
process (for continuous integration)
• The test management system contains the use cases
• The libraries contain the test cases (set-execute-compare-reset
phases)
• The test language describes possible interactions with the object-
to-test
25

Test file call
Test file call
Test file
Test case
Compare
Action
Action
Action
Action
Action
…
…
Test phase
Reset phase
Test case
Test case
Test case
== definition part ========================
Use case
Test case callTest case call
Test case call
Use case
Test case call
Use case
Test case call
== control part ==============================
TEST LIBRARIES
TEST CONTAINERS
TEST SCRIPTS
26

Testware Hierarchy for Test Automation

More Related Content

What's hot

Similar to Testware Hierarchy for Test Automation

Recently uploaded

Testware Hierarchy for Test Automation