Although classic test automation, which usually focuses on regression testing, has its its place in testing, there is much more you can do to improve testing productivity and its value to the project and your organization. Through experience-based examples, video clips, and demonstrations, John Fodeh shares one company’s innovation journey to improve its test automation practice. John illustrates how they learned to apply automated “test monkeys” that explore the software in new ways each time a test is executed. Then, he describes how the test team uses weighted probability tables to increase each test’s “intelligence” factor. Find out how they implemented model-based testing to improve automation effectiveness and how this practice led to the even more valuable behavior-driven testing approach they employ today. With these and other alternative approaches you, too, can get more mileage from your automation efforts. Join John to get inspired and start your own journey of innovation with new ideas that enhance your test automation strategy.

1. T15
Test Automation
5/2/2013 1:30:00 PM
Innovations in Test Automation: It's
Not All about Regression
Presented by:
John Fodeh
Cognizant
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. John Fodeh
Managing Cognizant's testing services in the Nordics, John Fodeh has more than fifteen years of
experience in software testing with extensive expertise in quality management, process improvement,
testing tools, and strategy. John has helped organizations throughout Europe deal with issues in the areas
of banking, insurance, retail, life sciences, and government. He has written a number of articles and
contributed to books, including Experiences of Test Automation. John is an active member of special
interest groups in software testing, chairman of the Danish Software Testing Board, and speaker at
international testing conferences including STAREAST, STARWEST, and EuroSTAR. Contact John at
john.fodeh@cognizant.com.

3. 4/16/2013
Innovations in Test
Automation
It’s Not All about Regression
John Fodeh
Cognizant Technology Solutions
john.fodeh@cognizant.com
Outline
•
•
•
•
•
•
Innovation and testing
The promise of automation
Going beyond regression testing
Automated Test Monkeys
Model-Based and Behaviour-Driven approaches
The human factor in automation
| ©2012, Cognizant
2
1

4. 4/16/2013
Innovation?
The word “innovation” derives
from the Latin word innovatus
The noun form of innovare is "to
renew or change," stemming
from in—"into" + novus—"new“
It is the process to renew an
existing idea
3
| ©2012, Cognizant
3
The Promise of Test Automation
• Replacing repetitive and
tedious manual tasks
• Ensuring the consistency and
repeatability of tests
• Performing tests that are
difficult to run manually
• Accelerating test execution
| ©2012, Cognizant
* Image source: Rosemont Engineering
4
2

5. 4/16/2013
“Insanity is doing the same thing over and
over again and expecting different results”
| ©2012, Cognizant
5
Traditional Test Automation
Typically
Automation of regression tests
Purpose
Testing of a previously tested program following
modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made. It is performed
when the software or its environment is changed.
[ISTQB Glossary, v2.1]
| ©2012, Cognizant
6
3

6. 4/16/2013
Characteristics of Traditional Test Automation
•
•
•
•
Static
Simple
Synchronized
Vulnerable
| ©2012, Cognizant
7
Testing Beyond the GUI
GUI
Business Logic
Middleware
Data
8
| ©2012, Cognizant
4

7. 4/16/2013
Monkey Testing
Monkey Testing refers to the
process of randomly exercising a
software program by means of an
automated test tool
| ©2012, Cognizant
9
Test Monkey
Action List
#script file: Test4.asc
KEY L;
KEY 5, LONG;
CP_KEY FOCUS_UP;
CP_KEY FOCUS_UP;
#KEY BOX_SIZE_UP
KEY 8;
TRACKBALL 0, 64;
KEY SHIFT, R;
Log file
#script file: Test4.asc
CP_KEY FOCUS_UP;
TRACKBALL 0, 64;
KEY L;
KEY 5, LONG;
KEY 8;
CP_KEY FOCUS_UP;
Random
Test Tool
Application
Under Test
Utilities
| ©2012, Cognizant
10
5

8. 4/16/2013
Basic Features
A test monkey should:
• Select randomly from input range
• Enter input to AUT through the user interface
• Detect “life signs” of AUT
• Have robust logging facility
• (Re-) start and initialize AUT
• Have limited application knowledge but general
environment awareness
| ©2012, Cognizant
11
Mean number of random operations between
Failures
Metrics – Application Reliability
100000
Release
Criterion
80000
60000
40000
Entry
Criterion
20000
0
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21
Build no.
| ©2012, Cognizant
12
6

9. 4/16/2013
Added Value
•
•
•
•
•
Early testing
Cost effective
Negative testing
Long and complex test runs
Reliability indicator
| ©2012, Cognizant
13
Limitations
•
•
•
•
Misses obvious defects
Does not emulate real use situations
Long tests runs can be difficult to debug
For a reliable statistical basis, long and multiple test
runs are needed
Monkey Testing is a supplement
to your manual and automated
testing - not a replacement!
| ©2012, Cognizant
14
7

10. 4/16/2013
Advanced Test Monkeys
• Wide application knowledge
• Application modeled in state table
• Illegal input to test error handling and recovery
• Emulate real and complex use scenarios
• Effective in finding defects
• Higher development and maintenance cost
| ©2012, Cognizant
15
State Transition Testing
State table
Start
Start state
State
Scanning
Action
Action
Adjust frequency
Input
End
Probability range
State 45
3
Adjust brightness
Abort
20
5
1
Script
End state
20 Scanning
100 Scanning
Displaying (abort)
Edit preferences
10
Editing preferences
Increment depth
20
Scanning
Random
Test Tool
Application
Under Test
Log file
Utilities
| ©2012, Cognizant
16
8

11. 4/16/2013
Intelligent Randomness
• Probability Tables
• Focus on selected areas
• Weight on specific use
scenarios
• Continuous expansion and
improvement
• Possibility to exclude
certain parts
Action
25%
10%
Enter Text
Select Text
5%
Delete
5%
Copy
5%
Cut
…
Task n
* source: Whitmill, Kelly
| ©2012, Cognizant
17
Model-Based Automation
Model table
State
Start
State
Action
Script
End
State
Test Tool
Application
Under Test
Log file
Utilities
| ©2012, Cognizant
18
9

12. 4/16/2013
Behavior-Driven approach
Based on Behavioral specifications
User Stories defined using ubiquitous language
Principles of Test Driven Development
Collaborative approach for business analysts,
Software development & testing
• Shared tools and process
•
•
•
•
| ©2012, Cognizant
19
People Issues
• Test automation depends on the competence,
creativity and motivation of your team
• Dedicated resources are required
• Management support is essential
| ©2012, Cognizant
20
10

13. 4/16/2013
Summary
• No “one size fits all” solution
• Maturity is about analyzing the current practice and
finding ways to do evolve
• Are you ready for your innovation journey?
| ©2012, Cognizant
21
11