Patterns—proven, repeatable solutions to common situations that occur again and again—are commonly used in development and to a lesser extent in testing. In addition to patterns, various anti-patterns have been discovered. These are common responses to recurring problems that, while promising, are actually counterproductive. Hans Buwalda describes a set of such anti-patterns that he commonly sees in automated test design and that he feels inhibit scalability and maintainability. Hans discusses anti-patterns including Enter Enter Click Click, Interaction Heavy, Lifeless, Lame, Clueless and a number of others. Knowing these anti-patterns will help you spot potential bottlenecks that could make your own tests unnecessarily difficult to automate. Hans presents thirteen distinct patterns that you can look for in your tests. Feel free to bring in your own anti-patterns and situations you have experienced that you feel are counterproductive toward achieving manageable and maintainable automation.

1. 5/13/16
1
© 2016 LogiGear Corporation. All Rights Reserved
STAREAST 2016, Track Session W3
Orlando, Florida
Wednesday, May 4th, 2016
11.30 AM – 12.30 PM
"Anti-Patterns" for
Automated Testing
Mr. Playback
Hans Buwalda
LogiGear
hans@logigear.com
@hansbuwalda
www.happytester.com
© 2016 LogiGear
Domain Language Approaches: Keywords
4 actions, each
with an action
keyword and
arguments
read from top
to bottom
fragment from a test with actions
acc nr first last
open account 123123 John Doe
acc nr amount
deposit 123123 10.11
deposit 123123 20.22
acc nr expected
check balance 123123 30.33
• The test developer creates tests using "actions" with keywords and
arguments
• Checks are, as much as possible, explicit (specified expected values)
• The automation task focuses on automating the keywords, each keyword
is automated only once
• This technique can be very scalable. A similar approach is behavior
based testing, which also works with human readable tests, but is more
verbose

2. 5/13/16
2
© 2016 LogiGear
High Level Test Design - Test Development Plan
Objectives
Test Module 1
Test Cases
Test Module 2 Test Module N
Actions
. . .
AUTOMATION
Objectives Objectives
interaction test business test
Overview Action Based Testing
define the "chapters"
create the "chapters"
create the "words"
make the words work
Test Cases Test Cases
window control value
enter log in user name jdoe
enter log in password car guy
window control property expected
check property log in ok button enabled true
user password
log in jdoe car guy
first last brand model
enter rental Mary Renter Ford Escape
last total
check bill Renter 140.42
© 2016 LogiGear
§ Business objects and business flows
- objects are like cars, invoices, locations, etc
- flows are like create, fulfill, pay and close an order
§ Other tests
- functions and features, like premium calculation or PDF output
- administration, users, security, authorizations
- graphics
- technologies, protocols, ...
- customization, extensibility
- interoperability
- . . .
§ Business versus interaction
- differentiate within business objects and all other categories
- interaction can be further differentiated into: values, UI, keyboard, etc
- also, for every category, look at negative tests, flows, aggressive
tests, etc
Examples of considerations

3. 5/13/16
3
© 2016 LogiGear
Example Top Level Structure
Project
create, retrieve, update, delete / deactivate ("CRUD")
copy, move
categorize, enumerate, identify
convert, serialize, export/import, ...
UI, dialogs, forms, pages
input (validation, defaulting, dependencies)
flows (primary paths, alternate paths)
keyboard shortcuts, keyboard controls, ...
. . .
<business object 1>
Lifecycles, data operations
Interaction
Functions and Features
Technologies, protocols, controls
Data (handling, quality, ETL, ...)
Security, authorization, admin
Graphics, multi-media, charts, ...
Load, performance
Interoperability
Customizing, extensibility
Business Flows
Concurrency, race conditions, ...
Business Objects
processes, transactions, end-to-end, day in the life,
combinations of flows, ...
calculations, analyses, PDF output, ...
© 2016 LogiGear
Eye on the ball, Scope
§ Always know the scope of a test module
§ The scope should be unambiguous
§ The scope determines many things:
- what the test objectives are
- which test cases to expect
- what level of actions to use
- what the checks are about and which events should
generate a warning or error (if a “lower” functionality
is wrong)

4. 5/13/16
4
© 2016 LogiGear
Example of a Test
Step Description Expected
step
16 Open
http://www.bigstore.com The
"BIG
Store"
main
page
is
displayed,
with
a
"sign
in"
link
step
17 Click
on
"Sign
In",
upper
right
corner A
sign
in
dialog
shows,
the
"Sign
in"
button
is
disabled
step
18 Enter
"johnd"
in
the
user
name
field The
"Sign
In"
button
is
still
disabled
step
19 Enter
"bigtester"
in
the
password
field Now
the
"Sign
In"
button
is
enabled
step
20 Click
on
the
"Sign
in"
button The
page
now
shows
"Hello
John"
in
the
upper
right
corner
step
21 Enter
"acme
watch"
in
the
search
field The
"Search"
button
is
enabled
step
22 Click
on
the
"Search"
button 5
watches
of
Acme
Corporation
are
displayed
step
23 Double
click
on
"Acme
Super
Watch
2" The
details
page
of
the
Acme
Super
Watch
2
is
displayed
step
24 Verify
the
picture
of
the
watch The
picture
should
show
a
black
Acme
Super
Watch
2
step
25 Select
"red"
in
the
"Color"
dropdown
list The
picture
now
shows
a
black
Acme
Super
Watch
2
step
26 Type
2
in
the
"Order
quantity"
textbox The
price
in
the
right
shows
"$79.00
+
Free
Shipping"
step
27 Click
on
"Add
to
cart" The
status
panel
shows
"Acme
Super
Watch
2"
added
step
28 Click
on
"Check
out" The
Cart
Check
Out
open,
with
the
2
Acme
Super
Watches
© 2016 LogiGear
GWT scenario
Scenario: Applicant is at Stop Test alert and chooses to Stop Test
Given User turns off Password required option for Drive Test
And User has logged in by Traffic Applicant account
And User is at the Assessments Take a Test page
And User clicks the Traffic Test link
And User clicks the Next button
And User clicks the Sheet radio button in Mode page if displayed
And User clicks the Start button
And User waits for test start
And User clicks the Stop Test button
When User clicks the Confirm Stop Test button
And User enters the correct applicant password
And User clicks the Confirm Stop Test button
Then The Test is Over should be displayed in the Message label
Then Value of the Message label should be The test is over.
And The Welcome to Traffic Testing page should be displayed

5. 5/13/16
5
© 2016 LogiGear
What is an "anti-pattern"
§ A practice considered harmful
- opposite of a "pattern"
§ Has an easy to remember name
§ Can be helpful to spot or describe risks
§ In the case of automated tests, the risks you usually
try to avoid are in areas like:
- effectiveness
- efficiency
- readability
- manageability
- maintainability
- etc
© 2016 LogiGear
Using the anti-patterns in this presentation
§ Experimental, input welcome
§ There are overlaps
- in not-so-great test designs multiple patterns tend to apply
simultaneously
§ Input for design decisions, not absolute truths
- "decisions have consequences", but those can be acceptable in
sometimes
- not everything applies to all situations, not every anti-pattern
occurrence is a bad thing
- see them more as a starting point of a discussion
§ See anti-patterns as symptoms
- lack of an overall high level test design
- lack of experience of the test developer

6. 5/13/16
6
© 2016 LogiGear
"Anti-patterns" (informal)
§ Interaction Heavy – Not having many business tests
§ Lame – No depth or variety, no testing techniques used
§ Enter Enter Click Click – Test steps are too detailed
§ No Life – Missing life cycle steps of business objects
§ Clueless – No clear scope for the tests
§ Cocktail – Interaction tests mixed with business tests
§ Over-Checking – Checks not relevant for the scope
§ Sneaky Checking – Checks hidden in actions
§ Action Explosion – Many actions, hardly different
§ Mystery Actions – Cryptic actions, unclear what they do
§ Techno – Actions and tests that look like code
§ Passive Timing – Hard-coded "sleeps" for timing issues
© 2016 LogiGear
Interaction Heavy
§ Not having many business tests
§ What to look for:
- seeing only test cases like these:
• can I successfully create a new customer
• can I successfully update an order
• ...
- seeing the test tree follow the UI organization
§ The UI is the most visible part of an application. It is not
uncommon to have a "what you see is what you test"
tendency
§ What is missing is a clear understanding and verification if
the application is fit for business

7. 5/13/16
7
© 2016 LogiGear
Lame
§ No depth or variety, no testing techniques used
§ What to look for:
- boredom
- no suprises
- no techniques used
- not many, and trivial, bugs
§ Consider exploratory approaches, not just for
manual testing via the UI, but also for test case
design
- in other words: make it more fun
- involve domain experts
© 2016 LogiGear
Enter Enter Click Click
§ Test steps are too detailed
§ What to look for:
- long sequences with only built-in low level actions like
"enter", "click", "select menu", etc
- high sensitivity to UI changes
- not able to understand the goal of the tests easily
§ Example:
- an application in the oil industry
- over 6000 test cases with a length of 2000 test lines or
more
§ Rather than trying to optimize, create a high-level
test design first ("from scratch")

8. 5/13/16
8
© 2016 LogiGear
No Life
§ Missing life cycle steps of business objects
- like CRUD: Create-Read-Update-Delete
§ What to look for:
- is it clear what the business objects are?
- do they show in the hierarchy
- in particular are there modifications ("update") and
removal/closure ("delete") tests?
- is there variation in such tests
§ Life-cycle tests are usually not difficult for define
§ I usually prefer to design life-cycle tests as
business tests, not as interaction tests
© 2016 LogiGear
Clueless
§ No clear scope for the tests
§ What to look for:
- look through the test lines. Assuming you know the
domain, is it clear to you what the purpose of the test
is?
- is the scope mixed. For example a customer is
created, and tested, and a order is placed by that
customer in the same test
§ Having clear scopes for tests (in particular test
modules) helps organize and maintain them
§ It also helps follow up
- for example: "Credit Card Payments" has fails

9. 5/13/16
9
© 2016 LogiGear
Cocktail
§ Interaction tests mixed with business tests
§ This is quite common
§ What to look for
- a test is testing business rules and processes, like
calculation of a home loan premiums in various situations
- and at the same time is doing UI checks
§ This anti-pattern is usually a symptom of other
anti-patterns (like "clueless")
© 2016 LogiGear
Over-Checking
§ Checks not relevant for the scope
§ Testers, and their management, often believe everything
a test script goes through needs to be tested
§ Automation engineers want to make sure the navigation is
still ok (consider using an "assert" action instead)
§ The result can be less clarity and harder maintenance
§ In a clear test design over-checking can quite easily be
avoided

10. 5/13/16
10
© 2016 LogiGear
Example of a Test Module
TEST MODULE Order processing
start system
TEST CASE TC 01 Order for tablets
user password
login jdoe doedoe
window
check window exists welcome
order id cust id article price quantity
create order AB123 W3454X tablet 198.95 5
order id total
check order total AB123 994.75
. . .
© 2016 LogiGear
Sneaky Checking
§ Checks hidden in actions
§ What to look for:
- in action definitions or scripts there are checks that are
not visible in the test module
- it are often interaction checks, like "does this window
exist", after a click
§ Such checks may influence your statistics, and
increase maintenance sensitivity
§ If it is "just to make sure", use an "assert" action:
- producing a warning or an error if the condition isn't met
- in most cases this is not needed, the tool will complain

11. 5/13/16
11
© 2016 LogiGear
Action Explosion
§ Many actions, hardly different
§ What to look for:
- every step and variation of a step has its own action
- many actions are used only once or twice
- there are about as many actions as tests
§ It is the opposite of the "enter enter click click"
- the challenge is to find a balance between not-enough-
actions and too-many-actions
§ Example:
open financials
open financials - payments tab
open financials - payments tab - approval sub tab
open financials - payments tab - approval sub tab - approved changes panel
© 2016 LogiGear
Consider using "mid-level" actions
§ "Low level": detailed interaction with the UI (or API)
- generic, do not show any functional or business logic
- examples: "click", "expand tree node", "select menu"
§ "High level": a business domain operation or check on the
application under test
- hide the interaction
- examples: "enter customer", "rent car", "check balance"
§ "Mid level": common sequences at a more detailed
application level
- usually to wrap a form or dialog
- for use in high level actions
- greatly enhance maintainability
- example: "enter address fields"
enter customer
enter address fields
enter select set . . .. . .

12. 5/13/16
12
© 2016 LogiGear
Mystery Actions
§ Cryptic actions, unclear what they do
§ What to look for:
- action or arguments that are not clear to those who know
the domain
- generic actions like "check invoice"
- are not business related or UI related
§ Unclear actions:
- might be used a wrongly
- might not be found, delimiting their re-use
- hinder impact analysis and follow up
© 2016 LogiGear
Techno
§ Actions and tests that look like code
- "preferably" C++ code
- work well to impress people, but not for much else
§ What to look for:
- not English or another natural language
- all kinds of special characters
- lack of spacing
- in general names that are _NOts0EasY_2REad
§ Often a result of developers at the helm of the
test design

13. 5/13/16
13
© 2016 LogiGear
§ Hard-coded "sleeps" for timing issues
§ What to look for
- "sleep" or "wait" actions
- sometimes hidden in other actions
- arguments are sometimes variables, this is not much of an improvement
§ Passive timing
- big nono, the number one archenemy of automation J
- at best slows down your tests
- but if not long enough make the test unstable
- in VM's the problem is even worse than in physical machines
- if used, the team should clearly escalate it, define it as a risk
§ Better approach: Active timing
- wait for a measurable event
- usually the wait is up to a, generous, maximum time
- for UI many such waits are built into the automation tool
- consider involving developers to help you (see my slides of my "BIG Testing" tutorial for
more suggestions)
Passive Timing
© 2016 LogiGear
Be careful in communication!
§ Avoid even the hint of attack
§ Make sure all people involved understand, and
appreciate, that anti-patterns are applied
§ In particular watch your writing, like emails and
reports
- good fuel for "flaming"
- avoid writing something like: these tests are "lame"
- talk about test design
- teamwork rules suppreme

14. 5/13/16
14
© 2016 LogiGear
Summary
§ Automation success depends on design decisions, most of
which are test design decisions
§ Anti-patterns can help identify risks, and start discussions
§ Anti-patterns are not (meant to be) absolute rules
§ Test design and automation are team work, part of a larger
route to success