Get Automated Testing "Done

How to get Automated Testing
"Done"
Hans Buwalda
LogiGear
www.LogiGear.com
hans@logigear.com
@hansbuwalda
www.TestArchitect.com
www.HappyTester.com

©2016 LogiGear Corporationwww.LogiGear.com
Who is your speaker
• Software testing company, around since 1994
– Foster City, California
– consultancy, training
– test development and automation services
– "test integrated" development services
• Products:
– TestArchitect™, TestArchitect for Visual Studio™
– integrating test development with test management and automation
– based on modularized keyword-driven testing
• LogiGear Magazine:
– themed issues, non-commercial
• Dutch guy, in California with LogiGear since 2001
• Background in math, computer science, management
• Since 1994 focusing on testing and automation
– keywords, agile testing, big testing
Hans Buwalda
LogiGear Corporation
www.LogiGear.com
hans@logigear.com
@hansbuwalda
www.TestArchitect.com
www.HappyTester.com

Relation to code Quality / depth Automation Scalability
Unit Testing
Close relationship
with the code
Singular test
scope, but deep
into the code
Fully automated
by nature
Scalable, grows
with the code,
easy to repeat
Functional
Testing
Usually does not
have a one-on-one
relation with code
Quality and scope
depends on test
design
In particular UI
based automation
can be a challenge
Often a bottle-
neck in scalability
Exploratory
Testing
Human driven, not
seeking a relation
with code
Usually deep and
thorough, good at
finding problems
May or may not
be automated
afterwards
Not meant to be
repeatable. Rather
do a new session
Some test kinds and their scalability (simplified)

• High degree of automation coverage
– under Agile and DevOps automation went from nice-to-have to
must-have
• Scrum teams are well suited to, collectively, own testing
and automation
• QA members (test development and automation) need to
stay in the same sprint with the rest of the team
– developers and product owners are a great help for testing, and
vice versa
– otherwise the task gets harder, and will delay even more
• However, it is not always practical to have fully
automated tests as part of "done" for a sprint . . .
Success factors in Agile (Scrum)

• More test cases needed than is time for
• Shooting a moving target, changes are common
• Lacking technology
• Since QA benefits greatly from close cooperation with PO's and Devs, matters might
get worse once out of sync
• Not always practical to let development wait for QA to finish (true meaning of "done")
QA role may have a hard time keeping
up with the rest of the team
Sprint Sprint Sprint
PO
Dev
QA
PO
Dev
QA
PO
Dev
QA
R
O
L
E
S
PO
Dev

• Design tests to be automation-friendly
– successful automation is not as much a technical challenge as it is a
test design challenge
• Make tests easy to understand for all involved
– so, not too technical please
– note: "easy to understand" does not necessarily mean "easy"
• Invest in testability of the system under test
– one of the main benefits of having a team
– works two ways: good tests facilitate "test driven" development
• Organize the activities well
– I like to consider tests, and their automation, as "products"
– let the testing and automation activities fit the sprint
– support the teams well with infrastructure, technology, services, etc
Approaches

Easy to understand ? ? ?
WebElement element = driver.findElement(By.name(name));
element.sendKeys("mystery magic");
element.submit();
(new WebDriverWait(driver, 10)).until(
new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
return d.getTitle()
.toLowerCase().startsWith("mystery");
}
}
);
System.out.println(driver.getTitle());

More easy to understand: actions
4 actions, each with
an action keyword
and arguments
read from top
to bottom
fragment from a test with actions
acc nr first last
open account 123123 John Doe
acc nr amount
deposit 123123 10.11
deposit 123123 20.22
acc nr expected
check balance 123123 30.33
• The test developer creates tests using actions with keywords and arguments
in a easy to understand spreadsheet format
• Checks are, as much as possible, explicit (specified expected values)
• The automation task focuses on automating the keywords, each keyword is
automated only once
• In addition to keywords you can also use "given-then-when" scenarios (BDD)
for good readability

Supporting BDD with actions
Given a customer previously bought a black sweater from me
And I currently have three black sweaters left in stock
When he returns the sweater for a refund
Then I should have four black sweaters in stock
customer buys, article, color
a customer previously bought a {color} {article} from me
set stock, article, color, amount
I currently have {amount} {color} {article} left in stock
my stock of {color} {sweater} is {amount}
return article, article, color
he returns the {color} {article} for a refund
check stock, article, color
I should have four {color} {article} in stock
article color
customer buys sweater black
article color amount
set stock sweater black 3
article color
return article sweater black
article color amount
check stock sweater black 4
email me for this tool:
hans@logigear.com

• Modularized and keyword-driven method for automated
testing
• Particularly suited for large and complex projects
• High degree of agility, even when used in traditional
projects
• High level of automation, 95% or more is the norm
• Puts the focus on test design, less on technology
"Action Based Testing"

High Level Test Design - Test Development Plan
Objectives
Test Module 1
Test Cases
Test Module 2 Test Module N
Actions
. . .
AUTOMATION
Objectives Objectives
interaction test business test
define the "chapters"
create the "chapters"
create the "words"
make the words work
Test Cases Test Cases
window control value
enter log in user name jdoe
enter log in password car guy
window control property expected
check property log in ok button enabled true
user password
log in jdoe car guy
first last brand model
enter rental Mary Renter Ford Escape
last total
check bill Renter 140.42
Action Based Testing

Test Module Sections
Test Module
Objectives
Initial - setup
Test cases
Final - cleanup

Variables and expressions with keywords
• This test does not need an absolute number for the
available cars, just wants to see if a stock is updated
• As a convention we denote an assignment with ">>"
• The "#" indicates an expression
TEST CASE TC 02 Rent some more cars
car available
get quantity Chevy Volt >> volt
first name last name car
rent car John Green Chevy Volt
rent car Jane White Chevy Volt
car expected
check quantity Chevy Volt # volt - 2

ACTION DEFINITION check balance
user
argument customer
argument amount
window control value
enter balance inquiry last name # customer
window control
click balance inquiry view balance
window control expected
check balance inquiry balance # amount
Re-use actions to make new actions
• In the below example we make a new action
• Existing actions are strung together to create new ones with a
broader scope
• Often steps in low level tests are re-used to create these action
definitions
:
customer amount
check balance Smith 223.45
check balance Jones 0.00
check balance James -330.45
:
use many times in tests:
define in one place:

Example action implementation in Python
# get table object, column number and column count
windowName = LIBRARY.NamedArgument("window")
tableName = LIBRARY.NamedArgument("table")
columnName = LIBRARY.NamedArgument("column")
table = ABT.OpenElement(windowName, tableName)
column = table.GetColumnIndex(columnName)
rowCount = table.GetRowCount()
# check the sort order, row by row
previous = table.GetCellText(0, column)
for i in range(1, rowCount):
current = table.GetCellText(i, column)
if current < previous :
LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)
return
previous = current
LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)
The following action script (in Python) will verify whether the rows in a table are sorted:
find the table in the UI
if a value is smaller than before, fail the test
if all rows are ascending, pass the test
get arguments from the test line
def action_checkSortOrder():

Using the new action
• By keeping an action generic it can be applied for a
variety of situations
• Some examples of using "check sort order":
window table column
check sort order view orders orders table ID
window table column
check sort order annual results regions revenue
window table column
check sort order inventory cars price
window table column
check sort order registration students last name

"Lead Deputy" Testing
• For "multi station" testing, when multiple machines have to
participate in a test in real-time
• For example if a supervisor needs to approve a withdrawal in a bank
teller system
• Can be "sync" and "parallel" (with a rendezvous point)
acc nr amount tx id
withdraw 123123 10,000 >> tx
name
use deputy supervisor
tx id amount
approve # tx 10,000
use lead
"bank teller"
"supervisor"
"bank teller"

Multiple System Access
System
Under Test
Action Automation
API
access
services
access
(REST etc)
UI
access
database
access
Test Modules

Detailed and unfocused tests are hard to automate
Step Description Expected
step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link
step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled
step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled
step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled
step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner
step 21 Enter "acme watch" in the search field The "Search" button is enabled
step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed
step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed
step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2
step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2
step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping"
step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added
step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches
The above test is hard to automate in a maintainable way. Not even the smartest
engineer can "fix" a test design that is not suitable for automation.

First step: breakdown in test modules
• Straightforward Criteria
– Functionality (customers, finances, management information, UI, ...)
– Architecture of the system under test (client, server, protocol, sub systems,
components, modules, ...)
– Kind of test (navigation flow, negative tests, response time, ...)
• Additional Criteria
– Stakeholders (like "Accounting", "Compliance", "HR", ...)
– Complexity of the test (put complex tests in separate modules)
– Technical aspects of execution (picture checks, special hardware, multi-station, ...)
– Overall project planning (availability of information, timelines, sprints, ...)
– Risks involved (extra test modules for high risk areas)
– Ambition level (smoke test, regression, aggressive, …)
• Business or interaction
– Business tests: does the application reflect the business well
– Interaction tests: can a user (or another system) interact with the application
– mixing business and interaction is one of the most common, and one of the most
fatal, "anti patterns" in automated testing

• Business objects
– examples: cars, invoices, locations, etc
– tests focus on life cycles
– a majority of tests tend to fit in this category
• Business flows
– end-to-end workflow style test,
– example: create, fulfill, pay and close an order
• "Other tests"
– functions and features, like premium calculation or PDF output
– administration, users, security, authorizations
– graphics
– technologies, protocols, ...
– customization, extensibility
– . . .
Example of breakdown considerations

Example Top Level Structure
Project create, update, delete/close
copy, move
categorize, enumerate, identify
convert, serialize, export/import, ...
UI, dialogs, forms, pages
input (validation, defaulting, dependencies)
flows (primary paths, alternate paths)
keyboard shortcuts, keyboard controls, ...
. . .
"Cars"
Lifecycles, data operations
Interaction
Functions and Features
Technologies, protocols, controls
Data (handling, quality, ETL, ...)
Security, authorization, admin
Graphics, multi-media, charts, ...
Interoperability
Customizing, extensibility
Business Flows
Concurrency, race conditions, ...
Business Objects
processes, transactions, end-to-end, day in the life,
combinations of flows, ...
calculations, analyses, PDF output, ...
"Customers"
. . .

What about existing tests?
• Compare to moving house:
– some effort can't be avoided
– be selective, edit your stuff,
• look at the future, not the past
– first decide where to put what, then put it there
– moving is an opportunity, you may not get such chance again soon
• Follow the module approach
– define the modules and their scope as if from scratch
– use the existing test cases in two ways:
• verify completeness
• harvest and re-use them for tests and for actions
– avoid porting over "step by step", in particular avoid over-checking

Scope of a test module
• Always know the scope of the test module
• The scope should be unambiguous
• The scope determines many things:
– what the test objectives are
– which test cases to expect
– what level of actions to use
– what the checks are about and which events should generate a
warning or error (if a “lower” functionality is wrong)
• In other words: keep your eye on the ball . . .

Tip: Use right level actions
project task
check task in project Drill Assembly Plan of Approach
 Better to have a business level action and hide the details in an action
definition:
 But do show details when it matters. The example below is too high
level, requires drill down into action definition to understand what is
tested.
window tree tree item path
click tree item main projects /Projects/Drill Assembly
window list item
check list item exists main tasks Plan of Approach

Stable and Robust Automation
• Test your automation
– develop separate tests for the actions and interface mappings themselves
• Identifying properties of UI elements
– a lot of time is spent on identifying and re-identifying UI elements
• "active" timing
– wait based on conditions in the system under test, avoid fixed “sleeps”
– if needed request developers to provide hooks
• Testing and automation friendly system designs
– with well-defined components, services, tiers, etc
• Keep an eye on the test design
– unsuccessful automation virtually always can be traced back to poor test design

Mapping an interface
• An interface mapping (common in test tools) will map windows and controls to names
• When the interface of an application changes, you only have to update this in one place
• The interface mapping is a key step in your automation success, allocate time to design
it well, in particular naming and choosing identifying properties
INTERFACE ENTITY library
interface entity setting title {.*Music Library}
name class label
interface element title text Title:
interface element artist text Artist:
interface element file size text File size (Kb):
name class position
interface element playing time text textbox 4
interface element file type text textbox 5
interface element bitrate text textbox 6
name class position
interface element music treeview treeview 1

Hidden Identifying Property

Mapping the interface
(style we use in our product)
• An interface mapping will map windows and controls to names
• When the interface of an application changes, you only have to update this in one place
• The interface mapping is a key step in your automation success, allocate time to design it well

• Testability should be understood as a "must have" requirement
– first question in a development project: "how do we test this?"
• Define (assign) identifying properties for any interface element (control,
HTML element, etc)
• Have hooks for timing
– avoid "hard sleeps" at all costs
• Have white-box access to anything interesting:
– input data (ability to emulate)
– output data (what is underlying data being displayed)
– random generators (can I set a seed?)
– states (like in a game)
– objects displayed (like monsters in a game)
– pay attention to security, don't leave hacking doors open
• A well-designed system, with tiers, components, services, is also very
helpful for test design and automation
Testability, some key items

Test design and automation in agile
High Level Test
Design (organization
of test modules)
Test Module Development
Interface Definition
Action Automation
Test Execution
Sprint Products
Product
Backlog
Test re-use
Automation re-use
product
owner
team
prod owner
& team
User stories
Documentation
Domain understanding
Acceptance Criteria
PO Questions
Situations
Relations
Agile life cycle
Test development
Main Level Test Modules
Interaction Test Modules
Cross over Test Modules

Pre-conditions for Sprints
• Agreement on the approach:
– is testability a requirement for the software?
• Understanding of subject matter, testing, automation,
method, best practices, etc
• Automation technology available and working
– UI, non-UI, custom controls, graphics, embedded, etc
• A plan on how you to deal with data and environments
To discuss an approach, consider daily "sit down" meetings with some or
all members to coach and evaluate
 an end-of-day counterpart to the early-morning "stand up" meetings
 short and friendly, not about progress and impediments, but about practices and
experiences with them (like "what actions did you use?")
 a few meetings may suffice

Facilitation outside sprints, example
Sprint
Have a facility (internal or outsourced) to help the agile life cycle
• keep up, catch up, continue beyond "done"
• one outsourced facility can support multiple teams, smooth out peaks and valleys
• allow sprint teams to focus on development priorities
• team is responsible, and decides if, what and when to outsource
Sprint
Test Definition & Development
T
E
A
M
Additional Test Development
Additional Automation
F
A
C
I
L
I
T
Y
Test Development
Automation
Automation
Test Definition & Development
Automation

• QA needs to keep up with the rest of the team
• Test design is a key driver for automation success/failure
• Testability, as a team responsibility, is a core requirement
• Organizing the testing well in the sprints gets more of it
"done"
• Having an agile facilitation model can support the QA's
to stay in sync
Summary

Get Automated Testing "Done

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Get Automated Testing "Done

Similar to Get Automated Testing "Done (20)

More from TEST Huddle

More from TEST Huddle (20)

Recently uploaded

Recently uploaded (20)

Get Automated Testing "Done