Why do test automation implementations often result in budget and schedule overruns and often diverge from the test objectives they were meant to support? Because test automation is a form of software development—but rarely is treated as such. Lee Barnes discusses how the application of software development practices, specifically agile principles, to test automation efforts helps ensure a reliable and maintainable solution that is closely aligned with both testing AND business objectives. Benefits of incorporating agile practices in test automation development include: increased customer involvement and visibility, scope flexibility, and budget and schedule control. Lee presents techniques for mapping agile processes, artifacts, and tools to test automation development projects. Discussion will cover topics of agile components including user stories, task board, backlog, and definition of done in a test automation context. In addition, Lee will examine the evolution of test automation artifacts over the course of several sprints. Take back techniques and examples for applying agile development principles to test automation efforts.

1.
T15
Test
Automation
5/11/17
13:30
Be
More
Effective:
Test
Automation
below
the
UI
Presented
by:
Ashish
Mehta
Infosys
Limited
Sohail
Farooqui
Allstate
Insurance
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Ashish
Mehta
Ashish
Mehta
has
more
than
fifteen
years
of
global
experience
in
software
quality
assurance
and
development.
He
has
worked
with
clients
to
establish
independent
QA
organizations,
multi-­‐stack
test
competencies,
and
mature
testing
practices
in
system
test
automation,
middleware
automation,
data-­‐warehouse,
and
end-­‐to-­‐end
integrations.
Ashish's
passion
for
new
technology
has
helped
him
collaborate
with
architects
and
developers
in
designing
and
developing
optimal
test
solutions.
Sohail
Farooqui
As
an
Enterprise
Test
and
Release
Manager
in
the
Allstate
Insurance
strategy
group,
Sohail
Farooqui
provides
strategic
direction
and
thought
leadership
in
the
functional,
regression,
performance,
and
service
architecture
testing
spaces
across
several
mission
critical
and
supporting
applications
while
also
serving
on
the
core
leadership
team
providing
guidance
in
the
conversion
of
Allstate
testing
organization
from
a
waterfall
to
an
agile
methodology.
As
a
Strategist,
Sohail
is
critically
focused
on
quality
and
balancing
cost
to
business
with
testing
efficiency.

3. BE
MORE
EFFECTIVE
MAY
7–12,
2017
|
ORLANDO,
FL
BELOW
THE
UI
Sohail
Farooqui
&
Ashish
Mehta
TEST
AUTOMATION

4. 2
High
Test
Automa;on
Scripts
Maintenance
Efforts
High
Test
Cycle
Times
Increased
Tool
Licenses
Cost
Large
Enterprise
Insurance
Policy
Administra;on
System
demanding
Con;nual
Regression
to
Maintain
Op;mal
Product
Quality
THE
PROBLEM

5. 3
11,000+
REGRESSION
SCENARIOS
10%
Year-­‐on-­‐Year
increase
in
Scripts
Volume
3,000+
HOURS
PER
MONTH
Higher
Test
Cycle
Times
Increasing
Licenses
Cost
UI
based
regression
automa;on
scripts
naviga;ng
across
several
pages
to
complete
policy
transac;ons
CURRENT
STATE
-­‐
UI
REGRESSION
TESTING
$500K
ANNUAL
LICENSES
COST

6. PROBLEM
DISCOVERY
No
visibility
in
the
tests
UI
based
Test
Automa;on
Applica;on
UI
Business
Layer
Components
Peripheral
Applica;ons
No
visibility
in
the
tests
Higher
defects
concentra;on
UI
Layer
Business
Layer
78%
22%
Defects
Distribu;on

7. PROBLEM
SOLVING
NEW
AUTOMATE
AT
BUSINESS
LAYER
(BELOW
THE
UI)
RENEW
OPTIMIZE
EXISTING
UI
REGRESSION
SUITE

8. AUTOMATE
AT
BUSINESS
LAYER
BELOW
THE
UI
Applica;on
UI
Eliminate
UI
dependency
Business
Layer
Components
1
Peripheral
Applica;ons
Test
run
2
Test
run
1
2 3
Planned
Release
(Test
version)
Baseline
Code
(Prod
version)
Reset
transacIon
(s)
dataset
to
iniIal
set
1
Run
dataset
through
baseline
code
2
Run
dataset
through
test
code
3
Auto
compare
results

9. AUTOMATE
AT
BUSINESS
LAYER
-­‐
BELOW
THE
UI
MIMIC
UI
TRANSACTIONS
AT
BUSINESS
LAYER
Goal
Guiding
Principles
Exploit
the
system
capability
Intercept
at
business
layer
Supply
historical
system
of
records

10. Not
all
UI
tests
can
be
moved
to
business
layer,
since
UI
layer
is
also
a
source
of
failure
point.
Regression
tests
at
UI
and
business
layer
need
to
co-­‐exist.
SHIFT
THE
TESTS
FROM
UI
>
BUSINESS
LAYER
OPTIMIZE
EXISTING
UI
REGRESSION
SUITE
Categorize
UI
workflows
UI
tests
covers
unique
UI
elements
Remaining
tests
moved
to
business
layer
Iden;fy
unique
UI
elements

11. 9
CO-­‐EXISTENCE
UI
&
BUSINESS
LAYER
AUTOMATION
Test
distribu;on
between
UI
&
business
layers
11,000
Test
Scenarios
2,500
at
Business
Layer
8,500
at
UI
Layer
NEW
RENEW

12. 10
EARLY
VALIDATION
TIME/TRANSACTION
VALID
DEFECTS
#
OUTCOMES
BUSINESS
LAYER
AUTOMATION
2
weeks
&
55%
defects
30
Min
17
Min
35%
65%
COST,
QUALITY
&
TIME
COMPARISON
UI
BASED
AUTOMATION

13. 33%
14%
52%
BL
DEFECTS
UI
DEFECTS
COMMON
DEFECTS
BETWEEN
BL
&
UI
BUSINESS
LAYER
AUTOMATION
UI
BASED
AUTOMATION
OUTCOMES
TEST
EFFECTIVENESS

14. 12
23%
Tests
Shi]ed
to
Business
Layer
REDUCED
COST
&
EFFORT
WITH
IMPROVED
QUALITY
BENEFITS
UI
-­‐
8,500
BL
-­‐
2,500
DISTRIBUTED
TESTS
15%
ReducIon
in
Monthly
Person
Efforts
35%
ReducIon
in
License
Costs
$500K
ANNUAL
SAVINGS
$200K
ANNUAL
SAVINGS

15. 13
TECHNOLOGY
MIGRATION
&
NEW
CONSUMER
ONBOARDING
LEVERAGE
FOR
END-­‐TO-­‐
END/REGRESSION
TESTS
FUNCTIONAL
TESTING
SHIFT-­‐LEFT:
EARLY
DEFECT
DETECTION
DURING
FUNCTIONAL
PHASES
PERFORMANCE
TESTING
TEST
PERFORMANCE
AT
BUSINESS
LAYER
IN
ISOLATION
FOR
BETTER
BOTTLENECK
ANALYSIS
SOLUTION
PROSPECTS

16. Q&A

17. sfarp@allstate.com
Please contact us at
amehw@allstate.com