This document summarizes a presentation about adaptive automation techniques that allow tests to recover instead of failing. It discusses using adaptive elements that will select a close match if the expected element is not found, logging the difference but allowing the test to continue. It also covers configuring properties to compare elements and determine the closest match, and using adaptive actions that define recovery patterns to allow tests to retry operations instead of failing. The presentation provides a demo and discusses considerations for when an element change indicates a valid bug versus normal variation.

1.
W9
Test
Automation
10/5/16
13:45
Adaptive
Automation:
Tests
that
Recover
Instead
of
Failing
Presented
by:
Scott
Miles
Aconex
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Scott
Miles
A
quality
engineer
and
ScrumMaster
at
Aconex,
Scott
Miles
is
an
experienced
automation
engineer
and
testing
enthusiast
from
Melbourne,
Australia.
Scott
enjoys
challenging
what
is
accepted
as
standard
practice
and
is
always
striving
to
get
more
out
of
his
team
and
tools.
To
this
end,
he
has
published
several
articles
and
papers
about
the
limitation
of
automation
techniques
and
is
constantly
promoting
the
use
of
smarter
and
better
tools.
A
prominent
member
of
the
Context
Driven
Tester
community
in
Melbourne,
Scott
is
an
organizer
of
both
the
Australian
Testing
Days
2016
conference
as
well
as
the
Test
Engineering
Alliance
of
Melbourne.

3. 9/21/16
1
ADAPTIVE AUTOMATION
Tests that recover instead of failing
Scott Miles
What is Adaptive Automation?
a series of techniques to
make automation more…
adaptive agile smarter

4. 9/21/16
2
In the beginning
Me
What we had
Submit Button
Parent = login_frame
Type = button
X Pos = 233
Y Pos = 841
Colour = #2d145e
…. and so on

5. 9/21/16
3
Required constant maintenance
UI Designers Testers
Caused delay in feedback
Run 1 Run 2 Run 3

6. 9/21/16
4
Update selectors?
Submit Button
Css = #btn_login
What would we miss?

7. 9/21/16
5
Two goals, one solution
Don’t block tests from continuing after failing
Reduce maintenance without reducing coverage
Adaptive Elements!
Adaptive Elements!
Don’t block tests from continuing after failing
If no exact match found, see if there is a close match,
log the difference and continue

8. 9/21/16
6
Adaptive Elements!
Don’t block tests from continuing after failing
If no exact match found, see if there is a close match,
log the difference and continue
Reduce maintenance without reducing coverage
If test continued with new element, allow the user to
automatically update tests with new element selector
When should it happen?
Find Element
Continue
located?
Adaptive Elements
Yes
No

9. 9/21/16
7
Figuring it out
If we cant find the element, which one do we use?
Figuring it out
If we cant find the element, which one do we use?
Compare all elements on the page and find the closest

10. 9/21/16
8
Figuring it out
If we cant find the element, which one do we use?
Compare all elements on the page and find the closest
A page has a lot of elements, how can we speed things up
and increase accuracy?
Figuring it out
If we cant find the element, which one do we use?
Compare all elements on the page and find the closest
A page has a lot of elements, how can we speed things up
and increase accuracy?
We can eliminate almost all elements by excluding
those that haven’t changed since last run

11. 9/21/16
9
Figuring it out
How much effort will it be to update my tests?
Figuring it out
How much effort will it be to update my tests?
Your tests and element definitions wouldn’t need to
change at all. Once the framework is modified, all
existing tests will work with adaptive elements.

12. 9/21/16
10
Figuring it out
How much effort will it be to update my tests?
Your tests and element definitions wouldn’t need to
change at all. Once the framework is modified, all
existing tests will work with adaptive elements.
How can determine which element is the closest match?
Figuring it out
How much effort will it be to update my tests?
Your tests and element definitions wouldn’t need to
change at all. Once the framework is modified, all
existing tests will work with adaptive elements.
How can determine which element is the closest match?
We can compare the each property of an expected
match with the same property of the missing element

13. 9/21/16
11
Configure Properties
Property Measurement
Css selector CSS Path
X (int) Distance
Y (int) Distance
Length (int) Size
Height (int) Size
Type (string) Hierarchy
Parent (string) CSS Path
Text (string) Text
Timestamp (string) -- ignore --
Configure Expectations
**Maximum threshold = 20
Measurement Weight
Distance 100px = 1
Size 10px = 1
Text 1 edit = 1
Hierarchy 1 step = 10
Css Path
1 step = 10,
1 property = 5

14. 9/21/16
12
Example
Definition
Text = “Login”
Type = Button2
Element A
Text = “Login”
Type = Button1
Element B
Text = “Login now”
Type = Button2
?
Example
Definition
Text = “Login”
Type = Button2
Element A
Text = “Login”
Type = Button1
Element B
Text = “Login now”
Type = Button2
1 hierarchy step
10 points
?
∆
4 edit distance
4 points∆

15. 9/21/16
13
Example
Definition
Text = “Login”
Type = Button2
Element B
Text = “Login now”
Type = Button2
Benefit
NEVER UPDATE AN
ELEMENT SELECTOR AGAIN!

16. 9/21/16
14
Demo
Demo removed to reduce upload size
What if…
What if the element changing is a valid bug?

17. 9/21/16
15
What if…
What if the element changing is a valid bug?
You don’t have to accept the elements changes and
at least the test didn’t stop at that point.
What if…
What if the element changing is a valid bug?
You don’t have to accept the elements changes and
at least the test didn’t stop at that point.
What if it selects the wrong element?

18. 9/21/16
16
What if…
What if the element changing is a valid bug?
You don’t have to accept the elements changes and
at least the test didn’t stop at that point.
What if it selects the wrong element?
You can still mark the test as failed afterwards, but at
least we tried!
Adaptive Actions
Find Element
Continue
located?
located?Adaptive Elements
Adaptive Actions
Yes
Yes
No
No

19. 9/21/16
17
In the beginning
In the beginning

20. 9/21/16
18
In the beginning
In the beginning

21. 9/21/16
19
In the beginning
Run Test Failed? Try Again!
Yes
Re-run restrictions

22. 9/21/16
20
Small steps
Run Test Failed? Fix Database
Yes
Select Tests
Try Again!
Shared environment

23. 9/21/16
21
Small steps
Run Test Failed? Fix Database
Yes
Select Tests
Try Again!
Config
Correct?
Fix Config
Try Again!
Yes
No
Interruptions

24. 9/21/16
22
Interruptions
Small steps
Run Test
Failed?
Fix Database
Yes
Select Tests
Try Again!
Config
Correct?
Fix Config
Try Again!
No
No
Popup?
Close Popup
Try Again!
Yes

25. 9/21/16
23
Adaptive Actions
Recovered?Defined RecoveryAdaptive Actions
Application
Recovery
Pattern Recovery
Recovered?
Recovered? Continue
No
Yes
No
Yes
Yes
Adaptive Actions

26. 9/21/16
24
Adaptive Reporting
What techniques?
Find Element
Continue
located?
located?Adaptive Elements
Adaptive Actions
Yes
Yes
No
No

27. 9/21/16
25
Adaptive Elements
Contact Me
Scott Miles
au.linkedin.com/in/scottymiles
@adaptivetester
http://www.adaptivetester.com
http://www.meetup.com/TestEngineeringAlliance/

28. 9/21/16
26
Questions
Any Questions?