In today’s environment of plummeting software delivery cycle times, test automation becomes a more critical and strategic necessity. How can we possibly keep up with software delivery's explosive pace while retaining satisfactory test coverage? How do we keep the reins on costs and reduce risk? Carl Nagle maintains that the long-term solution is a greater level of “sustainable” test automation. The Software Automation Framework Support (SAFS) method separates test design from test execution with a data-driven, action-based approach that encapsulates volatile application-specific data into readily localizable “maps” for simple maintenance. Test designs (scripts or programs) are completely independent of the ready-to-run SAFS engines that will execute them. And since the test design methodology does not change over long periods of time, testers can focus more on getting robust automation in place quickly and pay less attention to each new technology, testing tool, or test IDE. Join Carl to learn how test automation thrives when testers and tools are not tied up in application-specific silos.

1.
W15
Test
Automation
5/4/16
15:00
Fostering
Long-­‐Term
Test
Automation
Success
Presented
by:
Carl
Nagle
SAS
Institute,
Inc.
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.stareast.techwell.com/

2.
Carl
Nagle
SAS
Institute,
Inc.
A
principal
software
developer
at
SAS
Institute,
Carl
Nagle
has
focused
for
the
past
nineteen
years
on
the
development,
integration,
and
evolution
of
enterprise
software
test
automation
tools
via
the
open
source
SAFSDEV
project.
Â
For
more
than
thirty
years,
Carl
has
been
responsible
for
the
design
and
implementation
of
both
hardware
and
software
testing
systems.
Â
Carl
and
his
team
thrive
on
solving
“impossible”
software
test
automation
scenarios
across
multiple
processes,
platforms,
and
technologies.

3. Fostering Long-term
Test Automation Success
“Sustainable” Test Automation
with Carl Nagle,
SAS Institute

4. • Why are YOU here?
• Why am I here? (Has it really been 10 years?)
• How did we achieve initial success?
• Where are we now?
• Q & A
In A Nutshell

5. If these apply to YOU:
• NEED or WANT easy test automation NOW,
• For MY current team or project,
• Have NO concern about others, or even future projects,
• Hope to “fix” a failing automation strategy…
Then, you might be looking at the wrong presentation!
( But stay! We might be able to change your mind ;-)
Why Are YOU Here?

6. If these apply to YOU:
• NEED test automation processes that last
• To be shared and leveraged across projects
• That don’t change with technology or tools
• That might “replace” a “struggling” automation strategy…
Then, you might be looking at the right presentation!
Why Are YOU Here?

7. Over 19 years ago, SAS Institute tasked me to:
• Provide a common test automation framework--eliminate silos
• Make test automation accessible to developers *and* testers
• Handle all product and ever-changing technology domains
• Sustain test automation success over years, not cycles
• Develop it in an open-source environment
Today I report that this prospers still--even after 19 years!
Why am I Here?
1997 2002 2004 2006 Today

8. 1. We listened to the industry experts
“We must minimize the impact caused by changes in the applications we are
testing, *and* changes in the tools we use to test them.”
2. We listened to potential users
“We need to leverage and share our test expertise across current and future
projects regardless of the technologies or tools in use.”
3. We collaborated with professional peers
“Focus the test automation framework to deal with the common components
that make up all our unique applications.”
4. We implemented it, shared it, and it continues to evolve.
How did we achieve initial success?

9. The headlines read:
“In today’s environment of plummeting cycle times, test automation becomes
an increasingly critical and strategic necessity.
Assuming the level of testing in the past was sufficient (which is rarely the
case), how do we possibly keep up with this new explosive pace of web-
enabled deployment while retaining satisfactory test coverage and reducing
risk?
The answer is either more people for manual testing, or a greater level of test
automation. After all, a reduction in project cycle times generally correlates to
a reduction of time for test.”

10. The headlines read:
“With the onset and demand for rapidly developed and deployed web clients,
test automation is even more crucial.
Add to this the cold, hard reality that we are often facing more than one
active project at a time.
Perhaps the team is finishing up Version 1.0, adding missing features to
Version 1.1, and prototyping some new technologies for Version 2.0!”

11. IBM Rational Robot, Functional Tester
HP WinRunner, QuickTest Pro
AutoIt
“We must minimize the impact caused by changes in the applications we
are testing, *and* changes in the tools we use to test them.”
DIFFERENT Technologies
HTML 3.2, 4.0, 5, SAP, DOJO, GWT
Flex
Java
iOS, Android
VB, .NET, WPF, Office
DIFFERENT Testing Tools
SmartBear TestComplete, TestExecute
EggPlant, Sikuli, Image-Based Testing
Mobile SDKs, Robotium
Selenium 1.0, 2.0 WebDriver
Require
C/C++

12. • Separate the test design process and assets:
• Completely independent of the test execution “engines”
• Data-driven, action-based, keyword-driven, behavior-driven
• Support test-first principles where desirable
“We must minimize the impact caused by changes in the applications we
are testing, *and* changes in the tools we use to test them.”
Pseudo-Typical Example Test Records:
Google SignIn Click
Explorer FolderTree Click "Coords=50,200“ (tsk, tsk, tsk)
Explorer FolderTree Click My Documents (better)

13. • Separate the test design process and assets:
• Keep volatile test data in Maps for single-point of maintenance
“We must minimize the impact caused by changes in the applications we
are testing, *and* changes in the tools we use to test them.”
[Google]
SignIn=“id=gb_70”
[Explorer]
FolderTree=“Type=Treeview;Index=1”
[FolderTree]
My Documents=“Coords=50,200”

14. [Google]
SignIn=“id=gb_70”
[Explorer]
FolderTree=“Type=Treeview;Index=1”
[FolderTree]
My Documents=“Coords=50,200”
The Map:
The Test:
Keep volatile test data in
Maps for single-point of
maintenance.
Google SignIn Click
Explorer FolderTree Click My Documents

15. • Separate the test design process and assets:
• Common API despite years of evolving technologies and tools
“We must minimize the impact caused by changes in the applications we
are testing, *and* changes in the tools we use to test them.”
• Doesn’t matter what the runtime execution engine is:
SQA Robot, IBM Rational Robot, XDE Tester, Functional Tester
HP WinRunner, QuickTest Pro, SmartBear Test Complete, AutoIt,
EggPlant, Sikuli, IBT, Selenium, Mobile SDKs, Robotium …
Google SignIn Click
Explorer FolderTree Click My Documents

16. “We must minimize the impact caused by changes in the applications we
are testing, *and* changes in the tools we use to test them.”
http://safsdev.sourceforge.net/sqabasic2000/SAFSReference.php

17. • Separate the test design process and assets:
• Test IDE can be different, depending on skills and preferences
• CSV Text Files, Spreadsheets, Test Composer (CSV Exporters)
• Java IDE, Eclipse, SeleniumPlus (Java API Drivers)
• Custom IDE, or programs (COM, Ruby, Perl…) invoking the API
“We must minimize the impact caused by changes in the applications we
are testing, *and* changes in the tools we use to test them.”
Java API Examples:
Click(Map.Google.SignIn);
Click(Map.Explorer.FolderTree, “My Documents”);

18. Click(Map.Google.SignIn);
Click(Map.Explorer.FolderTree, “My Documents”);
Java API Test:
CSV Test:
The Map:
[Google]
SignIn=“id=gb_70”
[Explorer]
FolderTree=“Type=Treeview;Index=1”
[FolderTree]
My Documents=“Coords=50,200”
Then do safs action Click on SignIn in Google
Then do safs action Click on FolderTree in Explorer using “My Documents”
Cucumber Test:
Google SignIn Click
Explorer FolderTree Click My Documents

19. CallScript(Map.SeBuilderTest());
StopWebBrowser(Map.PageID());
Java API Test:
The Map:
[ApplicationConstants]
GoogleURL=“http://google.com”
PageID=“MainPage”
[Google]
SignIn=“id=gb_89”
Selenium Html Fit Test:
SeBuilder Test:
..{
“type”: “get”,
“url” : “${GoogleURL}”,
“id” : “${PageID}”
},…
Google Test Table
open ${GoogleURL} ${PageID}
click ${Map:Google:SignIn}

20. Application Map(s)
(Volatile SPOM)
SAFS Services and Engine(s)
Java, Web, Flex, Android
Application Under Test
Java, Web, Flex, Android
Logs
Functional Tester
TestComplete
SeleniumPlus
Robotium
Image-Based Testing
Executable
Test Designs
(CSV, Cucumber,
Java, SeleniumPlus)
What this looks like in a complex graph?

21. What are those internal Services?
Logging
Maps
Variables
Drivers
Engines
(But you really don’t have to care about this.)

22. Where are we now?
• We have grown to 4 full-time in-house framework developers.
• Internally, our user-base grows in the US, Canada, India, and China.
• We just completed development migration to SAFSDEV on GitHub.
• New discussion forums at safsdev.freeforums.net.
• Older forum archives at sourceforge.net/p/safsdev/mailman/
• We have doc and updated binaries: SAFSDEV on SourceForge.
• We have always welcomed contributors for new engines or features.
• Frankly, we have been VERY BUSY since our last public talk in 2006.

23. Fostering Long-term
Test Automation Success
“Sustainable” Test Automation
with Carl Nagle,
SAS Institute
Q & A