test automation strategy
Developing a
why are we here?
regression testing
the greatest cost of development
regression testing
you have already paid for this at least once
why regression test?
why regression test?
quality ensures what was working, still works
why regression test?
quality
risk
money
lost revenue, market, contractual
ensures what was working, still works
deliverables, release, schedule
why regression test?
a project without
thorough and continuous regression testing
is like a road trip
where you do not look out the window
until the end of the trip
why automate? (vs. manual)
why automate? (vs. manual)
TimeMoney
why automate? (vs. manual)
Money
why automate? (vs. manual)
Time
automation strategy
what to consider...
tests are living specifications of the system
automation strategy
what to consider...
ARE THE TESTS GOING TO BE USED?
automation strategy
what to consider...
business needs
return on investment
organizational structure and culture
automation strategy
business needs...
determine areas of focus of the automation
automation strategy
return on investment...
limits and prioritizes the areas of focus of the
automation
determines at which level to automate a test
determines the automation framework
determines the success or failure
determines the speed, ability & openness to change
determines the communication
determines the ability to learn
automation strategy
organization and culture...
automation strategy
organization and culture...
determines who uses and maintains the tests
determines the automation framework too
automation strategy
what else to consider...
automation strategy
what else to consider...
development process and lifecycle
maintenance of the tests
technology stacks
automation strategy
development process and lifecycle...
legacy (system without tests) to greenfields
automation strategy
development process and lifecycle...
system under test development process
methodology: waterfall agile xp ad hoc
is automation development part of the same process?
automation strategy
development process and lifecycle...
is the automator part of the development team?
how much access to the system developers?
automation strategy
maintenance of the tests...
who and how much can they maintain?
how are the system developers involved?
automation strategy
maintenance of the tests...
how timely can/must tests be maintained
what reporting needed to maintain tests
automation strategy
finally, technology stacks...
languages and technology already used
unit and integration testing
source code control, continuous integration
devops or operations
where to automate?
who is going to use the tests?
developers, maybe...
depends
where to automate?
https://msdn.microsoft.com
“Its essential point is that you should
have many more low-level unit tests than
high level end-to-end tests running
through a GUI.” Martin Fowler
where to automate?
UNIT LEVEL
INTEGRATION LEVEL
GUI INTEGRATION LEVEL
XPtest driven development
ACCEPTANCE LEVEL
BDD
ACCEPTANCE LEVEL
BDD
where to automate?
acceptance tests
only if the business is engaged and
going to use them
ACCEPTANCE LEVEL
BDD
where to automate?
what to consider...
DRY
look for low-hanging fruit
risk-based testing
what computers do well, humans not so much
fragility of test
where to automate?
what to consider...
which automation framework to use?
which automation framework to use?
Gherkin/Cucumber
Robot Framework
Selenium Webdriver
Spock/Geb
JUnit
Jasmine
TestNG
which automation framework to use?
what to consider...
ORGANIZATIONAL STRUCTURE AND CULTURE
which automation framework to use?
what to consider...
who is reading the tests?
who is writing and maintaining the tests?
balance of right tool with existing technology,
knowledge, and ability to learn
ORGANIZATIONAL STRUCTURE AND CULTURE
which automation framework to use?
what else to consider...
natural language for high level specification
goal is to develop tests NOT framework
and...
grids...
integration...
reporting...
teh sexy
gherkin/cucumber/ruby
but JRuby (JVM), Java, Groovy, JavaScript,
.NET (using SpecFlow)
libraries (Selenium, Watir)
Ruby has the richest ecosystem with gems
gherkin/cucumber
generally ruby based...
Feature: Cash Withdrawal
Scenario: Successful withdrawal from an account in credit
Given my account has been credited with $100
When I withdraw $20
Then $20 should be dispensed
And the balance of my account should be $80
Given /^my account has been credited with (#{CAPTURE_CASH_AMOUNT})$/
do |amount|
my_account.credit(amount)
end
also runs on Jython (JVM) & IronPython (.NET)
libraries (Selenium, Watir, SOAP, etc.)
keyword based
supports “given when then”
supports tabular/data driven
robot framework
python based...
*** Settings ***
Library Selenium Library
Suite Setup Start Selenium Server
Suite Teardown Stop Selenium Server
Test Setup Setup Test
Test Teardown Close All Browsers
*** Testcases ***
Login Should Succeed When the Correct
Username and Password are Entered
Enter Username AUser
Enter Password TestPass
Click the Login Button
Login is Successful
*** Keywords ***
Setup Test
Open Browser http://zdiles.chaosnet.org/ googlechrome
Maximize Browser Window
Enter Username [Arguments] ${username}
Input Text uname ${username}
Login Is Successful
Page Should Contain Welcome
*** Testcases ***
Login Should Succeed When the Correct Username and
Password are Entered
Given I Enter Username AUser
And I Enter Password TestPass
When I Click the Login Button
Then Login is Successful
given when then
spock/geb/groovy
developing in java or grails?
groovy-based browser automation tool
built on Selenium WebDriver
works with JUnit, TestNG, Spock
page models and jQuery selector syntax
spock/geb/groovy
geb...
groovy-based test and specification framework
supports “given when then”
tabular/data driven
spock/geb/groovy
spock...
import geb.spock.GebSpec
class GebHomepageSpec extends GebSpec {
def "can access The Book of Geb via homepage"() {
given:”One is on the Geb Home Page”
to GebHomePage
when: “I click the jQuery Link”
highlights.jQueryLikeApi.click()
then: “I should be at the jQuery Section”
highlights.jQueryLikeApi.isSelected
}
}
import geb.Page
class GebHomePage extends Page {
static url = "http://gebish.org"
static at = { title == "Geb - Very Groovy Browser Automation" }
static content = {
highlights { $("#sidebar .sidemenu").module(HighlightsModule) }
sectionTitles { $("#main h1")*.text() }
}
}
FRAMEWORKS
other options...
raw Selenium (Java)
excel spreadsheet data driven scripts
script capture record/playback
flat files in source control
other options...
which automation framework to use?
the simplest thing that could possibly work
tests
are specifications of the system...
how does it fit?
how does it fit...
design your approach into the organization
how does it fit...
find the gaps and tight squeezes
the specifics...
the mechanics...
so what’s left?
documentation of the strategy...
execution of the strategy...
if project management...
estimation...
resources...
Developing a test automation strategy by Brian Bayer

Developing a test automation strategy by Brian Bayer