SlideShare a Scribd company logo
1 of 63
You have Selenium... Now what?
Hans Buwalda
©2016 LogiGear Corporation
www.logigear.com
www.testarchitect.com
www.happytester.com
@hansbuwalda
hans@logigear.com
hans@happytester.com
©2016 LogiGear Corporation
• Topics:
– domain language approaches
– test design as driver for automation
– automation friendly test design techniques and examples
– special cases (data driven, multi-station, graphics, etc)
– drone perspective: overview with some selective details
• What this talk is not:
– an interactive workshop
– specific to Selenium only
– a technical class on automation
– an interactive workshop
– about component based development
– introduction to unit testing
Scope
Warning: there is an overlap with my talk yesterday!!
©2016 LogiGear Corporation
• A combination of open source products and standards, aimed at
web testing (including a Firefox based IDE)
• In the later versions it works with an interface called “WebDriver”,
implemented with drivers in browsers
• WebDriver has been implemented in open source libraries, but is
now also becoming implemented in browsers
• For example Microsoft Edge can only be accessed programmatically
via WebDriver
• Products like Appium try to implement WebDriver for mobile
browsing and even mobile apps
• Our product (TestArchitect) is moving to WebDriver, expect this to be
a common practice
Selenium
©2016 LogiGear Corporation
Relation to code Quality / depth Automation Scalability
Unit Testing
Close relationship
with the code
Singular test
scope, but deep
into the code
Fully automated
by nature
Scalable, grows
with the code,
easy to repeat
Functional/
Integration
Testing
Usually does not
have a one-on-one
relation with code
Quality and scope
depends on test
design
In particular UI
based automation
can be a challenge
Often a bottle-
neck in scalability
Exploratory
Testing
Human driven, not
seeking a relation
with code
Usually deep and
thorough, good at
finding problems
May or may not
be automated
afterwards
Not meant to be
repeatable. Rather
do a new session
Some test kinds and their scalability (simplified)
©2016 LogiGear Corporation
Test Pyramid
• Proposed by Mike Cohn (in his book: "Succeeding with Agile").
• My version would be less "pointy"
• In particular I like to separate technical and design considerations
• Also, I've seen API's that, for higher level operations, are quite complex. Trying to use
them can become like re-implementing the UI again
• Web UI's are getting richer and more complex to test (SPA's, AngularJS, etc)
UI
Service or
Component
Unit
UI
Unit
original my version (for now)
Service or
Component
©2016 LogiGear Corporation
Issues are not always obvious...
Downton Abbey
©2016 LogiGear Corporation
Brief test design exercise
• View this imaginary application
• It adds two numbers
• What tests would you create?
• Which tests would you automate?
©2016 LogiGear Corporation
The 5% Rules of Test Automation
• No more than 5% of all test cases should be
executed manually
• No more than 5% of all efforts around
testing should involve automating the tests
©2016 LogiGear Corporation
Don't just automate manual testing
©2016 LogiGear Corporation
Don't just automate manual testing
©2016 LogiGear Corporation
Don't just automate manual testing
Good automated testing is not the same as automating good manual testing. . .
©2016 LogiGear Corporation
Example Scripting, Java + Selenium
WebElement element = driver.findElement(By.name(name));
element.sendKeys("mystery magic");
element.submit();
(new WebDriverWait(driver, 10)).until(
new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
return d.getTitle()
.toLowerCase().startsWith("mystery");
}
}
);
System.out.println(driver.getTitle());
©2016 LogiGear Corporation
• Frameworks, libraries, patterns
– libraries: sets of functions, you call them
– frameworks: also sets of functions, but the framework is typically in
charge and calls you
– patterns: standardized ways to do certain tasks
• Selenium is often used with frameworks
– to execute tests and collect results (like unit testing frameworks)
– to offer higher level API's
– to support BDD, data driven testing or keywords
• A common practice is the Page Object pattern
– shield from HTML
– provide an application specific API, with functions like "getSongTitle"
• Examples of keyword frameworks: Robot Framework,
and our own TestArchitect
Libraries, frameworks, patterns
Framework
©2016 LogiGear Corporation
Domain language approaches: Actions
4 actions, each with
an action keyword
and arguments
read from top
to bottom
fragment from a test with actions
acc nr first last
open account 123123 John Doe
acc nr amount
deposit 123123 10.11
deposit 123123 20.22
acc nr expected
check balance 123123 30.33
• Stays out of the programming language
• The test developer creates tests using actions with keywords and
arguments
• Checks are, as much as possible, explicit (specified expected values)
• The automation task focuses on automating the keywords, each
keyword is automated only once
©2016 LogiGear Corporation
• In "low level" tests interaction details are not hidden, since they are
the target of the test
• The right level of abstraction depends on the scope of the test, and is
an outcome of your test design process
TEST MODULE Screen Flow
user
start system john
TEST CASE TC 01 Order button
window button
click main create order
window
check window exists new order
FINAL
close application
Example of a low level test
©2016 LogiGear Corporation
Variables and expressions with keywords
• This test does not need an absolute number for the
available cars, just wants to see if a stock is updated
• As a convention we denote an assignment with ">>"
• The "#" indicates an expression
TEST CASE TC 02 Rent some more cars
car available
get quantity Chevy Volt >> volt
first name last name car
rent car John Green Chevy Volt
rent car Jane White Chevy Volt
car expected
check quantity Chevy Volt # volt - 2
©2016 LogiGear Corporation
Data driven testing with keywords
• The test lines will be repeated for each row in the data set
• The values represented by "car", "first" and "last" come
from the selected row of the data set
TEST CASE TC 03 Check stocks
data set
use data set /cars
car available
get quantity # car >> quantity
first name last name car
rent car # first # last # car
car expected
check quantity # car # quantity - 1
repeat for data set
DATA SET cars
first last car
John Doe Chevy Volt
Mary Kane Ford Escape
Jane Collins Chrysler 300
Tom Anderson Buick Verano
Henry Smyth BMW 750
Vivian Major Toyota Corolla
©2016 LogiGear Corporation
ACTION DEFINITION check balance
user
argument customer
argument amount
window control value
enter balance inquiry last name # customer
window control
click balance inquiry view balance
window control expected
check balance inquiry balance # amount
Re-use actions to make new actions
• In the below example we make a new action
• Existing actions are strung together to create new ones with a
broader scope
• Often steps in low level tests are re-used to create these action
definitions
:
customer amount
check balance Smith 223.45
check balance Jones 0.00
check balance James -330.45
:
use many times in tests:
define in one place:
©2016 LogiGear Corporation
Example action implementation in Python
This action script (in Python, Java or C#) verifies whether the rows in a table are sorted:
# get table object, column number and column count
windowName = LIBRARY.NamedArgument("window")
tableName = LIBRARY.NamedArgument("table")
columnName = LIBRARY.NamedArgument("column")
table = ABT.OpenElement(windowName, tableName)
column = table.GetColumnIndex(columnName)
rowCount = table.GetRowCount()
# check the sort order, row by row
previous = table.GetCellText(0, column)
for i in range(1, rowCount):
current = table.GetCellText(i, column)
if current < previous :
LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)
return
previous = current
LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)
find the table in the UI
if a value is smaller than before, fail the test
if all rows are ascending, pass the test
get arguments from the test line
def action_checkSortOrder():
Note that the approach resembles the Page Object pattern, but in my view is not the
same. The abstraction is “any table”, even in non-Web UI’s or non-UI’s altogether. The
Page Object pattern could also work on top of this.
©2016 LogiGear Corporation
Using the new action
• By keeping an action generic it can be applied for a
variety of situations
• Some examples of using "check sort order":
window table column
check sort order view orders orders table ID
window table column
check sort order annual results regions revenue
window table column
check sort order inventory cars price
window table column
check sort order registration students last name
©2016 LogiGear Corporation
Behavior Driven Development (BDD)
• Uses natural language scenarios
• Tools like JBehave and Cucumber map these to code
• Structure is "Given-When-Then" (GWT)
• Example:
Given a customer previously bought a black sweater from me
And I currently have three black sweaters left in stock
When he returns the sweater for a refund
Then I should have four black sweaters in stock
(source: Wikipedia)
©2016 LogiGear Corporation
BDD with keywords (example)
GIVEN A customer previously bought a black sweater from me
first last id
add client John Jones >> client
AND I currently have three black sweaters left in stock
article color quantity
get stock sweater black >> sweaters
WHEN He returns the sweater for a refund
customer article color
return article # client sweater black
THEN I should have four black sweaters in stock
article color quantity
check stock sweaters black # sweaters + 1
©2016 LogiGear Corporation
Equivalence, conversion
Given a customer previously bought a black sweater from me
And I currently have three black sweaters left in stock
When he returns the sweater for a refund
Then I should have four black sweaters in stock
customer buys, article, color
a customer previously bought a {color} {article} from me
set stock, article, color, amount
I currently have {amount} {color} {article} left in stock
my stock of {color} {sweater} is {amount}
return article, article, color
he returns the {color} {article} for a refund
check stock, article, color
I should have four {color} {article} in stock
article color
customer buys sweater black
article color amount
set stock sweater black 3
article color
return article sweater black
article color amount
check stock sweater black 4
Test Module Sections
Test Module
Objectives
Initial - setup
Test cases
Final - cleanup
High Level Test Design - Test Development Plan
Objectives
Test Module 1
Test Cases
Test Module 2 Test Module N
Actions
. . .
AUTOMATION
Objectives Objectives
interaction test business test
define the "chapters"
create the "chapters"
create the "words"
make the words work
Test Cases Test Cases
window control value
enter log in user name jdoe
enter log in password car guy
window control property expected
check property log in ok button enabled true
user password
log in jdoe car guy
first last brand model
enter rental Mary Renter Ford Escape
last total
check bill Renter 140.42
Action Based Testing
©2016 LogiGear Corporation
Organize the tests
• What tests would you create for this application?
• How would you organize them?
• What, if anything, would you avoid?
©2016 LogiGear Corporation
Why Better Test Design?
• Better test design can improve quality of test
– many tests are often quite "mechanical" now, no surprises
– one to one related to specifications, user stories or requirements,
which often is ok, but lacks aggression
– no combinations, no unexpected situations, lame and boring
– such tests have a hard time finding (interesting) bugs
• Better test design can give (much) better automation
– when unneeded details are left out of tests, those details don't have
to be maintained
– avoiding "over checking": creating checks that are not in the scope
of a test, but may fail after system changes
– limit the impact of system changes on tests, making such impact
more manageable
I have become to believe that successful automation is usually less of a
technical challenge as it is a test design challenge.
©2016 LogiGear Corporation
What's the trick...
©2016 LogiGear Corporation
• Business objects and business flows
– objects are like cars, invoices, locations, etc
– flows are like create, fulfill, pay and close an order
• Other tests
– functions and features, like premium calculation or PDF output
– administration, users, security, authorizations
– graphics
– technologies, protocols, ...
– customization, extensibility
– . . .
• Business versus interaction
– differentiate within business objects and other categories
– interaction can be further differentiated into: values, UI, keyboard,
etc
Example of considerations
©2016 LogiGear Corporation
Example Top Level Structure
Project create, update, delete/close
copy, move
categorize, enumerate, identify
convert, serialize, export/import, ...
UI, dialogs, forms, pages
input (validation, defaulting, dependencies)
flows (primary paths, alternate paths)
keyboard shortcuts, keyboard controls, ...
. . .
<business object 1>
Lifecycles, data operations
Interaction
Functions and Features
Technologies, protocols, controls
Data (handling, quality, ETL, ...)
Security, authorization, admin
Graphics, multi-media, charts, ...
Load, performance
Integration
Customizing, extensibility
Business Flows
Concurrency, race conditions, ...
Business Objects
processes, transactions, end-to-end, day in the life,
combinations of flows, ...
calculations, analyses, PDF output, ...
Repeat for each
component or sub-system
©2016 LogiGear Corporation
Example: E-commerce site
• Articles
• Categories
• Customers
• Promotions
• Orders
• Invoices
• Payments
• Credit cards
• Staff members
• Countries
• . . .
©2016 LogiGear Corporation
• Promotion kinds
– percentage discounts
– fixed dollar discounts
– comprehensive promotions
– regional promotions
– volume discounts
– prompt payment discounts
– article based, comprehensive
– . . .
• Life cycles
– creation
– modification
• various properties
• before or after activation
– expiration
– cancellation
Promotions -- business tests
©2016 LogiGear Corporation
Promotions -- interaction tests
• Screen by screen
• Discount percentage/fixed
– does the amount field change
• Regional checkbox
– does the region list show up
– are the regions correct
– can the regions be selected/deselected
• Start, duration, end date
– are the date checks working
– can either duration or end date be entered
• Articles, categories
– can categories be selected, do they show up
– do the right articles show up, can they be selected
• . . .
©2016 LogiGear Corporation
What tests could look like
Business tests for business object "Promotions"
Interaction tests for business object "Promotions"
window control
click main new promotion
window type
select promotion town
window list value
check list item exists promotion towns Tietjerksteradeel
name start finish percentage category
create promotion christmas 12/1/2016 12/25/2016 11 all
create promotion tablets 12/20/2016 1/1/2017 20 tablets
date
time travel 12/23/2016
article price
check nett price iPad Mini 4 284.79
©2016 LogiGear Corporation
Example 2: a testing company in
California
• The company sells services like test development and
automation
• It also has consultancy and training offerings
• And it has a product for test development and
automation
• These various offerings are often combined in the orders
they receive
• Invoicing for services as monthly
• They're working on an internal system for order handling
and invoicing
©2016 LogiGear Corporation
Example 2: a testing company in
California
• Example business objects:
– projects, orders ("SOW's"), customers, employees, invoices,
payments
• Example business flows:
– customer place orders, often multiple ones over time
– one order can have multiple items
– tools are invoiced (and delivered) immediately
– services are delivered over time
– staff members write time-sheets with hours spent
• has elaborate UI's and processes
– invoices can combine items from multiple orders
– payments can cover multiple invoices
©2016 LogiGear Corporation
What about existing tests?
• Compare to moving house:
– some effort can't be avoided
– be selective, edit your stuff,
• look at the future, not the past
– first decide where to put what, then put it there
– moving is an opportunity, you may not get such chance again soon
• Follow the module approach
– define the modules and their scope as if from scratch
– use the existing test cases in two ways:
• verify completeness
• harvest and re-use them for tests and for actions
– avoid porting over "step by step", in particular avoid over-checking
©2016 LogiGear Corporation
Eye on the ball, Scope
• Always know the scope of the test module
• The scope should be unambiguous
• The scope determines many things:
– what the test objectives are
– which test cases to expect
– what level of actions to use
– what the checks are about and which events should generate a
warning or error (if a “lower” functionality is wrong)
©2016 LogiGear Corporation
Use the right level actions
Low level of UI interaction detail makes sense only with the module
scope is to test the UI
window tree tree item path
click tree item main projects /Projects/Drill Assembly
window list item
check list item exists main tasks Plan of Approach
Better to have a business level action and hide the details in an action
definition:
But do show details when it matters. The example below is too high level,
requires drill down into action definition to understand what is tested.
©2016 LogiGear Corporation
Example of a Test
Step Description Expected
step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link
step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled
step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled
step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled
step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner
step 21 Enter "acme watch" in the search field The "Search" button is enabled
step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed
step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed
step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2
step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2
step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping"
step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added
step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches
is this a good test? is it good for automation?
©2016 LogiGear Corporation
GWT scenario
Given User turns off Password required option for Drive Test
And User has logged in by Traffic Applicant account
And User is at the Assessments Take a Test page
And User clicks the Traffic Test link
And User clicks the Next button
And User clicks the Sheet radio button in Mode page if displayed
And User clicks the Start button
And User waits for test start
And User clicks the Stop Test button
When User clicks the Confirm Stop Test button
And User enters the correct applicant password
And User clicks the Confirm Stop Test button
Then The Test is Over should be displayed in the Message label
And the value of the Message label should be The test is over
And The Welcome to Traffic Testing page should be displayed
©2016 LogiGear Corporation
"Anti-patterns" (informal)
• Interaction Heavy – Not having many business tests
• Lame – No depth or variety, no testing techniques used
• Enter Enter Click Click – Test steps are too detailed
• No Life – Missing life cycle steps of business objects
• Clueless – No clear scope for the tests
• Cocktail – Interaction tests mixed with business tests
• Over-Checking – Checks not relevant for the scope
• Sneaky Checking – Checks hidden in actions
• Action Explosion – Many actions, hardly different
• Mystery Actions – Cryptic actions, unclear what they do
• Techno – Actions and tests that look like code
– often _NOts0EasY_2REad
– but great to impress non-technical people
©2016 LogiGear Corporation
Example of a test module
TEST MODULE Order processing
start system
TEST CASE TC 01 Order for tablets
user password
login jdoe doedoe
window
check window exists welcome
order id cust id article price quantity
create order AB123 W3454X tablet 198.95 5
order id total
check order total AB123 994.75
. . .
©2016 LogiGear Corporation
Low-level, high-level, mid-level actions
• Low-level (system level) actions
– detailed interaction with the UI (or API)
– generic, do not show any functional or business logic
– examples: "click", "expand tree node", "select menu"
• High-level (application and business level) actions
– represent a business function or event fitting the scope of the test
– hides the interaction
– examples: "enter customer", "rent car", "check balance"
• Mid-level actions: auxiliary actions that represent common
sequences of low level actions
– usually to wrap a page or dialog
– enhance maintainability
– example: "enter address fields“
– complex pages could be further split up
like with the Page Object pattern
enter customer
enter address fields
enter select set . . .. . .
Tip: Provide default values in actions
ACTION DEFINITION login
name default value
argument user tester
argument password testerpw
window control value
enter login window user name # user
enter login window password # password
window control
click login window login
user password
login tamaraj tj1234
text
check message Hello Tamara
login
date payee amount
make payment 1/1/12 Gas Co. 85.00
Use login action w/ arguments Use login default values
©2016 LogiGear Corporation
ABT in Agile
Test Module
Definition
(optional)
Test Module Development
Interface Definition
Action Automation
Test Execution
Sprint Products
Product
Backlog
Test re-use
Automation re-use
product
owner
team
prod owner
& team
User stories
Documentation
Domain understanding
Acceptance Criteria
PO Questions
Situations
Relations
Agile life cycle
Test development
Main Level Test Modules
Interaction Test Modules
Cross over Test Modules
©2016 LogiGear Corporation
Using ABT in Sprints (1)
• Try to keep the testers in the same sprint as the rest of
the team
– don't let automated tests clutter up and lack behind
– note that in many environments tests and their automation are not
highest priority
• Agree on the approach:
– is testability a requirement for the software?
– do we regard tests, or some of the tests, as products?
• would potentially become part of the backlog
• would not only be up to the team which tests to create
• Just like for development, use discussions with the team
and product owners
– deepen understanding, for the whole team
– help identify items like negative, alternate and unexpected
situations
– tests can work as driver for development (TDD and ATDD)
©2016 LogiGear Corporation
Using ABT in Sprints (2)
• Create good starting conditions for a sprint:
– automation technology available (UI, non-UI, custom controls, graphics, ...)
– know how you will deal with data and environments
– understanding of subject matter, testing, automation, etc
• Do interface mapping by hand, using developer provided
identifications
– saves time by not having to use the viewer or other spy tools
– recording of actions (not tests) will go better
To discuss an approach, consider daily "sit down" meetings with some or
all members to coach and evaluate
 an end-of-day counterpart to the early-morning "stand up" meetings
 short and friendly, not about progress and impediments, but about practices and
experiences with them (like "what actions did you use?")
 a few meetings may suffice
Multiple System Access
System
Under Test
Action Automation
Web service
access
command
line access
Web UI
access
database
access
Test Modules
©2016 LogiGear Corporation
Testing in, or near, production
• In an agile and DevOps approach continuous deployment is
becoming the norm
– in particular for service based system (like web, apps, SaaS, but also client-server)
– logic in services is easier to manage and updated than distributed and variable clients
• In a DevOps approach tight cooperation between development, test
and deployment pays off
– automation testability
– testable system designs
– good test design remains a success factor
– tight integration in the build, deployment and production processes
• A/B testing can help test vary complex systems
– used not only for testing, also for trying out new features
– dividing the incoming traffic into an A and B flow (B is the new situation)
– automated tests can use the B flow
– to do this, have it well integrated in your system designs
• Continuous testing with random regression testing or monkey testing
*see also: Ken Johnston's chapter in the book of Dorothy Graham and Mark Fewster, and his keynote at StarWest 2012
©2016 LogiGear Corporation
A/B testing with a reverse proxy
• A/B testing means part of traffic is routed through a different server
or component (see if it works, and/or how users react)
• B could be a real-life user, a test user or also an automated test
• The strategy can be done at any component level
• Watch your test design, easy to drown in technical only
A
A
B
Reverse
Proxy
Users
Servers
A
B
newcurrent
A
B
©2016 LogiGear Corporation
• Passive timing
– wait a set amount of time
– in large scale testing, try to avoid passive timing altogether:
• if wait too short, test will be interrupted
• if wait too long, time is wasted
• Active timing
– wait for a measurable event
– usually the wait is up to a, generous, maximum time
– common example: wait for a window or control to appear (usually the test tool will do this for
you)
• Even if not obvious, find something to wait for...
• Involve developers if needed
– relatively easy in an agile team, but also in traditional projects, give this priority
• If using a waiting loop
– make sure to use a "sleep" function in each cycle that frees up the processor (giving the AUT
time to respond)
– wait for an end time, rather then a set amount of cycles
Active Timing
©2016 LogiGear Corporation
Things to wait for...
• Wait for a last control or elements to load
– developers can help knowing which one that is
• Non-UI criteria
– API function
– existence of a file
• Criteria added in development specifically for this purpose, like:
– "disabling" big slow controls (like lists or trees) until they're done loading
– API functions or UI window or control properties
• Use a "delta" approach:
– every wait cycle, test if there was a change; if no change, assume that the
loading time is over:
– examples of changes:
• the controls on a window
• count of items in a list
• size a file (like a log file)
©2016 LogiGear Corporation
• Should be a "must have" requirement
– first question in a development project: "how do we test this?"
• Design of the system(s) under test:
– components, tiers, services,
• Hidden identifying properties
• Hooks for timing
• White-box access to anything relevant:
– input data (ability to emulate)
– output data (what is underlying data being displayed)
– random generators (can I set a seed?)
– states (like in a game)
– objects displayed (like monsters in a game)
• Emulation features, like time-travel and fake locations
Testability, key items
©2016 LogiGear Corporation
Versions, environments, configurations
• Many factors can influence details of automation
– language, localization
– hardware
– version of the system under test
– system components, like OS or browser
• Test design can reflect these
– certain test modules are more general
– others are specific, for example for a language
• But for tests that do not care about the differences, the
automation just needs to "deal" with them
– shield them from the tests
minimum safe distance
from a bear is 91 meters
LOCALIZATION
converting yards to meters
Capture variations of the system under test in the actions and interface
definitions, rather than in the tests (unless relevant there).
Variation Variation Variation
"Variations"
"Master Switch"
Actions, Interface Definitions
. . .
Possible set up of variations
linked variation
keyworded variation
Specify for example in a dialog when you start an execution:
©2016 LogiGear Corporation
• Not every test is straightforward UI based
• Always go back to the test: what is it I want
• Express that with actions
– make them up as you go
– assume everything is possible
• For the automation side, ask for advise
– you pattern might have been seen before
Special situations
©2016 LogiGear Corporation
"Lead Deputy" Testing
• For "multi station" testing, when multiple machines have to
participate in a test in real-time
• For example if a supervisor needs to approve a withdrawal in a bank
teller system
• Can be "synchronized" and "parallel" (with a rendezvous point)
lead machine
playing the
"bank teller"
deputy machine
playing the
"supervisor"
acc nr amount tx id
request 123123 10,000 >> tx
name
use deputy supervisor
tx id amount
approve # tx 10,000
use lead
tx id amount
pay out # tx 10,000
lead machine
playing the
"bank teller"
...
...
Device Testing
Software Under Test
Agent
ABT
Automation
Interface
Info
Testing Host
Device
Androi
d
©2016 LogiGear Corporation
• Approach applicable for pictures like graphics or icons
• The tester will add a line "check picture", that includes
a "question" as one of the arguments
• While the test is executed TA keeps the recorded
pictures
• After execution the pictures are shown to a manual
testing for approval
• Once approved unchanged same pictures won't be
presented to the manual tester again in future runs
Multimedia: The "check picture"
Approach
©2016 LogiGear Corporation
• Automation is not as much a technical challenge as it is a test
design challenge
• Domain language based approaches like BDD and Actions can
make tests are accessible, and easier to maintain
• However, some kind of systematic approach, like the ABT method, is
necessary to keep the tests manageable
• Testing is teamwork, including automation engineers, developers
and product owners
• From a test design perspective approaches for non-UI and UI driven
testing are very similar
Summary hans@logigear.com
That’s all folks for this talk. However, I have
much more stuff, please let me know any
questions.
©2016 LogiGear Corporation
• Automation is not as much a technical challenge as it is
a test design challenge
• Domain language based approaches like BDD and
Actions can make tests are accessible, and easier to
maintain
• However, some kind of systematic approach, like the
ABT method, is necessary to keep the tests manageable
• From a test design perspective approaches for non-UI
and UI driven testing are very similar
Summary hans@logigear.com
That’s all folks for this talk. However, I have
much more stuff, please let me know any
questions.

More Related Content

What's hot

How Capital One Scaled API Design to Deliver New Products Faster
How Capital One Scaled API Design to Deliver New Products FasterHow Capital One Scaled API Design to Deliver New Products Faster
How Capital One Scaled API Design to Deliver New Products FasterSmartBear
 
O365Engage17 - How to Automate SharePoint Provisioning with PNP Framework
O365Engage17 - How to Automate SharePoint Provisioning with PNP FrameworkO365Engage17 - How to Automate SharePoint Provisioning with PNP Framework
O365Engage17 - How to Automate SharePoint Provisioning with PNP FrameworkNCCOMMS
 
Building a Documentation Portal
Building a Documentation PortalBuilding a Documentation Portal
Building a Documentation Portalstc-siliconvalley
 
Why Embrace "Html5"?
Why Embrace "Html5"?Why Embrace "Html5"?
Why Embrace "Html5"?FossilDesigns
 
Operational API design anti-patterns (Jason Harmon)
Operational API design anti-patterns (Jason Harmon)Operational API design anti-patterns (Jason Harmon)
Operational API design anti-patterns (Jason Harmon)Nordic APIs
 
O365Engage17 - Protecting O365 Data in a Modern World
O365Engage17 - Protecting O365 Data in a Modern WorldO365Engage17 - Protecting O365 Data in a Modern World
O365Engage17 - Protecting O365 Data in a Modern WorldNCCOMMS
 
ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...
ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...
ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...Agile Testing Alliance
 
Good to Great SharePoint Governance
Good to Great SharePoint GovernanceGood to Great SharePoint Governance
Good to Great SharePoint GovernanceNCCOMMS
 
Rise of the Open Source Program Office for LinuxCon 2016
Rise of the Open Source Program Office for LinuxCon 2016Rise of the Open Source Program Office for LinuxCon 2016
Rise of the Open Source Program Office for LinuxCon 2016Gil Yehuda
 
Suguk Southampton CodePlex - March 2014
Suguk Southampton   CodePlex - March 2014Suguk Southampton   CodePlex - March 2014
Suguk Southampton CodePlex - March 2014Steven Andrews
 
Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...
Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...
Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...Sendachi
 
IoT in the Cloud: Build and Unleash the Value in your Renewable Energy System
IoT in the Cloud: Build and Unleash the Value in your Renewable Energy SystemIoT in the Cloud: Build and Unleash the Value in your Renewable Energy System
IoT in the Cloud: Build and Unleash the Value in your Renewable Energy SystemMark Heckler
 
Engage 2018 adm04 - The lazy admin wins
Engage 2018   adm04 - The lazy admin winsEngage 2018   adm04 - The lazy admin wins
Engage 2018 adm04 - The lazy admin winsMatteo Bisi
 
Solving Real World Challenges with Enterprise Search
Solving Real World Challenges with Enterprise SearchSolving Real World Challenges with Enterprise Search
Solving Real World Challenges with Enterprise SearchSPC Adriatics
 
O365Engage17 - What’s New in Office 365 Security
O365Engage17 - What’s New in Office 365 SecurityO365Engage17 - What’s New in Office 365 Security
O365Engage17 - What’s New in Office 365 SecurityNCCOMMS
 
DocOps — The Analytical Window to Your Customer’s Experience with Wade Clements
DocOps — The Analytical Window to Your Customer’s Experience with Wade ClementsDocOps — The Analytical Window to Your Customer’s Experience with Wade Clements
DocOps — The Analytical Window to Your Customer’s Experience with Wade ClementsInformation Development World
 

What's hot (17)

How Capital One Scaled API Design to Deliver New Products Faster
How Capital One Scaled API Design to Deliver New Products FasterHow Capital One Scaled API Design to Deliver New Products Faster
How Capital One Scaled API Design to Deliver New Products Faster
 
O365Engage17 - How to Automate SharePoint Provisioning with PNP Framework
O365Engage17 - How to Automate SharePoint Provisioning with PNP FrameworkO365Engage17 - How to Automate SharePoint Provisioning with PNP Framework
O365Engage17 - How to Automate SharePoint Provisioning with PNP Framework
 
Building a Documentation Portal
Building a Documentation PortalBuilding a Documentation Portal
Building a Documentation Portal
 
Why Embrace "Html5"?
Why Embrace "Html5"?Why Embrace "Html5"?
Why Embrace "Html5"?
 
Operational API design anti-patterns (Jason Harmon)
Operational API design anti-patterns (Jason Harmon)Operational API design anti-patterns (Jason Harmon)
Operational API design anti-patterns (Jason Harmon)
 
O365Engage17 - Protecting O365 Data in a Modern World
O365Engage17 - Protecting O365 Data in a Modern WorldO365Engage17 - Protecting O365 Data in a Modern World
O365Engage17 - Protecting O365 Data in a Modern World
 
ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...
ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...
ATAGTR2017 Cost-effective Security Testing Approaches for Web, Mobile & Enter...
 
Good to Great SharePoint Governance
Good to Great SharePoint GovernanceGood to Great SharePoint Governance
Good to Great SharePoint Governance
 
Rise of the Open Source Program Office for LinuxCon 2016
Rise of the Open Source Program Office for LinuxCon 2016Rise of the Open Source Program Office for LinuxCon 2016
Rise of the Open Source Program Office for LinuxCon 2016
 
Suguk Southampton CodePlex - March 2014
Suguk Southampton   CodePlex - March 2014Suguk Southampton   CodePlex - March 2014
Suguk Southampton CodePlex - March 2014
 
Web Test Automation
Web Test AutomationWeb Test Automation
Web Test Automation
 
Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...
Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...
Sendachi | 451 | GitHub Webinar: Demystifying Collaboration at Scale: DevOp...
 
IoT in the Cloud: Build and Unleash the Value in your Renewable Energy System
IoT in the Cloud: Build and Unleash the Value in your Renewable Energy SystemIoT in the Cloud: Build and Unleash the Value in your Renewable Energy System
IoT in the Cloud: Build and Unleash the Value in your Renewable Energy System
 
Engage 2018 adm04 - The lazy admin wins
Engage 2018   adm04 - The lazy admin winsEngage 2018   adm04 - The lazy admin wins
Engage 2018 adm04 - The lazy admin wins
 
Solving Real World Challenges with Enterprise Search
Solving Real World Challenges with Enterprise SearchSolving Real World Challenges with Enterprise Search
Solving Real World Challenges with Enterprise Search
 
O365Engage17 - What’s New in Office 365 Security
O365Engage17 - What’s New in Office 365 SecurityO365Engage17 - What’s New in Office 365 Security
O365Engage17 - What’s New in Office 365 Security
 
DocOps — The Analytical Window to Your Customer’s Experience with Wade Clements
DocOps — The Analytical Window to Your Customer’s Experience with Wade ClementsDocOps — The Analytical Window to Your Customer’s Experience with Wade Clements
DocOps — The Analytical Window to Your Customer’s Experience with Wade Clements
 

Viewers also liked

Copper metabolism and its clinical significance
Copper metabolism and its clinical significanceCopper metabolism and its clinical significance
Copper metabolism and its clinical significancerohini sane
 
SELENIUM METABOLISM
SELENIUM METABOLISMSELENIUM METABOLISM
SELENIUM METABOLISMYESANNA
 
Selenium in Health and Nutrition
Selenium in Health and NutritionSelenium in Health and Nutrition
Selenium in Health and NutritionCecille Alamag
 
Selenium metabolism and its clinical significance
Selenium metabolism and its clinical significanceSelenium metabolism and its clinical significance
Selenium metabolism and its clinical significancerohini sane
 
Sodium Powerpoint
Sodium PowerpointSodium Powerpoint
Sodium Powerpointguestc2f045
 
Ion selective electrodes
Ion selective electrodesIon selective electrodes
Ion selective electrodesShweta Singh
 
Iron metabolism, iron deficiency
Iron metabolism, iron deficiencyIron metabolism, iron deficiency
Iron metabolism, iron deficiencyGuvera Vasireddy
 

Viewers also liked (9)

Copper metabolism and its clinical significance
Copper metabolism and its clinical significanceCopper metabolism and its clinical significance
Copper metabolism and its clinical significance
 
SELENIUM METABOLISM
SELENIUM METABOLISMSELENIUM METABOLISM
SELENIUM METABOLISM
 
Selenium in Health and Nutrition
Selenium in Health and NutritionSelenium in Health and Nutrition
Selenium in Health and Nutrition
 
Selenium metabolism and its clinical significance
Selenium metabolism and its clinical significanceSelenium metabolism and its clinical significance
Selenium metabolism and its clinical significance
 
Sodium Powerpoint
Sodium PowerpointSodium Powerpoint
Sodium Powerpoint
 
trace minerals, Zn, Co
trace minerals, Zn, Cotrace minerals, Zn, Co
trace minerals, Zn, Co
 
Sodium metabolism
Sodium metabolismSodium metabolism
Sodium metabolism
 
Ion selective electrodes
Ion selective electrodesIon selective electrodes
Ion selective electrodes
 
Iron metabolism, iron deficiency
Iron metabolism, iron deficiencyIron metabolism, iron deficiency
Iron metabolism, iron deficiency
 

Similar to Selenium... Now What To Do Next

Integration Testing with Selenium
Integration Testing with SeleniumIntegration Testing with Selenium
Integration Testing with SeleniumAll Things Open
 
How to get Automated Testing "Done"
How to get Automated Testing "Done"How to get Automated Testing "Done"
How to get Automated Testing "Done"TEST Huddle
 
Augmenting Coded UI
Augmenting Coded UIAugmenting Coded UI
Augmenting Coded UItravisk
 
Automated Acceptance Tests & Tool choice
Automated Acceptance Tests & Tool choiceAutomated Acceptance Tests & Tool choice
Automated Acceptance Tests & Tool choicetoddbr
 
QA Role in Agile Teams
QA Role in Agile Teams QA Role in Agile Teams
QA Role in Agile Teams Synerzip
 
'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda 'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda TEST Huddle
 
Continuous Performance Testing and Monitoring in Agile Development
Continuous Performance Testing and Monitoring in Agile DevelopmentContinuous Performance Testing and Monitoring in Agile Development
Continuous Performance Testing and Monitoring in Agile DevelopmentDynatrace
 
Build the Right Regression Suite with Behavior-Driven Testing
Build the Right Regression Suite with Behavior-Driven TestingBuild the Right Regression Suite with Behavior-Driven Testing
Build the Right Regression Suite with Behavior-Driven TestingTechWell
 
How to Go Codeless for Automated Mobile App Testing
How to Go Codeless for Automated Mobile App TestingHow to Go Codeless for Automated Mobile App Testing
How to Go Codeless for Automated Mobile App TestingApplause
 
Automation Testing on Selenium by Quontra Solutions
Automation Testing on Selenium by Quontra SolutionsAutomation Testing on Selenium by Quontra Solutions
Automation Testing on Selenium by Quontra SolutionsQUONTRASOLUTIONS
 
How to scale your Test Automation
How to scale your Test AutomationHow to scale your Test Automation
How to scale your Test AutomationKlaus Salchner
 
Meaningful UI Test Automation
Meaningful UI Test AutomationMeaningful UI Test Automation
Meaningful UI Test AutomationRahul Verma
 
Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!
Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!
Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!Lorenzo Barbieri
 
Automated Testing of Web Applications
Automated Testing of Web ApplicationsAutomated Testing of Web Applications
Automated Testing of Web Applicationsjonatankronqvist
 
Meaningful UI Test Automation
Meaningful UI Test AutomationMeaningful UI Test Automation
Meaningful UI Test AutomationRahul Verma
 
Automated testing vs manual testing
Automated testing vs manual testingAutomated testing vs manual testing
Automated testing vs manual testingNeha Polke
 

Similar to Selenium... Now What To Do Next (20)

Integration Testing with Selenium
Integration Testing with SeleniumIntegration Testing with Selenium
Integration Testing with Selenium
 
How to get Automated Testing "Done"
How to get Automated Testing "Done"How to get Automated Testing "Done"
How to get Automated Testing "Done"
 
Augmenting Coded UI
Augmenting Coded UIAugmenting Coded UI
Augmenting Coded UI
 
Automated Acceptance Tests & Tool choice
Automated Acceptance Tests & Tool choiceAutomated Acceptance Tests & Tool choice
Automated Acceptance Tests & Tool choice
 
QA Role in Agile Teams
QA Role in Agile Teams QA Role in Agile Teams
QA Role in Agile Teams
 
'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda 'BIG Testing' with Hans Buwalda
'BIG Testing' with Hans Buwalda
 
Continuous Performance Testing and Monitoring in Agile Development
Continuous Performance Testing and Monitoring in Agile DevelopmentContinuous Performance Testing and Monitoring in Agile Development
Continuous Performance Testing and Monitoring in Agile Development
 
Build the Right Regression Suite with Behavior-Driven Testing
Build the Right Regression Suite with Behavior-Driven TestingBuild the Right Regression Suite with Behavior-Driven Testing
Build the Right Regression Suite with Behavior-Driven Testing
 
How to Go Codeless for Automated Mobile App Testing
How to Go Codeless for Automated Mobile App TestingHow to Go Codeless for Automated Mobile App Testing
How to Go Codeless for Automated Mobile App Testing
 
Automation Testing on Selenium by Quontra Solutions
Automation Testing on Selenium by Quontra SolutionsAutomation Testing on Selenium by Quontra Solutions
Automation Testing on Selenium by Quontra Solutions
 
How to scale your Test Automation
How to scale your Test AutomationHow to scale your Test Automation
How to scale your Test Automation
 
Meaningful UI Test Automation
Meaningful UI Test AutomationMeaningful UI Test Automation
Meaningful UI Test Automation
 
Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!
Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!
Azure DevOps Realtime Work Item Sync: the good, the bad, the ugly!
 
Automated Testing of Web Applications
Automated Testing of Web ApplicationsAutomated Testing of Web Applications
Automated Testing of Web Applications
 
Qtp - Introduction values
Qtp - Introduction valuesQtp - Introduction values
Qtp - Introduction values
 
jForce In Action
jForce In ActionjForce In Action
jForce In Action
 
Future of QA
Future of QAFuture of QA
Future of QA
 
Futureofqa
FutureofqaFutureofqa
Futureofqa
 
Meaningful UI Test Automation
Meaningful UI Test AutomationMeaningful UI Test Automation
Meaningful UI Test Automation
 
Automated testing vs manual testing
Automated testing vs manual testingAutomated testing vs manual testing
Automated testing vs manual testing
 

More from Great Wide Open

The Little Meetup That Could
The Little Meetup That CouldThe Little Meetup That Could
The Little Meetup That CouldGreat Wide Open
 
Lightning Talk - 5 Hacks to Getting the Job of Your Dreams
Lightning Talk - 5 Hacks to Getting the Job of Your DreamsLightning Talk - 5 Hacks to Getting the Job of Your Dreams
Lightning Talk - 5 Hacks to Getting the Job of Your DreamsGreat Wide Open
 
Dealing with Unstructured Data: Scaling to Infinity
Dealing with Unstructured Data: Scaling to InfinityDealing with Unstructured Data: Scaling to Infinity
Dealing with Unstructured Data: Scaling to InfinityGreat Wide Open
 
You Don't Know Node: Quick Intro to 6 Core Features
You Don't Know Node: Quick Intro to 6 Core FeaturesYou Don't Know Node: Quick Intro to 6 Core Features
You Don't Know Node: Quick Intro to 6 Core FeaturesGreat Wide Open
 
Using Cryptography Properly in Applications
Using Cryptography Properly in ApplicationsUsing Cryptography Properly in Applications
Using Cryptography Properly in ApplicationsGreat Wide Open
 
Lightning Talk - Getting Students Involved In Open Source
Lightning Talk - Getting Students Involved In Open SourceLightning Talk - Getting Students Involved In Open Source
Lightning Talk - Getting Students Involved In Open SourceGreat Wide Open
 
How Constraints Cultivate Growth
How Constraints Cultivate GrowthHow Constraints Cultivate Growth
How Constraints Cultivate GrowthGreat Wide Open
 
Troubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed DebuggingTroubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed DebuggingGreat Wide Open
 
The Current Messaging Landscape
The Current Messaging LandscapeThe Current Messaging Landscape
The Current Messaging LandscapeGreat Wide Open
 
Understanding Open Source Class 101
Understanding Open Source Class 101Understanding Open Source Class 101
Understanding Open Source Class 101Great Wide Open
 
Elasticsearch for SQL Users
Elasticsearch for SQL UsersElasticsearch for SQL Users
Elasticsearch for SQL UsersGreat Wide Open
 
Open Source Security Tools for Big Data
Open Source Security Tools for Big DataOpen Source Security Tools for Big Data
Open Source Security Tools for Big DataGreat Wide Open
 

More from Great Wide Open (20)

The Little Meetup That Could
The Little Meetup That CouldThe Little Meetup That Could
The Little Meetup That Could
 
Lightning Talk - 5 Hacks to Getting the Job of Your Dreams
Lightning Talk - 5 Hacks to Getting the Job of Your DreamsLightning Talk - 5 Hacks to Getting the Job of Your Dreams
Lightning Talk - 5 Hacks to Getting the Job of Your Dreams
 
Dealing with Unstructured Data: Scaling to Infinity
Dealing with Unstructured Data: Scaling to InfinityDealing with Unstructured Data: Scaling to Infinity
Dealing with Unstructured Data: Scaling to Infinity
 
You Don't Know Node: Quick Intro to 6 Core Features
You Don't Know Node: Quick Intro to 6 Core FeaturesYou Don't Know Node: Quick Intro to 6 Core Features
You Don't Know Node: Quick Intro to 6 Core Features
 
Hidden Features in HTTP
Hidden Features in HTTPHidden Features in HTTP
Hidden Features in HTTP
 
Using Cryptography Properly in Applications
Using Cryptography Properly in ApplicationsUsing Cryptography Properly in Applications
Using Cryptography Properly in Applications
 
Lightning Talk - Getting Students Involved In Open Source
Lightning Talk - Getting Students Involved In Open SourceLightning Talk - Getting Students Involved In Open Source
Lightning Talk - Getting Students Involved In Open Source
 
How Constraints Cultivate Growth
How Constraints Cultivate GrowthHow Constraints Cultivate Growth
How Constraints Cultivate Growth
 
Inner Source 101
Inner Source 101Inner Source 101
Inner Source 101
 
Running MySQL on Linux
Running MySQL on LinuxRunning MySQL on Linux
Running MySQL on Linux
 
Search is the new UI
Search is the new UISearch is the new UI
Search is the new UI
 
Troubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed DebuggingTroubleshooting Hadoop: Distributed Debugging
Troubleshooting Hadoop: Distributed Debugging
 
The Current Messaging Landscape
The Current Messaging LandscapeThe Current Messaging Landscape
The Current Messaging Landscape
 
Apache httpd v2.4
Apache httpd v2.4Apache httpd v2.4
Apache httpd v2.4
 
Understanding Open Source Class 101
Understanding Open Source Class 101Understanding Open Source Class 101
Understanding Open Source Class 101
 
Thinking in Git
Thinking in GitThinking in Git
Thinking in Git
 
Antifragile Design
Antifragile DesignAntifragile Design
Antifragile Design
 
Elasticsearch for SQL Users
Elasticsearch for SQL UsersElasticsearch for SQL Users
Elasticsearch for SQL Users
 
Open Source Security Tools for Big Data
Open Source Security Tools for Big DataOpen Source Security Tools for Big Data
Open Source Security Tools for Big Data
 
Access by Default
Access by DefaultAccess by Default
Access by Default
 

Recently uploaded

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024BookNet Canada
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentationphoebematthew05
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfngoud9212
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitecturePixlogix Infotech
 
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024BookNet Canada
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesSinan KOZAK
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Neo4j
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 

Recently uploaded (20)

Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptxVulnerability_Management_GRC_by Sohang Sengupta.pptx
Vulnerability_Management_GRC_by Sohang Sengupta.pptx
 
costume and set research powerpoint presentation
costume and set research powerpoint presentationcostume and set research powerpoint presentation
costume and set research powerpoint presentation
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Bluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdfBluetooth Controlled Car with Arduino.pdf
Bluetooth Controlled Car with Arduino.pdf
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
 
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen FramesUnblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 

Selenium... Now What To Do Next

  • 1. You have Selenium... Now what? Hans Buwalda ©2016 LogiGear Corporation www.logigear.com www.testarchitect.com www.happytester.com @hansbuwalda hans@logigear.com hans@happytester.com
  • 2. ©2016 LogiGear Corporation • Topics: – domain language approaches – test design as driver for automation – automation friendly test design techniques and examples – special cases (data driven, multi-station, graphics, etc) – drone perspective: overview with some selective details • What this talk is not: – an interactive workshop – specific to Selenium only – a technical class on automation – an interactive workshop – about component based development – introduction to unit testing Scope Warning: there is an overlap with my talk yesterday!!
  • 3. ©2016 LogiGear Corporation • A combination of open source products and standards, aimed at web testing (including a Firefox based IDE) • In the later versions it works with an interface called “WebDriver”, implemented with drivers in browsers • WebDriver has been implemented in open source libraries, but is now also becoming implemented in browsers • For example Microsoft Edge can only be accessed programmatically via WebDriver • Products like Appium try to implement WebDriver for mobile browsing and even mobile apps • Our product (TestArchitect) is moving to WebDriver, expect this to be a common practice Selenium
  • 4. ©2016 LogiGear Corporation Relation to code Quality / depth Automation Scalability Unit Testing Close relationship with the code Singular test scope, but deep into the code Fully automated by nature Scalable, grows with the code, easy to repeat Functional/ Integration Testing Usually does not have a one-on-one relation with code Quality and scope depends on test design In particular UI based automation can be a challenge Often a bottle- neck in scalability Exploratory Testing Human driven, not seeking a relation with code Usually deep and thorough, good at finding problems May or may not be automated afterwards Not meant to be repeatable. Rather do a new session Some test kinds and their scalability (simplified)
  • 5. ©2016 LogiGear Corporation Test Pyramid • Proposed by Mike Cohn (in his book: "Succeeding with Agile"). • My version would be less "pointy" • In particular I like to separate technical and design considerations • Also, I've seen API's that, for higher level operations, are quite complex. Trying to use them can become like re-implementing the UI again • Web UI's are getting richer and more complex to test (SPA's, AngularJS, etc) UI Service or Component Unit UI Unit original my version (for now) Service or Component
  • 6. ©2016 LogiGear Corporation Issues are not always obvious... Downton Abbey
  • 7. ©2016 LogiGear Corporation Brief test design exercise • View this imaginary application • It adds two numbers • What tests would you create? • Which tests would you automate?
  • 8. ©2016 LogiGear Corporation The 5% Rules of Test Automation • No more than 5% of all test cases should be executed manually • No more than 5% of all efforts around testing should involve automating the tests
  • 9. ©2016 LogiGear Corporation Don't just automate manual testing
  • 10. ©2016 LogiGear Corporation Don't just automate manual testing
  • 11. ©2016 LogiGear Corporation Don't just automate manual testing Good automated testing is not the same as automating good manual testing. . .
  • 12. ©2016 LogiGear Corporation Example Scripting, Java + Selenium WebElement element = driver.findElement(By.name(name)); element.sendKeys("mystery magic"); element.submit(); (new WebDriverWait(driver, 10)).until( new ExpectedCondition<Boolean>() { public Boolean apply(WebDriver d) { return d.getTitle() .toLowerCase().startsWith("mystery"); } } ); System.out.println(driver.getTitle());
  • 13. ©2016 LogiGear Corporation • Frameworks, libraries, patterns – libraries: sets of functions, you call them – frameworks: also sets of functions, but the framework is typically in charge and calls you – patterns: standardized ways to do certain tasks • Selenium is often used with frameworks – to execute tests and collect results (like unit testing frameworks) – to offer higher level API's – to support BDD, data driven testing or keywords • A common practice is the Page Object pattern – shield from HTML – provide an application specific API, with functions like "getSongTitle" • Examples of keyword frameworks: Robot Framework, and our own TestArchitect Libraries, frameworks, patterns Framework
  • 14. ©2016 LogiGear Corporation Domain language approaches: Actions 4 actions, each with an action keyword and arguments read from top to bottom fragment from a test with actions acc nr first last open account 123123 John Doe acc nr amount deposit 123123 10.11 deposit 123123 20.22 acc nr expected check balance 123123 30.33 • Stays out of the programming language • The test developer creates tests using actions with keywords and arguments • Checks are, as much as possible, explicit (specified expected values) • The automation task focuses on automating the keywords, each keyword is automated only once
  • 15. ©2016 LogiGear Corporation • In "low level" tests interaction details are not hidden, since they are the target of the test • The right level of abstraction depends on the scope of the test, and is an outcome of your test design process TEST MODULE Screen Flow user start system john TEST CASE TC 01 Order button window button click main create order window check window exists new order FINAL close application Example of a low level test
  • 16. ©2016 LogiGear Corporation Variables and expressions with keywords • This test does not need an absolute number for the available cars, just wants to see if a stock is updated • As a convention we denote an assignment with ">>" • The "#" indicates an expression TEST CASE TC 02 Rent some more cars car available get quantity Chevy Volt >> volt first name last name car rent car John Green Chevy Volt rent car Jane White Chevy Volt car expected check quantity Chevy Volt # volt - 2
  • 17. ©2016 LogiGear Corporation Data driven testing with keywords • The test lines will be repeated for each row in the data set • The values represented by "car", "first" and "last" come from the selected row of the data set TEST CASE TC 03 Check stocks data set use data set /cars car available get quantity # car >> quantity first name last name car rent car # first # last # car car expected check quantity # car # quantity - 1 repeat for data set DATA SET cars first last car John Doe Chevy Volt Mary Kane Ford Escape Jane Collins Chrysler 300 Tom Anderson Buick Verano Henry Smyth BMW 750 Vivian Major Toyota Corolla
  • 18. ©2016 LogiGear Corporation ACTION DEFINITION check balance user argument customer argument amount window control value enter balance inquiry last name # customer window control click balance inquiry view balance window control expected check balance inquiry balance # amount Re-use actions to make new actions • In the below example we make a new action • Existing actions are strung together to create new ones with a broader scope • Often steps in low level tests are re-used to create these action definitions : customer amount check balance Smith 223.45 check balance Jones 0.00 check balance James -330.45 : use many times in tests: define in one place:
  • 19. ©2016 LogiGear Corporation Example action implementation in Python This action script (in Python, Java or C#) verifies whether the rows in a table are sorted: # get table object, column number and column count windowName = LIBRARY.NamedArgument("window") tableName = LIBRARY.NamedArgument("table") columnName = LIBRARY.NamedArgument("column") table = ABT.OpenElement(windowName, tableName) column = table.GetColumnIndex(columnName) rowCount = table.GetRowCount() # check the sort order, row by row previous = table.GetCellText(0, column) for i in range(1, rowCount): current = table.GetCellText(i, column) if current < previous : LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0) return previous = current LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1) find the table in the UI if a value is smaller than before, fail the test if all rows are ascending, pass the test get arguments from the test line def action_checkSortOrder(): Note that the approach resembles the Page Object pattern, but in my view is not the same. The abstraction is “any table”, even in non-Web UI’s or non-UI’s altogether. The Page Object pattern could also work on top of this.
  • 20. ©2016 LogiGear Corporation Using the new action • By keeping an action generic it can be applied for a variety of situations • Some examples of using "check sort order": window table column check sort order view orders orders table ID window table column check sort order annual results regions revenue window table column check sort order inventory cars price window table column check sort order registration students last name
  • 21. ©2016 LogiGear Corporation Behavior Driven Development (BDD) • Uses natural language scenarios • Tools like JBehave and Cucumber map these to code • Structure is "Given-When-Then" (GWT) • Example: Given a customer previously bought a black sweater from me And I currently have three black sweaters left in stock When he returns the sweater for a refund Then I should have four black sweaters in stock (source: Wikipedia)
  • 22. ©2016 LogiGear Corporation BDD with keywords (example) GIVEN A customer previously bought a black sweater from me first last id add client John Jones >> client AND I currently have three black sweaters left in stock article color quantity get stock sweater black >> sweaters WHEN He returns the sweater for a refund customer article color return article # client sweater black THEN I should have four black sweaters in stock article color quantity check stock sweaters black # sweaters + 1
  • 23. ©2016 LogiGear Corporation Equivalence, conversion Given a customer previously bought a black sweater from me And I currently have three black sweaters left in stock When he returns the sweater for a refund Then I should have four black sweaters in stock customer buys, article, color a customer previously bought a {color} {article} from me set stock, article, color, amount I currently have {amount} {color} {article} left in stock my stock of {color} {sweater} is {amount} return article, article, color he returns the {color} {article} for a refund check stock, article, color I should have four {color} {article} in stock article color customer buys sweater black article color amount set stock sweater black 3 article color return article sweater black article color amount check stock sweater black 4
  • 24. Test Module Sections Test Module Objectives Initial - setup Test cases Final - cleanup
  • 25. High Level Test Design - Test Development Plan Objectives Test Module 1 Test Cases Test Module 2 Test Module N Actions . . . AUTOMATION Objectives Objectives interaction test business test define the "chapters" create the "chapters" create the "words" make the words work Test Cases Test Cases window control value enter log in user name jdoe enter log in password car guy window control property expected check property log in ok button enabled true user password log in jdoe car guy first last brand model enter rental Mary Renter Ford Escape last total check bill Renter 140.42 Action Based Testing
  • 26. ©2016 LogiGear Corporation Organize the tests • What tests would you create for this application? • How would you organize them? • What, if anything, would you avoid?
  • 27. ©2016 LogiGear Corporation Why Better Test Design? • Better test design can improve quality of test – many tests are often quite "mechanical" now, no surprises – one to one related to specifications, user stories or requirements, which often is ok, but lacks aggression – no combinations, no unexpected situations, lame and boring – such tests have a hard time finding (interesting) bugs • Better test design can give (much) better automation – when unneeded details are left out of tests, those details don't have to be maintained – avoiding "over checking": creating checks that are not in the scope of a test, but may fail after system changes – limit the impact of system changes on tests, making such impact more manageable I have become to believe that successful automation is usually less of a technical challenge as it is a test design challenge.
  • 29. ©2016 LogiGear Corporation • Business objects and business flows – objects are like cars, invoices, locations, etc – flows are like create, fulfill, pay and close an order • Other tests – functions and features, like premium calculation or PDF output – administration, users, security, authorizations – graphics – technologies, protocols, ... – customization, extensibility – . . . • Business versus interaction – differentiate within business objects and other categories – interaction can be further differentiated into: values, UI, keyboard, etc Example of considerations
  • 30. ©2016 LogiGear Corporation Example Top Level Structure Project create, update, delete/close copy, move categorize, enumerate, identify convert, serialize, export/import, ... UI, dialogs, forms, pages input (validation, defaulting, dependencies) flows (primary paths, alternate paths) keyboard shortcuts, keyboard controls, ... . . . <business object 1> Lifecycles, data operations Interaction Functions and Features Technologies, protocols, controls Data (handling, quality, ETL, ...) Security, authorization, admin Graphics, multi-media, charts, ... Load, performance Integration Customizing, extensibility Business Flows Concurrency, race conditions, ... Business Objects processes, transactions, end-to-end, day in the life, combinations of flows, ... calculations, analyses, PDF output, ... Repeat for each component or sub-system
  • 31. ©2016 LogiGear Corporation Example: E-commerce site • Articles • Categories • Customers • Promotions • Orders • Invoices • Payments • Credit cards • Staff members • Countries • . . .
  • 32. ©2016 LogiGear Corporation • Promotion kinds – percentage discounts – fixed dollar discounts – comprehensive promotions – regional promotions – volume discounts – prompt payment discounts – article based, comprehensive – . . . • Life cycles – creation – modification • various properties • before or after activation – expiration – cancellation Promotions -- business tests
  • 33. ©2016 LogiGear Corporation Promotions -- interaction tests • Screen by screen • Discount percentage/fixed – does the amount field change • Regional checkbox – does the region list show up – are the regions correct – can the regions be selected/deselected • Start, duration, end date – are the date checks working – can either duration or end date be entered • Articles, categories – can categories be selected, do they show up – do the right articles show up, can they be selected • . . .
  • 34. ©2016 LogiGear Corporation What tests could look like Business tests for business object "Promotions" Interaction tests for business object "Promotions" window control click main new promotion window type select promotion town window list value check list item exists promotion towns Tietjerksteradeel name start finish percentage category create promotion christmas 12/1/2016 12/25/2016 11 all create promotion tablets 12/20/2016 1/1/2017 20 tablets date time travel 12/23/2016 article price check nett price iPad Mini 4 284.79
  • 35. ©2016 LogiGear Corporation Example 2: a testing company in California • The company sells services like test development and automation • It also has consultancy and training offerings • And it has a product for test development and automation • These various offerings are often combined in the orders they receive • Invoicing for services as monthly • They're working on an internal system for order handling and invoicing
  • 36. ©2016 LogiGear Corporation Example 2: a testing company in California • Example business objects: – projects, orders ("SOW's"), customers, employees, invoices, payments • Example business flows: – customer place orders, often multiple ones over time – one order can have multiple items – tools are invoiced (and delivered) immediately – services are delivered over time – staff members write time-sheets with hours spent • has elaborate UI's and processes – invoices can combine items from multiple orders – payments can cover multiple invoices
  • 37. ©2016 LogiGear Corporation What about existing tests? • Compare to moving house: – some effort can't be avoided – be selective, edit your stuff, • look at the future, not the past – first decide where to put what, then put it there – moving is an opportunity, you may not get such chance again soon • Follow the module approach – define the modules and their scope as if from scratch – use the existing test cases in two ways: • verify completeness • harvest and re-use them for tests and for actions – avoid porting over "step by step", in particular avoid over-checking
  • 38. ©2016 LogiGear Corporation Eye on the ball, Scope • Always know the scope of the test module • The scope should be unambiguous • The scope determines many things: – what the test objectives are – which test cases to expect – what level of actions to use – what the checks are about and which events should generate a warning or error (if a “lower” functionality is wrong)
  • 39. ©2016 LogiGear Corporation Use the right level actions Low level of UI interaction detail makes sense only with the module scope is to test the UI window tree tree item path click tree item main projects /Projects/Drill Assembly window list item check list item exists main tasks Plan of Approach Better to have a business level action and hide the details in an action definition: But do show details when it matters. The example below is too high level, requires drill down into action definition to understand what is tested.
  • 40. ©2016 LogiGear Corporation Example of a Test Step Description Expected step 16 Open http://www.bigstore.com The "BIG Store" main page is displayed, with a "sign in" link step 17 Click on "Sign In", upper right corner A sign in dialog shows, the "Sign in" button is disabled step 18 Enter "johnd" in the user name field The "Sign In" button is still disabled step 19 Enter "bigtester" in the password field Now the "Sign In" button is enabled step 20 Click on the "Sign in" button The page now shows "Hello John" in the upper right corner step 21 Enter "acme watch" in the search field The "Search" button is enabled step 22 Click on the "Search" button 5 watches of Acme Corporation are displayed step 23 Double click on "Acme Super Watch 2" The details page of the Acme Super Watch 2 is displayed step 24 Verify the picture of the watch The picture should show a black Acme Super Watch 2 step 25 Select "red" in the "Color" dropdown list The picture now shows a black Acme Super Watch 2 step 26 Type 2 in the "Order quantity" textbox The price in the right shows "$79.00 + Free Shipping" step 27 Click on "Add to cart" The status panel shows "Acme Super Watch 2" added step 28 Click on "Check out" The Cart Check Out open, with the 2 Acme Super Watches is this a good test? is it good for automation?
  • 41. ©2016 LogiGear Corporation GWT scenario Given User turns off Password required option for Drive Test And User has logged in by Traffic Applicant account And User is at the Assessments Take a Test page And User clicks the Traffic Test link And User clicks the Next button And User clicks the Sheet radio button in Mode page if displayed And User clicks the Start button And User waits for test start And User clicks the Stop Test button When User clicks the Confirm Stop Test button And User enters the correct applicant password And User clicks the Confirm Stop Test button Then The Test is Over should be displayed in the Message label And the value of the Message label should be The test is over And The Welcome to Traffic Testing page should be displayed
  • 42. ©2016 LogiGear Corporation "Anti-patterns" (informal) • Interaction Heavy – Not having many business tests • Lame – No depth or variety, no testing techniques used • Enter Enter Click Click – Test steps are too detailed • No Life – Missing life cycle steps of business objects • Clueless – No clear scope for the tests • Cocktail – Interaction tests mixed with business tests • Over-Checking – Checks not relevant for the scope • Sneaky Checking – Checks hidden in actions • Action Explosion – Many actions, hardly different • Mystery Actions – Cryptic actions, unclear what they do • Techno – Actions and tests that look like code – often _NOts0EasY_2REad – but great to impress non-technical people
  • 43. ©2016 LogiGear Corporation Example of a test module TEST MODULE Order processing start system TEST CASE TC 01 Order for tablets user password login jdoe doedoe window check window exists welcome order id cust id article price quantity create order AB123 W3454X tablet 198.95 5 order id total check order total AB123 994.75 . . .
  • 44. ©2016 LogiGear Corporation Low-level, high-level, mid-level actions • Low-level (system level) actions – detailed interaction with the UI (or API) – generic, do not show any functional or business logic – examples: "click", "expand tree node", "select menu" • High-level (application and business level) actions – represent a business function or event fitting the scope of the test – hides the interaction – examples: "enter customer", "rent car", "check balance" • Mid-level actions: auxiliary actions that represent common sequences of low level actions – usually to wrap a page or dialog – enhance maintainability – example: "enter address fields“ – complex pages could be further split up like with the Page Object pattern enter customer enter address fields enter select set . . .. . .
  • 45. Tip: Provide default values in actions ACTION DEFINITION login name default value argument user tester argument password testerpw window control value enter login window user name # user enter login window password # password window control click login window login user password login tamaraj tj1234 text check message Hello Tamara login date payee amount make payment 1/1/12 Gas Co. 85.00 Use login action w/ arguments Use login default values
  • 46. ©2016 LogiGear Corporation ABT in Agile Test Module Definition (optional) Test Module Development Interface Definition Action Automation Test Execution Sprint Products Product Backlog Test re-use Automation re-use product owner team prod owner & team User stories Documentation Domain understanding Acceptance Criteria PO Questions Situations Relations Agile life cycle Test development Main Level Test Modules Interaction Test Modules Cross over Test Modules
  • 47. ©2016 LogiGear Corporation Using ABT in Sprints (1) • Try to keep the testers in the same sprint as the rest of the team – don't let automated tests clutter up and lack behind – note that in many environments tests and their automation are not highest priority • Agree on the approach: – is testability a requirement for the software? – do we regard tests, or some of the tests, as products? • would potentially become part of the backlog • would not only be up to the team which tests to create • Just like for development, use discussions with the team and product owners – deepen understanding, for the whole team – help identify items like negative, alternate and unexpected situations – tests can work as driver for development (TDD and ATDD)
  • 48. ©2016 LogiGear Corporation Using ABT in Sprints (2) • Create good starting conditions for a sprint: – automation technology available (UI, non-UI, custom controls, graphics, ...) – know how you will deal with data and environments – understanding of subject matter, testing, automation, etc • Do interface mapping by hand, using developer provided identifications – saves time by not having to use the viewer or other spy tools – recording of actions (not tests) will go better To discuss an approach, consider daily "sit down" meetings with some or all members to coach and evaluate  an end-of-day counterpart to the early-morning "stand up" meetings  short and friendly, not about progress and impediments, but about practices and experiences with them (like "what actions did you use?")  a few meetings may suffice
  • 49. Multiple System Access System Under Test Action Automation Web service access command line access Web UI access database access Test Modules
  • 50. ©2016 LogiGear Corporation Testing in, or near, production • In an agile and DevOps approach continuous deployment is becoming the norm – in particular for service based system (like web, apps, SaaS, but also client-server) – logic in services is easier to manage and updated than distributed and variable clients • In a DevOps approach tight cooperation between development, test and deployment pays off – automation testability – testable system designs – good test design remains a success factor – tight integration in the build, deployment and production processes • A/B testing can help test vary complex systems – used not only for testing, also for trying out new features – dividing the incoming traffic into an A and B flow (B is the new situation) – automated tests can use the B flow – to do this, have it well integrated in your system designs • Continuous testing with random regression testing or monkey testing *see also: Ken Johnston's chapter in the book of Dorothy Graham and Mark Fewster, and his keynote at StarWest 2012
  • 51. ©2016 LogiGear Corporation A/B testing with a reverse proxy • A/B testing means part of traffic is routed through a different server or component (see if it works, and/or how users react) • B could be a real-life user, a test user or also an automated test • The strategy can be done at any component level • Watch your test design, easy to drown in technical only A A B Reverse Proxy Users Servers A B newcurrent A B
  • 52. ©2016 LogiGear Corporation • Passive timing – wait a set amount of time – in large scale testing, try to avoid passive timing altogether: • if wait too short, test will be interrupted • if wait too long, time is wasted • Active timing – wait for a measurable event – usually the wait is up to a, generous, maximum time – common example: wait for a window or control to appear (usually the test tool will do this for you) • Even if not obvious, find something to wait for... • Involve developers if needed – relatively easy in an agile team, but also in traditional projects, give this priority • If using a waiting loop – make sure to use a "sleep" function in each cycle that frees up the processor (giving the AUT time to respond) – wait for an end time, rather then a set amount of cycles Active Timing
  • 53. ©2016 LogiGear Corporation Things to wait for... • Wait for a last control or elements to load – developers can help knowing which one that is • Non-UI criteria – API function – existence of a file • Criteria added in development specifically for this purpose, like: – "disabling" big slow controls (like lists or trees) until they're done loading – API functions or UI window or control properties • Use a "delta" approach: – every wait cycle, test if there was a change; if no change, assume that the loading time is over: – examples of changes: • the controls on a window • count of items in a list • size a file (like a log file)
  • 54. ©2016 LogiGear Corporation • Should be a "must have" requirement – first question in a development project: "how do we test this?" • Design of the system(s) under test: – components, tiers, services, • Hidden identifying properties • Hooks for timing • White-box access to anything relevant: – input data (ability to emulate) – output data (what is underlying data being displayed) – random generators (can I set a seed?) – states (like in a game) – objects displayed (like monsters in a game) • Emulation features, like time-travel and fake locations Testability, key items
  • 55. ©2016 LogiGear Corporation Versions, environments, configurations • Many factors can influence details of automation – language, localization – hardware – version of the system under test – system components, like OS or browser • Test design can reflect these – certain test modules are more general – others are specific, for example for a language • But for tests that do not care about the differences, the automation just needs to "deal" with them – shield them from the tests minimum safe distance from a bear is 91 meters LOCALIZATION converting yards to meters
  • 56. Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there). Variation Variation Variation "Variations" "Master Switch" Actions, Interface Definitions . . .
  • 57. Possible set up of variations linked variation keyworded variation Specify for example in a dialog when you start an execution:
  • 58. ©2016 LogiGear Corporation • Not every test is straightforward UI based • Always go back to the test: what is it I want • Express that with actions – make them up as you go – assume everything is possible • For the automation side, ask for advise – you pattern might have been seen before Special situations
  • 59. ©2016 LogiGear Corporation "Lead Deputy" Testing • For "multi station" testing, when multiple machines have to participate in a test in real-time • For example if a supervisor needs to approve a withdrawal in a bank teller system • Can be "synchronized" and "parallel" (with a rendezvous point) lead machine playing the "bank teller" deputy machine playing the "supervisor" acc nr amount tx id request 123123 10,000 >> tx name use deputy supervisor tx id amount approve # tx 10,000 use lead tx id amount pay out # tx 10,000 lead machine playing the "bank teller" ... ...
  • 60. Device Testing Software Under Test Agent ABT Automation Interface Info Testing Host Device Androi d
  • 61. ©2016 LogiGear Corporation • Approach applicable for pictures like graphics or icons • The tester will add a line "check picture", that includes a "question" as one of the arguments • While the test is executed TA keeps the recorded pictures • After execution the pictures are shown to a manual testing for approval • Once approved unchanged same pictures won't be presented to the manual tester again in future runs Multimedia: The "check picture" Approach
  • 62. ©2016 LogiGear Corporation • Automation is not as much a technical challenge as it is a test design challenge • Domain language based approaches like BDD and Actions can make tests are accessible, and easier to maintain • However, some kind of systematic approach, like the ABT method, is necessary to keep the tests manageable • Testing is teamwork, including automation engineers, developers and product owners • From a test design perspective approaches for non-UI and UI driven testing are very similar Summary hans@logigear.com That’s all folks for this talk. However, I have much more stuff, please let me know any questions.
  • 63. ©2016 LogiGear Corporation • Automation is not as much a technical challenge as it is a test design challenge • Domain language based approaches like BDD and Actions can make tests are accessible, and easier to maintain • However, some kind of systematic approach, like the ABT method, is necessary to keep the tests manageable • From a test design perspective approaches for non-UI and UI driven testing are very similar Summary hans@logigear.com That’s all folks for this talk. However, I have much more stuff, please let me know any questions.