Managing Successful Test Automation
Contents
Session 0: Introduction to the tutorial
Objectives, What we cover (and don’t ...
0-1

Managing
Successful Test Automation
Prepared and presented by

Dorothy Graham
www.DorothyGraham.co.uk
email: info@dor...
0-2

Tutorial contents
1) planning & managing test automation
2) testware architecture
3) pre and post processing
4) scrip...
0-3

What is today about? (and not about)
•  test execution automation (not other tools)
•  I will NOT cover:
–  demos of ...
0-4

Current tool adoption (Australia)
•  K.J. Ross & Associates survey 2010
–  19%
–  7%

good benefits
using tools but n...
1-Managing

Successful Test Automation

Planning & Managing Test Automation

1 Managing

2 Architecture

3 Pre- and Post

...
1-Managing

What is an automated test?
•  a test!
–  designed by a tester for a purpose

•  test is executed
–  implemente...
1-Managing

Responsibilities
Testers
•  test the software
–  design tests
–  select tests for automation
•  requires plann...
1-Managing

Recent poll
Testers as automators?
Every tester should be able to
write code
Testers can be automators if
they...
1-Managing

Agile automation: Lisa Crispin
–  starting point: buggy code, new functionality
needed, whole team regression ...
1-Managing

Automation in agile/iterative development
A

manual testing of
this release (testers)
A

B

regression testing...
1-Managing

Managing

Successful Test Automation

1

2

3

4

5

6

Contents
Responsibilities
Pilot project
Test automatio...
1-Managing

Pilot project
•  reasons
–  you’re unique
–  many variables /
unknowns at start

•  benefits
–  find the best ...
1-Managing

After the pilot…
•  having processes & standards is only the start
–  30% on new process
–  70% on deployment
...
1-Managing

An automation effort
•  is a project
–  with goals, responsibilities, and monitoring
–  but not just a project...
1-Managing

Good objectives for automation?
–  run regression tests evenings and weekends
–  give testers a new skill / en...
Test Automation Objectives Exercise

Test Automation Objectives Exercise
The following are some possible test automation o...
Test Automation Objectives Exercise

Possible test automation objectives

Good
automation objective?
(If not, why not)

Al...
1-Managing

Reduce test execution time
edit tests
(maintenance) set-up

execute

analyse
failures

clear-up

Manual
testin...
1-Managing

Manual vs automated
manual
tests

automated
tests

tests not
automated
yet

tests not
worth
automating

tests ...
1-Managing

Exploratory Test Automation - 2
•  set off lots of tests that produce something that
can be checked with an au...
1-Managing

What is automated?
most often
automated
likelihood of
finding bugs

regression tests

exploratory testing
1-29...
1-Managing

Good objectives for test automation
•  measurable
–  EMTE (Equivalent Manual Test Effort)
–  tests run, covera...
1-Managing

Managing

Successful Test Automation

1

2

3

4

5

6

Contents
Responsibilities
Pilot project
Test automatio...
1-Managing

Automation measures?
•  aspects of automation
–  number of times tests are run
–  time to run the automated te...
1-Managing

EMTE – how does it work?
a manual test

Manual testing

Automate the manual testing?
the manual test
now autom...
1-Managing

EMTE example
•  example
–  automated tests take 2 hours
–  if those same tests were run manually, 4 days

•  f...
1-Managing

Managing

Successful Test Automation

1

2

3

4

5

6

Contents
Responsibilities
Pilot project
Test automatio...
1-Managing

What can you show as a benefit?
•  number of additional times tests are run
–  how many times run manually & a...
1-Managing

EMTE over releases
EMTE per release
800
700
600
500
400
300
200
100
0
1

2

3

4

5

2708 Total “manual” hours...
1-Managing

ROI example
Human Time
to run
Savings per
automated
automated No. automated Total
run
runs per release savings...
1-Managing

Example ROI graph using MBT
1400

1200

1000

People
Hours

800
Manual hrs
Automated hrs

600

400

200

0
1

...
1-Managing

Large S Africa bank: Michael Snyman
•  was project-based, too late, lessons not learned
–  “our shelves were l...
1-Managing

Why is measuring ROI dangerous?
•  focus on what’s easy to measure (tester time)
•  other factors may be much ...
1-Managing

Sample ‘starter kit’ for metrics for test
automation (and testing)
•  some measure of benefit
–  e.g. EMTE or ...
1-Managing

Managing

Successful Test Automation

1

2

3

4

5

6

Summary: key points
• 
• 
• 
• 
• 

Assign responsibil...
Test Automation Objectives Solution

Test Automation Objectives Solution
We have given some ideas as to which objectives a...
Test Automation Objectives Solution

Good
automation objective?
(If not, why not)

Possible test automation objectives
Imp...
Test Automation Objectives Solution
Test Automation Objectives: Selection and Measurement
On this page, record the test ob...
2-Architecture

Successful Test Automation

Testware Architecture

1 Managing

2 Architecture

3 Pre- and Post

4 Scriptin...
2-Architecture

Testware architecture

testware'architecture'

Testers''
write'tests'(in'DSTL)'

abstraction here:
easier ...
2-Architecture

Easy way out: use the tool’s architecture
•  tool will have its own way of organising tests
–  where to pu...
2-Architecture

Learning is incremental: Molly Mahai
–  book learning – knew about investment, not
replace people, don’t a...
2-Architecture

A test for you
•  show me one of your automated tests running
–  how long will it take before it runs?

• ...
2-Architecture

Terms

- Testware artefacts
Testware

Test Materials

Test Results
Products

inputs

By-Products

scripts
...
2-Architecture

Testware by type
Testware
Test Materials
countries.scp
open.scp saveas.scp
testdef.txt
countries.dcm

Test...
2-Architecture

Testware Sets
•  Testware Set:
–  a collection of testware artifacts
–  four types:
•  Test Set - one or m...
2-Architecture

Separate test results

Test
suite

Software
under test

Test
results

2-17

Incremental approach: Ursula F...
2-Architecture
Architecture

1

2

3

4

5

Successful Test Automation

6

Summary: key points
• 
• 
• 

Structure your au...
3-Pre and Post Processing

Successful Test Automation

Pre- and Post-Processing

1 Managing

2 Architecture

3 Pre- and Po...
3-Pre and Post Processing
What is pre- and post-processing?
•  Pre-processing
–  automation of setup tasks necessary to fu...
3-Pre and Post Processing
Examples
•  pre-processing
–  copy scripts from a
common script set (e.g.
open, saveas)
–  delet...
3-Pre and Post Processing
Automation +

DSTL
structured
testware
architecture

Dis

po
scr sabl
ipts e

execution
manual
c...
3-Pre and Post Processing
Test status – pass or fail?
•  tool cannot judge pass or fail
–  only “match” or “no match”
–  a...
3-Pre and Post Processing
Pre and Post

1

2

3

4

5

Successful Test Automation

6

Summary: key points
• 
• 

Pre- and ...
4-Scripting

Successful Test Automation

1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice
...
4-Scripting

Objectives of scripting techniques
•  implement your
testware architecture
•  to reduce costs
–  make it easi...
4-Scripting

Capture/replay = linear script
test
scripts
record
/edit

software under test

(manual) test
procedures

Test...
4-Scripting

Example captured script
captured individual
interactions
with software

duplicated actions
result in duplicat...
4-Scripting

About structured scripting
•  script library for re-used scripts
–  part of testware architecture implementat...
4-Scripting

Usable (re-usable) scripts
•  To re-use a script, need to know:
–  what does this script do?
–  what does it ...
4-Scripting

About data driven
•  test data extracted from scripts
–  placed into separate data files

•  control script r...
4-Scripting

Data-driven: example data file
Data file: TestCase1
FILE

ADD

MOVE DELETE

countries
Sweden
USA

Headings in...
4-Scripting

About keywords
•  single control script (Interactive Test Environment)
–  improvements to this benefit all te...
4-Scripting

1

2

4

5

Successful Test Automation

3
6

Scripting

Contents
Objectives of scripting techniques
Different...
4-Scripting

Domain Specific Test Language (DSTL)
•  test procedures and test definitions similar
–  both describe sequenc...
4-Scripting

Example use of keywords
Create a new account, order 2 items and check out
Firstname

Surname

Email address

...
4-Scripting

Example keyword documentation - 1
Name

Create account

Purpose

Creates a new account

Parameters

*First na...
4-Scripting

Implementing keywords
•  ways to implement keywords
–  scripting language (of a tool)
–  programming language...
4-Scripting

Execution-tool-independent framework
some tests run manually

Test
Tool

Another
Test
Tool

test
procedures
/...
Keyword-Driven Exercise
Exercise (Part 1)
Using only the keywords described below, specify two or three test cases each
co...
Keyword-Driven Exercise

Test Case

© Grove Consultants

Action

Argument

ATT090117

Argument

Page 2 of 3
Keyword-Driven Solution

Test Case
New user can
create account and
place order

Action

Argument

Argument

Create a valid...
5-Comparison

Successful Test Automation

1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice...
5-Comparison

Perverse persistence: Michael Williamson
–  testing Webmaster tools at Google (new to testing)
–  QA used Eg...
5-Comparison

General comparison guidelines
• 
• 
• 
• 
• 
• 

keep as simple as possible
well documented
standardise as m...
5-Comparison

Comparison types compared
scribble1.scp

Test script:
- test input
- comparison
instructions

dynamic compar...
5-Comparison

1

2

4

5

Successful Test Automation

3
6

Comparison

Contents
Automated test verification
Test sensitivi...
5-Comparison

Sensitive versus specific(robust) test
Test is supposed
to change only
this field

Specific test
verifies th...
5-Comparison

Using test sensitivity
•  sensitive tests:
–  few, at high level
–  breadth / sanity checking tests
–  good ...
Comparison Illustration

Comparison Illustration
The following examples illustrate some of the problems in automated compa...
Comparison Illustration
Computer matching attempt No. 1:
Expected outcome

Actual outcome

Test run number X43578

Message...
Comparison Illustration
After applying the filter:
Sort into alphabetical order:
Expected outcome

Actual outcome

Add to ...
Comparison Illustration
Here are some other filters we could apply:
Filter: After {Time} change if in the correct format t...
6-Advice

Successful Test Automation

1 Managing

2 Architecture

3 Pre- and Post

4 Scripting

5 Comparison

6 Advice

Fi...
6-Advice

Strategy exercise
•  your automation strategy / action plan
–  review your objectives for today (p1)
–  review y...
6-Advice

Dealing with high level management
•  management support
–  building good automation takes time and effort
–  se...
6-Advice

Standards and technical factors
•  standards for the testware architecture
–  where to put things
–  what to nam...
6-Advice

Information and web sites
–  www.AutomatedTestingInstitute.com
•  TestKIT Conference, Autumn, near Washington DC...
6-Advice

any more questions?
please email me!
info@DorothyGraham.co.uk
Thank you for coming today
I hope this will be use...
32

BETTER SOFTWARE

JULY/AUGUST 2009

www.StickyMinds.com
“Why automate?”

This seems such
an easy question to answer; yet many
people don’t achieve the success they
hoped for. If ...
measured by the increased use of resources and by EMTE, but should also
include a measure of the value of the
tests run, f...
is the responsibility of the developers,
not the testers or the test automators.
Finally, how much time is spent maintaini...
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Managing Successful Test Automation
Upcoming SlideShare
Loading in...5
×

Managing Successful Test Automation

1,088

Published on

Many organizations never achieve the significant benefits that are promised from automated test execution. What are the secrets to test automation success? There are no secrets, but the paths to success are not commonly understood. Dorothy Graham describes the most important automation issues that you must address, both management and technical, and helps you understand and choose the best approaches for your organization—no matter which automation tools you use. If you don’t begin with good objectives for your automation, you will set yourself up for failure later. If you don’t show return on investment (ROI) from automation, your automation efforts may be doomed, no matter how technically good they are. Join Dot to learn how to identify achievable and realistic objectives for automation, show ROI from automation, understand technical issues including testware architecture, pick up useful tips, learn what works in practice, and devise an effective automation strategy.

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,088
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
90
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Managing Successful Test Automation

  1. 1. Managing Successful Test Automation Contents Session 0: Introduction to the tutorial Objectives, What we cover (and don’t cover) today Session 1: Planning and Managing Test Automation Responsibilities Pilot project Test automation objectives (and exercise) Measures for automation Return on Investment (ROI) (and exercise) Session 2: Testware Architecture Importance of a testware architecture What needs to be organised Session 3: Pre- and Post-Processing Automating more than tests Test status Session 4: Scripting Techniques Objectives of scripting techniques Different types of scripts Domain specific test language Session 5: Automated Comparison Automated test verification Test sensitivity Comparison example Session 6: Final Advice, Q&A and Direction Strategy exercise Final advice Questions and Answers Appendix (useful stuff) That’s no reason to automate (Better Software article) Effective Automated Testing with a DSTL, Martin Gijsen Man and Machine, Jonathan Kohl (Better Software) Technical vs non-technical skills in test automation
  2. 2. 0-1 Managing Successful Test Automation Prepared and presented by Dorothy Graham www.DorothyGraham.co.uk email: info@dorothygraham.co.uk Twitter: @DorothyGraham © Dorothy Graham 2013 0-1 Objectives of this tutorial •  help you achieve better success in automation –  independent of any particular tool •  mainly management and some technical issues –  objectives for automation –  showing Return on Investment (ROI) –  importance of testware architecture –  practical tips for a few technical issues –  what works in practice (case studies) •  help you plan an effective automation strategy 0-2 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  3. 3. 0-2 Tutorial contents 1) planning & managing test automation 2) testware architecture 3) pre and post processing 4) scripting techniques 5) automated comparison 6) final advice, Q&A and direction 0-3 Shameless commercial plug Part 1: How to do automation - still relevant today, though we plan to update it at some point New book! www.DorothyGraham.co.uk info@dorothygraham.co.uk 0-4 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  4. 4. 0-3 What is today about? (and not about) •  test execution automation (not other tools) •  I will NOT cover: –  demos of tools (time, which one, expo) –  comparative tool info (expo, web) –  selecting a tool* •  at the end of the day –  understand technical and non-technical issues –  have your own automation objectives –  plan your own automation strategy * I will email you Ch 10 of the STA book on request – info@dorothygraham.co.uk 0-5 Test automation survey 2004 - 2010 –  survey by Erik van Veenendaal in Professional Tester magazine (Nov/Dec 2010) –  –  have test execution tools –  have shelfware tools 2004 29% 26% 2010 48% 28% •  most common shelfware – test execution tools (33%) –  achieving many benefits 27% 50% 0-6 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  5. 5. 0-4 Current tool adoption (Australia) •  K.J. Ross & Associates survey 2010 –  19% –  7% good benefits using tools but not anticipated ROI 26% success (partial) 74% failure! –  35% –  14% –  12% –  10% –  3% don’t have skills to adopt or maintain automation just an added burden, no time tools abandoned – too much maintenance tools bought, no resources to adopt no budget to buy or adopt tools Source: www,kjross.com.au (good webinar on automation) 0-7 About you •  your Summary and Strategy document –  where are you now with your automation? –  what are your most pressing automation problems? –  why are you here today? There is small group work throughout the day (maximum of 5 per group); introduce yourselves within your group. 0-8 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  6. 6. 1-Managing Successful Test Automation Planning & Managing Test Automation 1 Managing 2 Architecture 3 Pre- and Post 4 Scripting 5 Comparison 6 Advice 1-1 Managing Successful Test Automation 1 2 3 4 5 6 Contents Responsibilities Pilot project Test automation objectives Measures for automation Return on Investment (ROI) 1-2 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  7. 7. 1-Managing What is an automated test? •  a test! –  designed by a tester for a purpose •  test is executed –  implemented / constructed to run automatically using a tool –  could be run manually also •  who decides what tests to run? •  who decides how a test is run? 1-3 Existing perceptions of automation skills •  many books & articles don’t mention automation skills –  or assume that they must be acquired by testers •  test automation is technical in some ways •  using the test execution tool directly (script writing) •  designing the testware architecture (framework / regime) •  debugging automation problems –  this work requires technical skill –  most people now realise this (but many still don’t) See article: “Technical vs non-technical skills in test automation” presented by Dorothy Graham info@dorothygraham.co.uk 1-4 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  8. 8. 1-Managing Responsibilities Testers •  test the software –  design tests –  select tests for automation •  requires planning / negotiation Automators •  automate tests (requested by testers) •  support automated testing –  allow testers to execute tests –  help testers debug failed tests –  provide additional tools (homegrown) •  execute automated tests –  should not need detailed technical expertise •  analyse failed automated tests –  report bugs found by tests –  problems with the tests may need help from the automation team •  predict –  maintenance effort for software changes –  cost of automating new tests •  improve the automation –  more benefits, less cost 1-5 Test manager’s dilemma •  who should undertake automation work –  not all testers can automate (well) –  not all testers want to automate –  not all automators want to test! •  conflict of responsibilities –  automate tests vs. run tests manually •  get additional resources as automators? –  contractors? borrow a developer? tool vendor? 1-6 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  9. 9. 1-Managing Recent poll Testers as automators? Every tester should be able to write code Testers can be automators if they want to Testers shouldn't have to become automators Testers should never be automators 0 10 20 30 40 50 60 70 80 EuroStar Webinar, Intelligent Mistakes in Test Automation, 20 Sept 2012 1-7 Roles within the automation team •  Testware architect –  designs the overall structure for the automation •  Champion –  “sells” automation to managers and testers •  Tool specialist / toolsmith –  technical aspects, licensing, updates to the tool •  Automated test (& script) developers –  write new keyword scripts as needed –  debug automation problems 1-8 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  10. 10. 1-Managing Agile automation: Lisa Crispin –  starting point: buggy code, new functionality needed, whole team regression tests manually –  testable architecture: (open source tools) •  want unit tests automated (TDD), start with new code •  start with GUI smoke tests - regression •  business logic in middle level with FitNesse –  100% regression tests automated in one year •  selected set of smoke tests for coverage of stories –  every 6 mos, engineering sprint on the automation –  key success factors •  management support & communication •  whole team approach, celebration & refactoring 1-9 Chapter 1, pp 17-32, Experiences of Test Automation Automation and agile •  agile automation: apply agile principles to automation –  multidisciplinary team –  automation sprints –  refactor when needed •  fitting automation into agile development –  ideal: automation is part of “done” for each sprint •  Test-Driven Design = write and automate tests first –  alternative: automation in the following sprint -> •  may be better for system level tests See www.satisfice.com/articles/agileauto-paper.pdf (James Bach) presented by Dorothy Graham info@dorothygraham.co.uk 1-10 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  11. 11. 1-Managing Automation in agile/iterative development A manual testing of this release (testers) A B regression testing (automators automate the best tests) A B C run automated tests (testers) A B C D E F 1-11 Requirements for agile test framework •  Support manual and automated testing –  using the same test construction process •  Support fully manual execution at any time –  requires good naming convention for components •  Support manual + automated execution –  so test can be used before it is 100% automated •  Implement reusable objects •  Allow “stubbing” objects before GUI available Source: Dave Martin, LDSChurch.org, email presented by Dorothy Graham info@dorothygraham.co.uk 1-12 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  12. 12. 1-Managing Managing Successful Test Automation 1 2 3 4 5 6 Contents Responsibilities Pilot project Test automation objectives Measures for automation Return on Investment (ROI) 1-13 A tale of two projects: Ane Clausen –  Project 1: 5 people part-time, within test group •  no objectives, no standards, no experience, unstable •  after 6 months was closed down –  Project 2: 3 people full time, 3-month pilot •  worked on two (easy) insurance products, end to end •  1st month: learn and plan, 2nd & 3rd months: implement •  started with simple, stable, positive tests, easy to do •  close cooperation with business, developers, delivery •  weekly delivery of automated Business Process Tests –  after 6 months, automated all insurance products Chapter 6, pp 105-128, Experiences of Test Automation presented by Dorothy Graham info@dorothygraham.co.uk 1-14 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  13. 13. 1-Managing Pilot project •  reasons –  you’re unique –  many variables / unknowns at start •  benefits –  find the best way for you (best practice) –  solve problems once –  establish confidence (based on experience) –  set realistic targets •  objectives –  demonstrate tool value –  gain experience / skills in the use of the tool –  identify changes to existing test process –  set internal standards and conventions –  refine assessment of costs and achievable benefits 1-15 What to explore in the pilot •  build / implement automated tests (architecture) –  different ways to build stable tests (e.g. 10 – 20) •  maintenance –  different versions of the application –  reduce maintenance for most likely changes •  failure analysis –  support for identifying bugs –  coping with common bugs affecting many automated tests Also: naming conventions, reporting results, measurement presented by Dorothy Graham info@dorothygraham.co.uk 1-16 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  14. 14. 1-Managing After the pilot… •  having processes & standards is only the start –  30% on new process –  70% on deployment Source: Eric Van Veenendaal, successful test process improvement •  marketing, training, coaching •  feedback, focus groups, sharing what’s been done •  the (psychological) Change Equation –  change only happens if (x + y + z) > w x = dissatisfaction with the current state y = shared vision of the future z = knowledge of the steps to take to get from here to there w = psychological / emotional cost to change for this person 1-17 Managing Successful Test Automation 1 2 3 4 5 6 Contents Responsibilities Pilot project Test automation objectives Measures for automation Return on Investment (ROI) 1-18 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  15. 15. 1-Managing An automation effort •  is a project –  with goals, responsibilities, and monitoring –  but not just a project – ongoing effort is needed •  not just one effort – different projects –  when acquiring a tool – pilot project –  when anticipated benefits have not materialized –  different projects at different times •  with different objectives •  objectives are important for automation efforts –  where are we going? are we getting there? 1-19 A manual vs an automated test an automated test Effective Economic Economic first Run of an automated test an interactive (manual) test Evolvable Exemplary 1-20 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  16. 16. 1-Managing Good objectives for automation? –  run regression tests evenings and weekends –  give testers a new skill / enhance their image –  run tests tedious and error-prone if run manually –  gain confidence in the system –  reduce the number of defects found by users 1-21 Objectives Exercise 1-22 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  17. 17. Test Automation Objectives Exercise Test Automation Objectives Exercise The following are some possible test automation objectives. Evaluate each objective – is it a suitable objective for automation? If not, why not? Which are already in place in your own organisation? Possible test automation objectives Achieve faster performance for the system Good automation objective? (If not, why not) Already in place? NO – this is not an objective for test execution automation, nor is it an objective for performance testing! Performance test tools may help by giving the measurements to see whether the system is faster. Achieve good results and quick payback with no additional resources, effort or time Automate all tests Build a long-lasting automation regime that is easy to maintain Easy to add new automated tests Ensure repeatability of regression tests Ensure that we meet our release deadlines Find more bugs Find defects in less time Free testers from repeated (boring) test execution to spend more time in test design © Dorothy Graham STA1110126 Page 1 of 5
  18. 18. Test Automation Objectives Exercise Possible test automation objectives Good automation objective? (If not, why not) Already in place? Improve our testing Reduce elapsed time for testing by x% Reduce the cost and time for test design Reduce the number of test staff Run more tests Run regression tests more often Run tests every night on all PCs Achieve a positive Return on Investment in no more than <x> test interations (where x = ?) Other objectives: © Dorothy Graham STA1110126 Page 2 of 5
  19. 19. 1-Managing Reduce test execution time edit tests (maintenance) set-up execute analyse failures clear-up Manual testing Same tests automated More mature automation 1-23 Automate x% of the tests •  are your existing tests worth automating? –  if testing is in chaos, automating gives you faster chaos •  which tests to automate (first)? •  what % of manual tests should be automated? –  “100%” sounds impressive but may not be wise •  what else can be automated –  automation can do things not possible or practical in manual testing! 1-24 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  20. 20. 1-Managing Manual vs automated manual tests automated tests tests not automated yet tests not worth automating tests (& verification) not possible to do manually manual tests automated (% manual) exploratory test automation 1-25 Exploratory Test Automation - 1 •  sounds like an oxymoron? •  when you are exploring, you might say –  “that’s weird – might there be any more like that?” •  is there a way to check a result automatically? –  a heuristic oracle •  can you generate lots of different (random) inputs? •  have you got lots of computer power available? Source: Harry Robinson tutorial at CAST 2010, August 2010 presented by Dorothy Graham info@dorothygraham.co.uk 1-26 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  21. 21. 1-Managing Exploratory Test Automation - 2 •  set off lots of tests that produce something that can be checked with an automated oracle •  alerts a human when something unusual occurs •  it’s exploratory – we don’t know what we will find, we don’t have a planned route through the system •  it’s automated – a “shotgun” approach – lots of different (random) inputs •  can find a class of bug too hard to find manually –  because so many tests can be run 1-27 Success = find lots of bugs? •  tests find bugs, not automation •  automation is a mechanism for running tests •  the bug-finding ability of a test is not affected by the manner in which it is executed •  this can be a dangerous objective –  especially for regression automation! 1-28 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  22. 22. 1-Managing What is automated? most often automated likelihood of finding bugs regression tests exploratory testing 1-29 When is “find more bugs” a good objective for automation? •  objective is “fewer regression bugs missed” •  when the first run of a given test is automated –  MBT, Exploratory test automation, automated test design –  keyword-driven (e.g. users populate spreadsheet) •  find bugs in parts we wouldn’t have tested? 1-30 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  23. 23. 1-Managing Good objectives for test automation •  measurable –  EMTE (Equivalent Manual Test Effort) –  tests run, coverage (e.g. features tested) •  •  •  •  realistic and achievable short and long term regularly re-visited and revised should be different objectives for testing and for automation •  automation should support testing activities 1-31 Trying to get started: Tessa Benzie –  consultancy to start automation effort •  project, needs a champion – hired someone •  training first, something else next, etc. –  contract test manager – more consultancy •  bought a tool – now used by a couple contractors •  TM moved on, new QA manager has other priorities –  just wanting to do it isn’t enough •  needs dedicated effort •  now have “football teams” of manual testers Chapter 29, pp 535, Experiences of Test Automation presented by Dorothy Graham info@dorothygraham.co.uk 1-32 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  24. 24. 1-Managing Managing Successful Test Automation 1 2 3 4 5 6 Contents Responsibilities Pilot project Test automation objectives Measures for automation Return on Investment (ROI) 1-33 Useful measures •  a useful measure: “supports effective analysis and decision making, and that can be obtained relatively easily.” Bill Hetzel, “Making Software Measurement Work”, QED, 1993. •  easy measures may be more useful even though less accurate (e.g. car fuel economy) •  ‘useful’ depends on objectives, i.e. what you want to know 1-34 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  25. 25. 1-Managing Automation measures? •  aspects of automation –  number of times tests are run –  time to run the automated tests (& manual equiv) –  effort in automation and running automated tests •  effort to build new automated tests •  effort to run automated tests (by people) •  effort to analyse failed automated tests •  effort to maintain automated tests –  number of test failures caused by one s/w fault –  number of automation scripts 1-35 EMTE – what is it? •  Equivalent Manual Test Effort –  given a set of automated tests, –  how much effort would it take •  IF those tests were run manually •  note –  you would not actually run these tests manually –  EMTE = what you could have tested manually •  and what you did test automatically –  used to show test automation benefit 1-36 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  26. 26. 1-Managing EMTE – how does it work? a manual test Manual testing Automate the manual testing? the manual test now automated doesn’t make sense – can run them more only time to run the tests 1.5 times 1-37 EMTE – how does it work? (2) Automated testing EMTE 1-38 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  27. 27. 1-Managing EMTE example •  example –  automated tests take 2 hours –  if those same tests were run manually, 4 days •  frequency –  automated tests run every day for 2 weeks (including once at the weekend), 11 times •  calculation –  EMTE = 1-39 Examples of good (measurable) objectives for test automation •  achieve positive ROI in less than 6 test iterations –  measured by comparing to manual testing •  run most important tests using spare resource –  top 10% of usefulness rating, run out of hours •  reduce elapsed time of tool-supported activities –  measured for maintenance & failure analysis time •  improve automation support for testers –  testers rate usefulness of automation support –  how often utilities/automation features are used 1-40 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  28. 28. 1-Managing Managing Successful Test Automation 1 2 3 4 5 6 Contents Responsibilities Pilot project Test automation objectives Measures for automation Return on Investment (ROI) 1-41 Why measure automation ROI? •  to justify and confirm starting automation –  business case for purchase/investment decision, to confirm ROI has been achieved e.g. after pilot –  both compare manual vs automated testing •  to monitor on-going automation –  for increased efficiency, continuous improvement –  build time, maintenance time, failure analysis time, refactoring time •  on-going costs – what are the benefits? 1-42 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  29. 29. 1-Managing What can you show as a benefit? •  number of additional times tests are run –  how many times run manually & automated •  how long tests take to run –  execution time for manual and automated, EMTE •  how much effort to run tests –  human effort for manual testing and kicking off/ dealing with automated tests •  data variety and/or coverage –  different sets of data run manually & automated –  different parts of the system now tested 1-43 An example comparative benefits chart 80 70 60 50 40 man aut 30 20 10 0 exec speed 14 x faster times run data variety tester work 5 x more often 4 x more data 12 x less effort ROI spreadsheet – email me for a copy 1-44 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  30. 30. 1-Managing EMTE over releases EMTE per release 800 700 600 500 400 300 200 100 0 1 2 3 4 5 2708 Total “manual” hours of testing done automatically 339 Days of testing performed 1-45 Return on Investment (ROI) •  ROI = (benefit – cost) / cost –  Investment: costs can be expressed as effort •  which can be converted to money –  Return: benefits •  reduced tester time effort •  but what about things like “faster execution”, “run more often”, “greater coverage” “quicker time to market”? •  how to convert these to money? –  possible calculation based on tester time – the other things are bonuses? •  comparing to manual testing 1-46 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  31. 31. 1-Managing ROI example Human Time to run Savings per automated automated No. automated Total run runs per release savings tests Manual time testing 66 7 Average Build time for new aut Maintenance time tests 15 59 9 531 Failure Other Total ROI = analysis automation automation (Gain time time spent investment Cost) / Cost 50 35 22 122 3.4 Free ROI spreadsheet available 1-47 MBT @ ESA: Stefan Mohacsi, Armin Beer –  home-grown tool interfaced to commercial tools •  Model-Based Testing and Test Case Generation •  layers of abstraction for maintainability –  define model before software is ready •  capture and assign GUI objects later •  developers build in testability –  ROI calculations •  invest 460 hours in automation infrastructure •  break-even after 4 test cycles Chapter 9, pp 155-175, Experiences of Test Automation presented by Dorothy Graham info@dorothygraham.co.uk 1-48 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  32. 32. 1-Managing Example ROI graph using MBT 1400 1200 1000 People Hours 800 Manual hrs Automated hrs 600 400 200 0 1 2 3 4 5 6 1-49 Source: Stefan Mohacsi & Armin Beer Database testing: Henri van de Scheur –  tool developed in-house (now open source) •  agreed requirements with relevant people up front •  9 months, 4 developers in Java (right people) •  good architecture, start with quick wins –  flexible configuration, good reporting, metrics used to improve –  results: 2400 times more efficient •  from: 20 people run 40 tests on 6 platforms in 4 days •  to: 1 person runs 200 tests on 10 platforms in 1 day •  quick dev tests, nightly regression, release tests •  life cycle of automated tests •  little maintenance, machines used 24x7, better quality Chapter 2, pp 33-48, Experiences of Test Automation presented by Dorothy Graham info@dorothygraham.co.uk 1-50 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  33. 33. 1-Managing Large S Africa bank: Michael Snyman •  was project-based, too late, lessons not learned –  “our shelves were littered with tools..” •  2006: automation project, resourced, goals –  formal automation process •  ROI after 3 years –  US$4m on testing project, automation $850K –  savings $8m, ROI 900% •  20 testers for 4 weeks to 2 in 1 week –  automation ROI justified the testing project •  only initiative that was measured accurately Chapter 29, pp 562-567, Experiences of Test Automation 1-51 Example ROI graph Savings % vs Tests 100% 50% 0% 0 500 1000 1500 2000 2500 -50% -100% -150% -200% monthly weekly daily Source: Lars Wahlberg, Chapter 18 in “Experiences of Test Automation” presented by Dorothy Graham info@dorothygraham.co.uk 1-52 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  34. 34. 1-Managing Why is measuring ROI dangerous? •  focus on what’s easy to measure (tester time) •  other factors may be much more important –  time to market, greater coverage, faster testing •  defining ROI only by tester time may give the impression that the tools replace the testers –  this is dangerous –  tools replace some aspects of some of what testers do, with increased cost elsewhere –  we want a net benefit, even if hard to quantify 1-53 How important is ROI? •  “automation is an enabler for success, not a cost reduction tool” (Yoram Mizrachi*) •  many achieve lasting success without measuring ROI (depends on your context) –  need to be aware of benefits (and publicize them) *“Planning a mobile test automation strategy that works, ATI magazine, July 2012 presented by Dorothy Graham info@dorothygraham.co.uk 1-54 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  35. 35. 1-Managing Sample ‘starter kit’ for metrics for test automation (and testing) •  some measure of benefit –  e.g. EMTE or coverage •  average time to automate a test (or set of related tests) •  total effort spent on maintaining automated tests (expressed as an average per test) •  also measure testing, e.g. Defect Detection Percentage (DDP) – test effectiveness –  more info on DDP on my web site & blog 1-55 Recommendations •  don’t measure everything! •  choose three or four measures –  applicable to your most important objectives •  monitor for a few months –  see what you learn •  change measures if they don’t give useful information •  be careful with ROI if based on tester time –  may give impression tools can replace people! 1-56 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  36. 36. 1-Managing Managing Successful Test Automation 1 2 3 4 5 6 Summary: key points •  •  •  •  •  Assign responsibility for automation (and testing) Use a pilot project to explore the best ways of doing things Know your automation objectives Measure what’s important to you Show ROI from automation 1-57 Good objectives for automation? –  run regression tests evenings and weekends not a good objective, unless they are worthwhile tests! –  give testers a new skill / enhance their image not a good objective, could be a useful by-product –  run tests tedious and error-prone if run manually good objective –  gain confidence in the system an objective for testing, but automated regression tests help achieve it –  reduce the number of defects found by users not a good objective for automation, good objective for testing! 1-58 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  37. 37. Test Automation Objectives Solution Test Automation Objectives Solution We have given some ideas as to which objectives are good and why the others are not. Good automation objective? (If not, why not) Possible test automation objectives Achieve faster performance for the system NO – this is not an objective for test execution automation, nor is it an objective for performance testing! Performance test tools may help by giving the measurements to see whether the system is faster. Achieve good results and quick payback with no additional resources, effort or time NO – this is totally unrealistic – expecting a miracle with no investment! Automate all tests NO – automating ALL tests is not realistic nor sensible. Automate only those tests that are worth automating. Build a long-lasting automation regime that is easy to maintain YES – this is an excellent objective for test automation, and it is measurable. Easy to add new automated tests YES. with a good automation regime, it can be easier to add a new automated test than to run that test manually. Ensure repeatability of regression tests YES. The tools will run the same test in the same way every time. Ensure that we meet our release deadlines NO. Automation may help to run some tests that are required before release, but there are many more factors that go into a release decision. Find more bugs NO. Automation just runs tests. It is the tests that find the bugs, whether they are run manually or are automated. Find defects in less time Not really. Some types of defects (regression bugs) will be found more quickly by automated tests, but it may actually take longer to analyse the failures found. Free testers from repeated (boring) test YES. This is a good objective for test execution execution to spend more time in test automation. design © Dorothy Graham STA110126 Page 3 of 5
  38. 38. Test Automation Objectives Solution Good automation objective? (If not, why not) Possible test automation objectives Improve our testing NO. Better testing practices and better use of techniques will improve testing. Reduce elapsed time for testing by x% NO. Elapsed time depends on many factors, and not much on whether tests are automated (see further explanation in the slides). Reduce the cost and time for test design NO. Test design is independent from automation – the time spent in design is not affected by how those tests are executed. Reduce the number of test staff NO. You will need more staff to implement the automation, not less. It can make existing staff more productive by spending more time on test design. Run more tests YES but only long term. Short term, you may actually run fewer tests because of the effort taken to automate them. Run regression tests more often YES – this is what the test execution tools do best. Run tests every night on all PCs NO. It may look impressive, but what tests are being run? Are they useful? If not, this is a waste of electricity. Achieve a positive Return on Investment in no more than <6> test interations YES. This is a good objective, if the number of iterations is a reasonable number (e.g. 6). Other objectives: © Dorothy Graham STA110126 Page 4 of 5
  39. 39. Test Automation Objectives Solution Test Automation Objectives: Selection and Measurement On this page, record the test objectives that would be most appropriate for your organisation (and why), and how you will measure them (what to measure and how to measure it). I suggest that you include at least one about showing Return on Investment. If you currently have automation objectives in place in your organisation that are not good ones, make sure that they are removed and replaced by the better ones below! Proposed test automation objective (with justification) What to measure and how to measure it Add any comments or thoughts here or on the back of this page. © Dorothy Graham STA110126 Page 5 of 5
  40. 40. 2-Architecture Successful Test Automation Testware Architecture 1 Managing 2 Architecture 3 Pre- and Post 4 Scripting 5 Comparison 6 Advice Ref. Chapter 5: Testware Architecture “Software Test Automation” 2-1 Architecture 1 2 3 4 5 Successful Test Automation 6 Contents Importance of a testware architecture What needs to be organised 2-2 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  41. 41. 2-Architecture Testware architecture testware'architecture' Testers'' write'tests'(in'DSTL)' abstraction here: easier to write automated tests -> widely used High Level Keywords structured' testware' Test Automator(s) Structured Scripts Test'Execu/on'Tool' runs'scripts' abstraction here: easier to maintain, and change tools -> long life 2-3 Architecture – abstraction levels •  most critical factor for success –  worst: close ties between scripts, tool & tester •  separate testers’ view from technical aspects –  so testers don’t need tool knowledge •  for widespread use of automation •  scripting techniques address this •  separate tests from the tool – modular design –  likely changes confined to one / few module(s) –  re-use of automation functions –  for minimal maintenance and long-lived automation 2-4 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  42. 42. 2-Architecture Easy way out: use the tool’s architecture •  tool will have its own way of organising tests –  where to put things (for the convenience of the tool!) –  will “lock you in” to that tool – good for vendors! •  a better way (gives independence from tools) –  organise your tests to suit you –  as part of pre-processing, copy files to where the tool needs (expects) to find them –  as part of post-processing, copy back to where you want things to live 2-5 Tool-specific script ratio Testers'' Testers'' Not Toolspecific Tool-specific scripts Test'Execu/on'Tool' High maintenance and/or tooldependence Test'Execu/on'Tool' 2-6 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  43. 43. 2-Architecture Learning is incremental: Molly Mahai –  book learning – knew about investment, not replace people, don’t automate everything, etc. –  set up good architecture? books not enough –  picked something to get started •  after a while, realised limitations •  too many projects, library cumbersome –  re-designed architecture, moved things around –  didn’t know what we needed till we experienced the problems for ourselves •  like trying to educate a teenager 2-7 Chapter 29, pp 527-528, Experiences of Test Automation Architecture 1 2 3 4 5 Successful Test Automation 6 Contents Importance of a testware architecture What needs to be organised 2-8 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  44. 44. 2-Architecture A test for you •  show me one of your automated tests running –  how long will it take before it runs? •  typical problems –  fails: forgot a file, couldn’t find a called script –  can’t do it (yet): •  Joe knows how but he’s out, •  environment not right, •  haven’t run in a while, •  don’t know what files need to be set up for this script •  why not: run up control script, select test, GO 2-9 Key issues •  scale –  the number of scripts, data files, results files, benchmark files, etc. will be large and growing •  shared scripts and data –  efficient automation demands reuse of scripts and data through sharing, not multiple copies •  multiple versions –  as the software changes so too will some tests but the old tests may still be required •  multiple environments / platforms 2-10 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  45. 45. 2-Architecture Terms - Testware artefacts Testware Test Materials Test Results Products inputs By-Products scripts doc (specifications) data env utilities logs actual results expected results status differences differences summary summary 2-11 Testware for example test case open.scp saveas.scp Shared Shared script Test script: script - test input countries.scp log.txt Log countries.dcm Initial Document testspec.txt Test Specification presented by Dorothy Graham info@dorothygraham.co.uk Scribble Edited Document countries2.dcm Expected Output countries2.dcm Compare diffs.txt Differences 2-12 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  46. 46. 2-Architecture Testware by type Testware Test Materials countries.scp open.scp saveas.scp testdef.txt countries.dcm Test Results Products By-Products log.txt status.txt countries2.dcm countries2.dcm diff.txt 2-13 Benefits of standard approach •  tools can assume knowledge (architecture) –  they need less information; are easier to use; fewer errors will be made •  can automate many tasks –  checking (completeness, interdependencies); documentation (summaries, reports); browsing •  portability of tests –  between people, projects, organisations, etc. •  shorter learning curve 2-14 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  47. 47. 2-Architecture Testware Sets •  Testware Set: –  a collection of testware artifacts –  four types: •  Test Set - one or more test cases •  Script Set - scripts used by two or more Test Sets •  Data Set - data files used by two or more Test Sets •  Utility Set - utilities used by two or more Test Sets –  good software practice: look for what is common, and keep it in only one place! –  Keep your testware DRY! 2-15 Testware library –  repository of master versions of all Testware Sets –  uncategorised scripts worse than no scripts –  Onaral & Turkmen –  CM is critical –  “If it takes too long to update your test library, automation introduces delay instead of adding efficiency” –  Linda Hayes, AST magazine, Sept 2010 presented by Dorothy Graham info@dorothygraham.co.uk d_ScribbleTypical d_ScribbleTypical d_ScribbleVolume s_Logging s_ScribbleDocument s_ScribbleDocument s_ScribbleDocument s_ScribbleNavigate t_ScribbleBreadth t_ScribbleCheck t_ScribbleFormat t_ScribbleList t_ScribbleList t_ScribbleList t_ScribblePrint t_ScribbleSave t_ScribbleTextEdit t_ScribbleTextEdit u_ScribbleFilters u_GeneralCompare v1 v2 v1 v1 v1 v2 v3 v1 v1 v1 v1 v1 v2 v3 v1 v1 v1 v2 v1 v1 Version numbers 2-16 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  48. 48. 2-Architecture Separate test results Test suite Software under test Test results 2-17 Incremental approach: Ursula Friede –  large insurance application –  first attempt failed •  no structure (architecture), data in scripts –  four phases (unplanned) •  parameterized (dates, claim numbers, etc) •  parameters stored in single database for all scripts •  improved error handling (non-fatal unexpected events) •  automatic system restart –  benefits: saved 200 man-days per test cycle •  €120,000! Chapter 23, pp 437-445, Experiences of Test Automation presented by Dorothy Graham info@dorothygraham.co.uk 2-18 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  49. 49. 2-Architecture Architecture 1 2 3 4 5 Successful Test Automation 6 Summary: key points •  •  •  Structure your automation testware to suit you Testware comprises many files, etc. which need to be given a home Use good software development standards 2-19 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  50. 50. 3-Pre and Post Processing Successful Test Automation Pre- and Post-Processing 1 Managing 2 Architecture 3 Pre- and Post 4 Scripting 5 Comparison 6 Advice Ref. Chapter 6: Automating Pre- and Post-Processing “Software Test Automation” 3-1 Pre and Post 1 2 3 4 5 Successful Test Automation 6 Contents Automating more than tests Test status 3-2 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  51. 51. 3-Pre and Post Processing What is pre- and post-processing? •  Pre-processing –  automation of setup tasks necessary to fulfil test case prerequisites •  Post-processing –  automation of post-execution tasks necessary to complete verification and house-work •  These terms are useful because: –  there are lots of tasks, they come in packs, many are the same, and they can be easily automated 3-3 Automated tests/automated testing Automated tests Select / identify test cases to run Set-up test environment: •  create test environment •  load test data Repeat for each test case: •  set-up test pre-requisites •  execute •  compare results •  log results •  analyse test failures •  report defect(s) •  clear-up after test case Clear-up test environment: •  delete unwanted data •  save important data Summarise results Manual process presented by Dorothy Graham info@dorothygraham.co.uk Automated testing Select / identify test cases to run Set-up test environment: •  create test environment •  load test data Repeat for each test case: •  set-up test pre-requisites •  execute •  compare results •  log results •  clear-up after test case Clear-up test environment: •  delete unwanted data •  save important data Summarise results Analyse test failures Report defects Automated process 3-4 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  52. 52. 3-Pre and Post Processing Examples •  pre-processing –  copy scripts from a common script set (e.g. open, saveas) –  delete files that shouldn’t exist when test starts –  set up data in files –  copy files to where the tool expects to find them –  save normal default file and rename the test file to the default (for this test) •  post-processing –  copy results to where the comparison process expects to find them –  delete actual results if they match expected (or archive if required) –  rename a file back to its normal default 3-5 Outside the box: Jonathan Kohl –  task automation (throw-away scripts) •  entering data sets to 2 browsers (verify by watching) •  install builds, copy test data –  support manual exploratory testing –  testing under the GUI to the database (“side door”) –  don’t believe everything you see •  1000s of automated tests pass too quickly •  monitoring tools to see what was happening •  “if there’s no error message, it must be ok” –  defects didn’t make it to the test harness –  overloaded system ignored data that was wrong Chapter 19, pp 355-373, Experiences of Test Automation presented by Dorothy Graham info@dorothygraham.co.uk 3-6 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  53. 53. 3-Pre and Post Processing Automation + DSTL structured testware architecture Dis po scr sabl ipts e execution manual comparison testing s litie d Uti ta loa da eg loosen your oracles ost & p ing Pre ess c pro traditional test automation Me tric e.g s EM . TE ETA, monkeys Pre and Post 1 2 5 Successful Test Automation 3 4 3-7 6 Contents Automating more than tests Test status 3-8 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  54. 54. 3-Pre and Post Processing Test status – pass or fail? •  tool cannot judge pass or fail –  only “match” or “no match” –  assumption: expected results are correct •  when a test fails (i.e. the software fails) –  need to analyse the failure •  true failure? write up bug report •  test fault? fix the test (e.g. expected result) •  known bug or failure affecting many automated tests? –  this can eat a lot of time in automated testing –  solution: additional test statuses 3-9 Test statuses for automation Compare to (true) expected outcome expected fail outcome don’t know / missing No differences found Differences found Pass Fail Expected Fail Unknown Unknown Unknown •  other possible additional test statuses –  environment problem (e.g. network down, timeouts) –  set-up problems (files missing) –  things that affect the tests that aren’t related to the assessment of matching expected results –  test needs to be changed but not done yet 3-10 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  55. 55. 3-Pre and Post Processing Pre and Post 1 2 3 4 5 Successful Test Automation 6 Summary: key points •  •  Pre- and post processing to automate setup and clear-up tasks Test status is more than just pass / fail 3-11 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  56. 56. 4-Scripting Successful Test Automation 1 Managing 2 Architecture 3 Pre- and Post 4 Scripting 5 Comparison 6 Advice Scripting Techniques Ref. Chapter 3: Scripting Techniques “Software Test Automation” 4-1 1 2 4 5 Successful Test Automation 3 6 Scripting Contents Objectives of scripting techniques Different types of scripts Domain specific test language 4-2 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  57. 57. 4-Scripting Objectives of scripting techniques •  implement your testware architecture •  to reduce costs –  make it easier to build automated tests •  avoid duplication •  greater return on investment –  better testing support –  greater portability •  environments & hardware platforms •  enhance capabilities –  avoid excessive maintenance costs •  greater reuse of functional, modular scripting –  achieve more testing for same (or less) effort •  testing beyond traditional manual approaches best achieved by keywords or DSTL 4-3 1 2 4 5 Successful Test Automation 3 6 Scripting Contents Objectives of scripting techniques Different types of scripts Domain specific test language 4-4 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  58. 58. 4-Scripting Capture/replay = linear script test scripts record /edit software under test (manual) test procedures Test Tool instructions and test data testers automators 4-5 About capture/replay •  fair amount of effort to produce scripts –  similar for each test procedure –  subsequent editing may also be necessary •  dominated by maintenance costs –  scripts are exact mould for software –  one software change can mean many scripts change –  script is linear – sequence of every step 4-6 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  59. 59. 4-Scripting Example captured script captured individual interactions with software duplicated actions result in duplicated instructions FocusOn ‘Open File’ Type ‘countries’ LeftMouseClick ‘Open’ FocusOn ‘Scribble’ SelectOption ‘List/Add Item’ FocusOn ‘Add Item’ Type ‘Sweden’ LeftMouseClick ‘OK’ FocusOn ‘Scribble’ SelectOption ‘List/Add Item’ FocusOn ‘Add Item’ Type ‘USA’ LeftMouseClick ‘OK’ FocusOn ‘Scribble’ SelectOption ‘List/Move Item’ duplicated actions result in duplicated instructions 4-7 Structured scripting low-level “how to” instructions create Test Tool testers script library automators software under test high-level instructions and test data test (manual) test scripts procedures 4-8 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  60. 60. 4-Scripting About structured scripting •  script library for re-used scripts –  part of testware architecture implementation •  shared scripts interface to software under test •  all other scripts interface to shared scripts •  reduced costs –  maintenance •  fewer scripts affected by software changes –  build •  individual control scripts are smaller and easier to read (a ‘higher’ level language is used) 4-9 Structured scripting example test script LeftMouseDouble ‘Scribble’ Call OpenFile(‘countries’) Call AddItem(‘Sweden’) Call AddItem(‘USA’) Call MoveItem(4,1) Call AddItem(‘Norway’) Call DeleteItem(2) Call DeleteItem(7) FocusOn ‘Delete Error’ LeftMouseClick ‘OK’ FocusOn ‘Scribble’ Call CloseSaveAs(‘countries2’) SelectOption ‘File/Exit’ shared scripts OpenFile DeleteItem :::::::::::::::::::::: AddItem :::::::::::::::::::::: :::::::::::::::::::::: MoveItem :::::::::::::::::::::: CloseSaveAs :::::::::::::::::::::: 4-10 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  61. 61. 4-Scripting Usable (re-usable) scripts •  To re-use a script, need to know: –  what does this script do? –  what does it need? –  what does it deliver? –  what state when it starts? –  what state when it finishes? •  Information in a standard place for every script –  can search for the answers to these questions 4-11 Data driven test data data files control scripts create Test Tool create testers script library automators low-level “how to” instructions software under test (manual) test procedures high-level instructions 4-12 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  62. 62. 4-Scripting About data driven •  test data extracted from scripts –  placed into separate data files •  control script reads data from data file –  one script implements several tests by reading different data files (reduces script maintenance per test) •  reduced build cost –  faster and easier to automate similar test procedures –  many test variations using different data •  multiple control scripts required –  one for each type of test (with varying data) 4-13 Data-driven example OpenDataFile(TESTCASEn) ReadDataFile(RECORD) Data file: TestCase1 FILE ADD MOVE DELETE For each record ReadDataFile(RECORD) Case (Column(RECORD)) countries Sweden Data file: TestCase2 USA FILE Europe Norway ADD 4,1 France Germany MOVE DELETE 2 7 1,3 2,2 5,3 FILE: OpenFile(INPUTFILE) ADD: AddItem(ITEM) 1 presented by Dorothy Graham info@dorothygraham.co.uk Control script For each TESTCASE MOVE: MoveItem(FROM, TO) DELETE: DeleteItem(ITEM) ….. Next record Next TESTCASE 4-14 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  63. 63. 4-Scripting Data-driven: example data file Data file: TestCase1 FILE ADD MOVE DELETE countries Sweden USA Headings indicate actions - but may not be read by a control script (they could be just “comments”) 4,1 Norway Position represents data for the action (data in the wrong column will cause problems) 2 7 Tests are not identical – first one has 3 countries, 2 moves, next test had 2 countries, 3 moves. The test data drives the test for things it can deal with – the script responds to data in a fixed position in the data file. 4-15 Keywords (basic) single control script: “interpreter” / ITE control script create Test Tool test definitions testers presented by Dorothy Graham info@dorothygraham.co.uk script library automators software under test high-level instructions and test data (manual) test procedures low-level “how to” instructions and keywords 4-16 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  64. 64. 4-Scripting About keywords •  single control script (Interactive Test Environment) –  improvements to this benefit all tests (ROI) –  extracts high-level instructions from scripts •  ‘test definition’ Unit test: calculate one interest payment –  independent of tool scripting language –  a language tailored to testers’ requirements System test: summarise interest for one customer •  software design •  application domain •  business processes Acceptance test: end of day run, all interest payments •  more tests, fewer scripts 4-17 Comparison of data files data-driven approach FILE ADD MOVE DELETE SAVE Europe France Italy 1,3 2,2 1 5,2 Test2 which is easier to read/understand? presented by Dorothy Graham info@dorothygraham.co.uk keyword approach ScribbleOpen Europe AddToList France Italy MoveItem 1 to 3 MoveItem 2 to 2 DeleteItem 1 MoveItem 5 to 2 SaveAs Test2 what happens when the test becomes large and complex? this looks more like a test 4-18 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  65. 65. 4-Scripting 1 2 4 5 Successful Test Automation 3 6 Scripting Contents Objectives of scripting techniques Different types of scripts Domain specific test language 4-19 Merged test procedure/test/ definition Advanced keyword-driven action words single control script: “interpreter” / ITE control script create Test Tool test test procedures definitions /definitions / language for testers presented by Dorothy Graham info@dorothygraham.co.uk script library low-level “how to” instructions and keyword scripts software under test high-level instructions and test data (manual) test procedures 4-20 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  66. 66. 4-Scripting Domain Specific Test Language (DSTL) •  test procedures and test definitions similar –  both describe sequences of test cases •  giving test inputs and expected results •  combine into one document –  can include all test information –  avoids extra ‘translation’ step –  testers specify tests regardless manual/automated –  automators implement required keywords 4-21 Keywords in the test definition language •  multiple levels of keywords possible –  high level for business functionality –  low level for component testing •  composite keywords –  define keywords as sequence of other keywords –  gives greater flexibility (testers can define composite keywords) but risk of chaos •  format –  freeform, structured, or standard notation •  (e.g. XML) 4-22 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  67. 67. 4-Scripting Example use of keywords Create a new account, order 2 items and check out Firstname Surname Email address Password Edward Brown ebrown@gmail.com apssowdr Item Num Items Check Price for Items Order Item 1579 3 15.30 Order Item 2598 Create Account 12.99 Checkout 4-23 Documenting keywords Name Purpose Parameters Preconditions Postconditions Error conditions Example Name for this keyword What this keyword does Any inputs needed, outputs produced What needs to be true before using it, where valid What will be true after it finishes What errors it copes with, what is returned An example of the use of the keyword Source: Martin Gijsen. See also Hans Buwalda book & articles presented by Dorothy Graham info@dorothygraham.co.uk 4-24 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  68. 68. 4-Scripting Example keyword documentation - 1 Name Create account Purpose Creates a new account Parameters *First name: 2 to 32 characters *Last name: 2 to 32 characters *Email address: also serves as account id *Password: 4 - 32 characters Pre-conditions Account doesn't exist for this person Post-conditions Account created (including email confirmation) order Screen displayed Error conditions Example (see example) *mandatory 4-25 Example keyword documentation - 2 Name Purpose Parameters Pre-conditions Post-conditions Error conditions Example Order item Orders items from the online supplier *Item number: 1000 to 9999, in catalogue Number of items wanted: 1 to Max-for-item. If blank, assumes 1 Valid account logged in Items in stock (at least one) Prices available for all items in stock Total for all items ordered is available (for checkout) Number of available items decreased by number ordered Item(s) not in stock (see example) *mandatory presented by Dorothy Graham info@dorothygraham.co.uk 4-26 © Dorothy Graham 2013 www.DorothyGraham.co.uk
  69. 69. 4-Scripting Implementing keywords •  ways to implement keywords –  scripting language (of a tool) –  programming language (e.g. Java) –  use what your developers are familiar with! •  ways of creating an architecture to support a DSTL –  commercial, open source or home-grown framework –  spreadsheet or database for test descriptions 4-27 Tools / frameworks •  commercial tools –  ATRT, Axe, Certify, eCATT, FASTBoX, Liberation, Ranorex, TestComplete, TestDrive, Tosca Testsuite •  open source –  FitNesse, JET, Open2Test, Power Tools, Rasta, Robot Framework, SAFS, STAF, TAF Core •  I can email you my Tool List –  test execution and framework tools –  info@dorothygraham.co.uk 4-28 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  70. 70. 4-Scripting Execution-tool-independent framework some tests run manually Test Tool Another Test Tool test procedures /definitions tool independent 1 2 5 4-29 tool dependent Successful Test Automation 3 4 sut tool independent scripting language framework software under test software under test script libraries 6 Scripting Summary •  Objectives of good scripting –  to reduce costs and enhance capabilities •  Many types of scripting –  e.g. capture playback, structured, data-driven, keyword •  Keyword/DSTL the most sophisticated –  yields significant benefits for investment •  Increased tester productivity –  tailored front end, test language, tool independence 4-30 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  71. 71. Keyword-Driven Exercise Exercise (Part 1) Using only the keywords described below, specify two or three test cases each comprising a short sequence of keywords for an online grocery store. Keywords The following keywords are available. Keyword Arguments Basket should contain “n items” where n is the number of items in the basket Checkout using credit card Card type, Card number, Security code, Expiry date Create a valid user Username, Password, Address, Postcode Last order status should be One of: “Delivery scheduled”, “Delivery due”, “Delivered” Login Username, Password Logout Put items in basket List of items Example The following is an example of short test case. Test Case Existing user can place an order Action Argument Argument Login Joe Shopper Passwurd Put items in basket Carrots Cabbage … Potatoes Basket should contain 3 items Checkout using credit card Mastercard 1234 5678 0123 … 123 01-11 Logout Note that “…” in the Action column indicate a continuation line, i.e. row contains more arguments for the previous keyword. Multiple continuation lines are allowed. Exercise (Part 2) Design 2 new additional keywords suitable for testing this online grocery store and specify one or two additional test cases that use your new keywords. © Grove Consultants ATT090117 Page 1 of 3
  72. 72. Keyword-Driven Exercise Test Case © Grove Consultants Action Argument ATT090117 Argument Page 2 of 3
  73. 73. Keyword-Driven Solution Test Case New user can create account and place order Action Argument Argument Create a valid user Joe Shopper Passwurd … 6 Lower Road Worcester WR1 2AB Put items in basket Cornflakes Milk … Tea bags Coffee … Sugar Biscuits Basket should contain 6 items Checkout using credit card Mastercard 1234 5678 0123 … 123 01-11 Login Joe Shopper Passwurd Last order status should be Delivery scheduled Logout Existing user can query order Logout © Grove Consultants ATT090117 Page 3 of 3
  74. 74. 5-Comparison Successful Test Automation 1 Managing 2 Architecture 3 Pre- and Post 4 Scripting 5 Comparison 6 Advice Automated Comparison Ref. Chapter 4: Automated Comparison “Software Test Automation” 5-1 1 2 4 5 Successful Test Automation 3 6 Comparison Contents Automated test verification Test sensitivity 5-2 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  75. 75. 5-Comparison Perverse persistence: Michael Williamson –  testing Webmaster tools at Google (new to testing) –  QA used Eggplant (image processing tool) –  new UI broke existing automation –  automate 4 or 5 functions –  comparing bitmap images – inaccurate and slow –  testers had to do automation maintenance •  not worth developers learning tool’s language –  after 6 months, went for more appropriate tools –  QA didn’t use the automation, tested manually! •  tool was just running in the background 5-3 Chapter 17, pp 321-338, Experiences of Test Automation Checking versus testing –  checking confirms that things are as we think •  e.g. check that the code still works as before –  testing is a process of exploration, discovery, investigation and learning •  e.g. what are the threats to value to stakeholders, give information –  checks are machine-decidable •  if it’s automated, it’s probably a check –  tests require sapience •  including “are the checks good enough” Source: Michael Bolton, www.developsense.com/blog/2009/08/testing-vs-checking/ 5-4 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  76. 76. 5-Comparison General comparison guidelines •  •  •  •  •  •  keep as simple as possible well documented standardise as much as possible avoid bit-map comparison poor comparisons destroy good tests divide and conquer: –  use a multi-pass strategy –  compare different aspects in each pass 5-5 Two types of comparison •  dynamic comparison –  done during test execution –  performed by the test tool –  can be used to direct the progress of the test •  e.g. if this fails, do that instead –  fail information written to test log (usually) •  post-execution comparison –  done after the test execution has completed –  good for comparing files or databases –  can be separated from test execution –  can have different levels of comparison •  e.g. compare in detail if all high level comparisons pass 5-6 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  77. 77. 5-Comparison Comparison types compared scribble1.scp Test script: - test input - comparison instructions dynamic comparison Error message as expected? log.txt Log countries.dcm Initial Document Scribble countries2.dcm Edited Document countries2.dcm Expected Output Compare post-execution comparison diffs.txt Differences 5-7 Comparison process •  few tools for post-execution comparison •  simple comparators come with operating systems but do not have pattern matching –  e.g, Unix ‘diff’, Windows ‘UltraCompare’ •  text manipulation tools widely available –  sed, awk, grep, egrep, Perl, Tcl, Python •  use pattern matching tools with a simple comparator to make a ‘comparison process’ •  use masks and filters for efficiency 5-8 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  78. 78. 5-Comparison 1 2 4 5 Successful Test Automation 3 6 Comparison Contents Automated test verification Test sensitivity 5-9 Test sensitivity •  the more data there is available: –  the easier it is to analyse faults and debug •  the more data that is compared: –  the more sensitive the test •  the more sensitive a test: –  the more likely it is to fail –  (this can be both a good and bad thing) 5-10 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  79. 79. 5-Comparison Sensitive versus specific(robust) test Test is supposed to change only this field Specific test verifies this field only Test outcome Unexpected change occurs Sensitive test verifies the entire outcome 5-11 Too much sensitivity = redundancy Three tests, each changes a different field If all tests are specific, the unexpected change is missed Test outcome Unexpected change occurs for every test If all tests are sensitive, they all show the unexpected change 5-12 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  80. 80. 5-Comparison Using test sensitivity •  sensitive tests: –  few, at high level –  breadth / sanity checking tests –  good for regression / maintenance •  specific/robust tests: –  many, at detailed level –  focus on specific aspects –  good for development 1 2 3 4 5 A good test automation strategy will plan a combination of sensitive and specific tests 5-13 Successful Test Automation 6 Comparison Summary: key points •  •  •  Balance dynamic and post-execution comparison Balance sensitive and specific tests Use masking and filters to adjust sensitivity 5-14 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  81. 81. Comparison Illustration Comparison Illustration The following examples illustrate some of the problems in automated comparison. Below are the expected outcome and the actual outcome for a test. Do they match? I.e. does the test pass? The expected results were taken from a previous test that was confirmed as correct. Test objective: Ensure that the core processing is correct, i.e. the correct error messages are produced, the progress of the order is correctly logged, and the right number of things have been ordered. It doesn’t matter when the test is run, what specific things are ordered, or the exact number assigned to the order. The total should be correct for the things that have been ordered, however. Test input (high level conditions): Log on as existing customer (software gets the first one in the list of existing customers). Order one thing that is not in stock (from a pre-defined table.) Order three things that are in stock (3 things at random locations in the stock database). Total the cost of items ordered. Expected outcome Order number X43578 Date: 24 Jan 2008 Time 11:02 Message: “Log-on accepted, J. Smith” Message: “Add to basket, 1 toaster” Message: “Out of stock, mountain bike” Message: “Add to basket, 1 computer mouse” Message: “Add to basket, 1 pedometer” Add to total: £15.95 Add to total: £14.99 Add to total: £4.95 Message: “Check out J Smith” Message: “Total due is £35.89” Actual outcome Message: “Log-on accepted, M. Jones” Order number X43604 Message: “Out of stock, anti-gravity boots” Message: “Add to basket, 1 nose stud” Add to total: £4.99 Message: “Add to basket, 1 car booster seat” Add to total: £69.95 Message: “Add to basket, 1 pedometer” Add to total: £4.95 Message: “Total due is £79.99” Message: “Check out J Smith” Date: 18 Feb 2008 Time 13:08 Has this test passed or failed? How long did it take you to decide? Would automated comparison be better / quicker? © Dorothy Graham STA110126 Page 1 of 4
  82. 82. Comparison Illustration Computer matching attempt No. 1: Expected outcome Actual outcome Test run number X43578 Message: “Log-on accepted, M. Jones” Comparison result FAIL Date: 24 Jan 2008 Test run number X43604 FAIL Time 11:02 Message: “Out of stock, anti-gravity boots” FAIL Message: “Log-on accepted, J. Smith” Message: “Add to basket, 1 nose stud” FAIL Message: “Add to basket, 1 toaster” Add to total: £4.99 FAIL Message: “Out of stock, mountain bike” Message: “Add to basket, 1 car booster seat” FAIL Message: “Add to basket, 1 computer mouse” Add to total: “£69.95” FAIL Message: “Add to basket, 1 pedometer” Message: “Add to basket, 1 pedometer” PASS Add to total: £15.95 Add to total: £4.95 FAIL Add to total: £14.99 Message: “Total due is £79.99” FAIL Add to total: £4.95 Message: “Check out J Smith” FAIL Message: “Check out J Smith” Date: 18 Feb 2008 FAIL Message: “Total due is £35.89 Time 13:08 FAIL Well, this isn’t terribly helpful, is it! The only thing that passed was just a coincidence (ordering the same item in the two tests). What filters could we use to get the computer’s simple comparison to give a more meaningful result? We can see that some things would have a better chance of matching if the results were in the same order. For example, Date and Time come last in the actual outcome but nearly first in our expected outcome. Let’s apply a filter to both sets of results to put the individual lines in alphabetical order. Note that this now destroys the sequence of steps as a test (this may be OK for the purposes of comparing the individual steps). © Dorothy Graham STA110126 Page 2 of 4
  83. 83. Comparison Illustration After applying the filter: Sort into alphabetical order: Expected outcome Actual outcome Add to total: £14.95 Add to total: £4.95 Comparison result FAIL Add to total: £15.95 Add to total: £4.99 FAIL Add to total: £4.99 Add to total: £69.95 FAIL Date: 24 Jan 2008 Date: 18 Feb 2008 FAIL Message: “Add to basket, 1 computer mouse” Message: “Add to basket, 1 car booster seat” FAIL Message: “Add to basket, 1 pedometer” Message: “Add to basket, 1 nose stud” FAIL Message: “Add to basket, 1 toaster” Message: “Add to basket, 1 pedometer” FAIL Message: “Check out J Smith” Message: “Check out J Smith” PASS Message: “Log-on accepted, J. Smith” Message: “Log-on accepted, M. Jones” FAIL Message: “Out of stock, mountain bike” Message: “Out of stock, anti-gravity boots” FAIL Message: “Total due is £35.89 Message: “Total due is £79.99” FAIL Test run number X43578 Test run number X43604 FAIL Time 11:02 Time 13:08 FAIL We don’t seem to be much further forward now, although a different item has now passed! There are two items that would have matched if they had been in a row higher or lower. There is also something very strange about this result – the thing that has passed is ‘Message: “Check out J Smith”’. But we logged in as two different people! So the one thing shown as a Pass is actually a bug!! What other filters could we apply to get a better result from automated comparison? © Dorothy Graham STA110126 Page 3 of 4
  84. 84. Comparison Illustration Here are some other filters we could apply: Filter: After {Time} change if in the correct format to <nn:nn> After {Test run number} change if in the correct format to <Xnnnnn> After {Date} change if in the correct format to <nn Aaa nn> After {Message: “Add to basket, } change to <item> all text before {”} After {Message: “Out of stock, } change to <item> all text before {“}” Here is our result after applying these filters: Expected outcome Actual outcome Add to total: £14.95 Add to total: £4.95 Comparison result FAIL Add to total: £15.95 Add to total: £4.99 FAIL Add to total: £4.99 Add to total: £69.95 FAIL Date: <nn Aaa nn> Date: <nn Aaa nn> Message: “Add to basket, <item>” Message: “Add to basket, <item>” Message: “Add to basket, <item>” Message: “Add to basket, <item>” PASS PASS PASS Message: “Add to basket, <item>” Message: “Add to basket, <item>” Message: “Check out J Smith” Message: “Check out J Smith” PASS PASS Message: “Log-on accepted, J. Smith” Message: “Log-on accepted, M. Jones” FAIL Message: “Out of stock, <item>” Message: “Out of stock, <item>” PASS Message: “Total due is £35.89 Message: “Total due is £79.99” FAIL Test run number <Xnnnnn> Test run number <Xnnnnn> Time <nn:nn> Time <nn:nn> PASS PASS We were not interested in checking what items were ordered – if we aren’t interested in the amounts either, we could apply further filters to the amounts? Note that there are two aspects where simple comparisons such as those shown here are not really adequate. We want to know that the total for the amount ordered is correct (by the way, the other bug is that the total for the Actual outcome should be £79.89). In order to check this automatically, we would need to define a variable, add the individual item amounts to it, and check that its total was the same as that calculated by the application. This can certainly be done, but it takes additional effort to program it into the comparison process! Similarly, we could store the name used at log-on and compare it to the name used at check out, and this would find that bug (but also requires additional work). Conclusion: automated comparison is not trivial! © Dorothy Graham STA110126 Page 4 of 4
  85. 85. 6-Advice Successful Test Automation 1 Managing 2 Architecture 3 Pre- and Post 4 Scripting 5 Comparison 6 Advice Final Advice and Direction 6-1 What next? •  we have looked at a number of ideas about test automation today •  what is your situation? –  what are the most important things for you now? –  where do you want to go? –  how will you get there? •  make a start on your test automation strategy now –  adapt it to your own situation tomorrow 6-2 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  86. 86. 6-Advice Strategy exercise •  your automation strategy / action plan –  review your objectives for today (p1) –  review your “take-aways” so far (p2) –  identify the top 3 changes you want to make to your automation (top of p3) –  note your plans now on p3 •  discuss with your neighbour or small group –  exchange emails, keep in touch –  form a support group for each other 6-3 6-4 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  87. 87. 6-Advice Dealing with high level management •  management support –  building good automation takes time and effort –  set realistic expectations •  benefits and ROI –  make benefits visible (charts on the walls) –  metrics for automation •  to justify it, compare to manual test costs over iterations •  on-going continuous improvement –  build cost, maintenance cost, failure analysis cost –  coverage of system tested 6-5 Dealing with developers •  critical aspect for successful automation –  automation is development •  may need help from developers •  automation needs development standards to work –  testability is critical for automatability –  why should they work to new standards if there is “nothing in it for them”? –  seek ways to cooperate and help each other •  run tests for them –  in different environments –  rapid feedback from smoke tests •  help them design better tests? 6-6 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  88. 88. 6-Advice Standards and technical factors •  standards for the testware architecture –  where to put things –  what to name things –  how to do things •  but allow exceptions if needed •  new technology can be great –  but only if the context is appropriate for it (e.g. Model-Based Testing) •  use automation “outside the box” 6-7 On-going automation •  you are never finished –  don’t “stand still” - schedule regular review and refactoring of the automation –  change tools, hardware when needed –  re-structure if your current approach is causing problems •  regular “pruning” of tests –  don’t have “tenured” test suites •  check for overlap, removed features •  each test should earn its place 6-8 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  89. 89. 6-Advice Information and web sites –  www.AutomatedTestingInstitute.com •  TestKIT Conference, Autumn, near Washington DC •  Test Automation certificate (TABOK) –  tool information •  commercial and open source: http://testertools.com •  open source tools –  www.opensourcetesting.org –  http://sourceforge.net –  http://riceconsulting.com (search on “cheap and free tools”) –  www.ISTQB.org •  Expert level in Test Automation (in progress) 6-9 1 2 3 4 5 Successful Test Automation 6 Summary: successful test automation •  assigned responsibility for automation tasks •  realistic, measured objectives (testing ≠ automation) •  technical factors – testware architecture, levels of abstraction, DSTL, scripting and comparison, pre and post processing •  management support, ROI, continuous improvement 6-10 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  90. 90. 6-Advice any more questions? please email me! info@DorothyGraham.co.uk Thank you for coming today I hope this will be useful for you All the best in your automation! 6-11 presented by Dorothy Graham info@dorothygraham.co.uk © Dorothy Graham 2013 www.DorothyGraham.co.uk
  91. 91. 32 BETTER SOFTWARE JULY/AUGUST 2009 www.StickyMinds.com
  92. 92. “Why automate?” This seems such an easy question to answer; yet many people don’t achieve the success they hoped for. If you are aiming in the wrong direction, you will not hit your target! This article explains why some testing objectives don’t work for automation, even though they may be very sensible goals for testing in general. We take a look at what makes a good test automation objective; then we examine six commonly held—but misguided— objectives for test execution automation, explaining the good ideas behind them, where they fail, and how these objectives can be modified for successful test automation. Good Objectives for Test Automation A good objective for test automation should have a number of characteristics. First of all, it should be measurable so that you can tell whether or not you have achieved it. Objectives for test automation should support testing activities but should not be the same as the objectives for testing. Testing and automation are different and distinct activities. Objectives should be realistic and achievable; otherwise, you will set yourself up for failure. It is better to have smaller-scale goals that can be met than far-reaching goals that seem impossible. Of course, many small steps can take you a long way! Automation objectives should be both short and long term. The shortterm goals should focus on what can be achieved in the next month or quarter. The long-term goals focus on where you want to be in a year or two. Objectives should be regularly revised in the light of experience. Misguided Objectives for Test Automation OBJECTIVE 1: FIND MORE BUGS Good ideas behind this objective: tomated testing should find them quicker. run more tests and find even more bugs. so we should also find bugs in the parts we weren’t able to test manually. Basing the success of automation on finding bugs—especially the automation of regression tests—is not a good thing to do for several reasons. First, it is the quality of the tests that determines whether or not bugs are found, and this has very little, if anything, to do with automation. Second, if tests are first run manually, any bugs will be found then, and they may be fixed by the time the automated tests are run. Finally, it sets an expectation that the main purpose of test automation is to find bugs, but this is not the case: A repeated test is much less likely to find a new bug than a new test. If the software is really good, automation may be seen as a waste of time and resources. Regression testing looks for unexpected, detrimental side effects in unchanged software. This typically involves running a lot of tests, many of which will not find any defects. This is ideal ground for test automation as it can significantly reduce the burden of this repetitive work, freeing the testers to focus on running manual tests where more defects are likely to be. It is the testing that finds bugs—not the automation. It is the testers who may be able to find more bugs, if the automation frees them from mundane repetitive work. The number of bugs found is a misleading measure for automation in any case. A better measure would be the percentage of regression bugs found (compared to a currently known total). This is known as the defect detection percentage (DDP). See the StickyNotes for more information. Sometimes this objective is phrased in a slightly different way: “Improve the quality of the software.” But identifying bugs does nothing to improve software—it is the fixing of bugs that improves the software, and this is a development task. If finding more bugs is something that you want to do, make it an objective for measuring the value of testing, not for measuring the value of automation. Better automation objective: Help teswww.StickyMinds.com ters find more regression bugs (so fewer regression failures occur in operation). This could be measured by increased DDP for regression bugs, together with a rating from the testers about how well the automation has supported their objectives. OBJECTIVE 2: RUN REGRESSION TESTS OVERNIGHT AND ON WEEKENDS Good ideas behind this objective: nings and weekends). “while we sleep.” At first glance, this seems an excellent objective for test execution automation, and it does have some good points. Once you have a good set of automated regression tests, it is a good idea to run the tests unattended overnight and on weekends, but resource use is not the most important thing. What about the value of the tests that are being run? If the regression tests that would be run “off peak” are really valuable tests, giving confidence that the main areas of the system are still working correctly, then this is useful. But the focus needs to be on supporting good testing. It is too easy to meet this stated objective by just running any test, whether it is worth running or not. For example, if you ran the same one test over and over again every night and every weekend, you would have achieved the goal as stated, but it is a total waste of time and electricity. In fact, we have heard of someone who did just this! (We think he left the company soon after.) Of course, automated tests can be run much more often, and you may want some evidence of the increased test execution. One way to measure this is using equivalent manual test effort (EMTE). For all automated tests, estimate how long it would have taken to run those tests manually (even though you have no intention of doing so). Then each time the test is run automatically, add that EMTE to your running total. Better automation objective: Run the most important or most useful tests, employing under-used computer resources when possible. This could be partially JULY/AUGUST 2009 BETTER SOFTWARE 33
  93. 93. measured by the increased use of resources and by EMTE, but should also include a measure of the value of the tests run, for example, the top 25 percent of the current priority list of most important tests (priority determined by the testers for each test cycle). OBJECTIVE 3: REDUCE TESTING STAFF Good ideas behind this objective: tool, so we should be able to save elsewhere. and staff costs are high. This is an objective that seems to be quite popular with managers. Some managers may go even further and think that the tool will do the testing for them, so they don’t need the testers—this is just wrong. Perhaps managers also think that a tool won’t be as argumentative as a tester! It is rare that staffing levels are reduced when test automation is introduced; on the contrary, more staff are usually needed, since we now need people with test script development skills in addition to people with testing skills. You wouldn’t want to let four testers go and then find that you need eight test automators to maintain their tests! Automation supports testing activities; it does not usurp them. Tools cannot make intelligent decisions about which tests to run, when, and how often. This is a task for humans able to assess the current situation and make the best use of the available time and resources. Furthermore, automated testing is not automatic testing. There is much work for people to do in building the automated tests, analyzing the results, and maintaining the testware. Having tests automated does—or at least should—make life better for testers. The most tedious and boring tasks are the ones that are most amenable for automation, since the computer will happily do repetitive tasks more consistently and without complaining. Automation can make test execution more efficient, but it is the testers who make the tests themselves effective. We have yet to see a tool that can think up tests as well as a human being can! 34 BETTER SOFTWARE JULY/AUGUST 2009 The objective as stated is a management objective, not an appropriate objective for automation. A better management objective is “Ensure that everyone is performing tasks they are good at.” This is not an automation objective either, nor is “Reducing the cost of testing.” These could be valid objectives, but they are related to management, not automation. Better automation objective: The total cost of the automation effort should be significantly less than the total testing effort saved by the automation. This could be partially measured by an increase in tests run or coverage achieved per hour of human effort. OBJECTIVE 4: REDUCE ELAPSED TIME FOR TESTING Good ideas behind this objective: way we can save time is good. testing will help overall. This one seems very sensible at first and sometimes it is even quantified— “Reduce elapsed time by X%”—which sounds even more impressive. However, this objective can be dangerous because of confusion between “testing” and “test execution.” The first problem with this objective is that there are much easier ways to achieve it: run fewer tests, omit long tests, or cut regression testing. These are not good ideas, but they would achieve the objective as stated. The second problem with this objective is its generality. Reducing the elapsed time for “testing” gives the impression we are talking about reducing the elapsed time for testing as a whole. However, test execution automation tools are focused on the execution of the tests (the clue is in the name!) not the whole of testing. The total elapsed time for testing may be reduced only if the test execution time is reduced sufficiently to make an impact on the whole. What typically happens, though, is that the tests are run more frequently or more tests are run. This can result in more bugs being found (a good thing), that take time to fix (a fact of life), and www.StickyMinds.com increase the need to run the tests again (an unavoidable consequence). The third problem is that there are many factors other than execution that contribute to the overall elapsed time for testing: How long does it take to set up the automated run and clear up after it? How long does it take to recognize a test failure and find out what is actually wrong (test fault, software fault, environment problem)? When you are testing manually, you know the context—you know what you have done just before the bug occurs and what you were doing in the previous ten minutes. When a tool identifies a bug, it just tells you about the actual discrepancy at that time. Whoever analyzes the bug has to put together the context for the bug before he or she can really identify the bug. In figures 1 and 2, the blocks represent the relative effort for the different activities involved in testing. In manual testing, there is time taken for editing tests, maintenance, set up of tests, executing the tests (the largest component of manual testing), analyzing failures, and clearing up after tests have completed. In figure 1, when those same tests are automated, we see the illusion that automating test execution will save us a lot of time, since the relative time for execution is dramatically reduced. However, figure 2 shows us the true picture— total elapsed time for testing may actually increase, even though the time for test execution has been reduced. When test automation is more mature, then the total elapsed time for all of the testing activities may decrease below what it was initially for manual testing. Note that this is not to scale; the effects may be greater than we have illustrated. We now can see that the total elapsed time for testing depends on too many things that are outside the control or influence of the test automator. The main thing that causes increased testing time is the quality of the software—the number of bugs that are already there. The more bugs there are, the more often a test fails, the more bug reports need to be written up, and the more retesting and regression testing are needed. This has nothing to do with whether or not the tests are automated or manual, and the quality of the software
  94. 94. is the responsibility of the developers, not the testers or the test automators. Finally, how much time is spent maintaining the automated tests? Depending on the test infrastructure, architecture, or framework, this could add considerably to the elapsed time for testing. Maintenance of the automated tests for later versions of the software can consume a lot of effort that also will detract from the savings made in test execution. This is particularly problematic when the automation is poorly implemented, without thought for maintenance issues when designing the testware architecture. We may achieve our goal with the first release of software, but later versions may fail to repeat the success and may even become worse. Here is how the automator and tester should work together: The tester may request automated support for things that are difficult or time consuming, for example, a comparison or ensuring that files are in the right place before a test runs. The automator would then provide utilities or ways to do them. But the automator, by observing what the tester is doing, may suggest other things that could be supported and “sell” additional tool support to the tester. The rationale is to make life easier for the tester and to make the testing faster, thus reducing elapsed time. Better automation objective: Reduce the elapsed time for all tool-supported Figure 1 Figure 2 www.StickyMinds.com testing activities. This is an ongoing objective for automation, seeking to improve both manual and existing automated testing. It could be measured by elapsed time for specified testing activities, such as maintenance time or failure analysis time. OBJECTIVE 5: RUN MORE TESTS Good ideas behind this objective: better coverage. must be better. More is not better! Good testing is not found in the number of tests run, but in the value of the tests that are run. In fact, the fewer tests for the same value, the better. It is definitely the quality of the tests that counts, not the quantity. Automating a lot of poor tests gives you maintenance overhead with little return. Automating the best tests (however many that is) gives you value for the time and money spent in automating them. If we do want to run more tests, we need to be careful when choosing which additional tests to run. It may be easier to automate tests for one area of the software than for another. However, if it is more valuable to have automated tests for this second area than the first, then automating a few of the more difficult tests is better than automating many of the easier (and less useful) tests. A raw count of the number of automated tests is a fairly useless way of gauging the contribution of automation to testing. For example, suppose testers decide there is a particular set of tests that they would like to automate. The real value of automation is not that the tests are automated but the number of times they are run. It is possible that the testers make the wrong choice and end up with a set of automated tests that they hardly ever use. This is not the fault of the automation, but of the testers’ choice of which tests to automate. It is important that automation is responsive, flexible, and able to automate different tests quickly as needed. Although we try to plan which tests to automate and when, we should always start automating the most important tests first. Once we are running the tests, JULY/AUGUST 2009 BETTER SOFTWARE 35

×