Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
THE LOW-HANGING FRUIT
OF TEST AUTOMATION -
WHAT DO WE AUTOMATE
FIRST?
RANDALL RICE, CTAL
WWW.RICECONSULTING.COM
March 19, ...
2
BIO - RANDALL W. RICE
•  Over 35 years experience in building and testing
information systems in a variety of industries...
3
FEASIBILITY
1.  Is it easy (or even feasible) to automate the function/
component?
•  Problematic issues
•  Custom objec...
4
CONTROL
2. Who controls/develops the application?
•  Are any components (or even the entire application) vendor-
develop...
5
OTHER QUESTIONS
3. Is the function easily repeatable?
•  Do you feel like a robot testing it?
4. Is the function easily ...
6
NUMBER OF TESTS
Timeperfunctionalexecution
Number of times executed per test
Long term ROI (not so good)
7
TEST FREQUENCY
Timepertestsuiteexecution
Number of times tested per day, week, etc.
Good choice
Long term ROI (not so go...
8
REAL-LIFE EXAMPLE
•  A financial services client had a simple
function that took about 5 minutes per
condition to test.
...
9
HELPFUL TEST
AUTOMATION METRICS (2)
•  Effort saved by test automation (manual vs. automation)
•  Effort needed to autom...
10
EQUIVALENT MANUAL
TEST EFFORT
•  Also known as EMTE
•  How much time would it take if the automated tests were
performe...
11
EMTE EXAMPLE
•  Automated test takes 1 hour to run and are performed for
10 cycles.
•  Manual version of test takes 4 d...
12
RETURN ON
INVESTMENT (ROI)
Time
Effort
Spent on
Automating
Tests
Development
Effort on
Production
Code
Initial
Effort
I...
13
RETURN ON INVESTMENT (2)
- NOT SO GOOD
Time
Effort
Spent on
Automating
Tests
Development
Effort on
Production
Code
Init...
14
MORE QUESTIONS
7. Does the function require:
•  Human judgment for outcomes?
•  Creative types of testing
•  “Off the s...
15
NEW OR MAINTENANCE?
9. Is this new software or maintenance?
•  Great debate: When should the automation be created –
du...
16
WHO AND WHEN?
10. Who will create and use the automation?
•  Simple capture/playback (when it works) can be created by
...
17
NEED FOR PRECISION
12. What is the need for precision?
•  Exact regression testing requires
automation
•  One mistake c...
18
KNOWLEDGE AND
STABILITY
13. Do you have a sufficient way to compare the test results
for pass/fail?
•  You must have a ...
19
20
CONTACT INFORMATION
Randall W. Rice, CTAL
Rice Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73170
Ph: 40...
Upcoming SlideShare
Loading in …5
×

What Do We Automate First

768 views

Published on

Randy Rice of http://www.riceconsulting.com presents 14 questions you can ask to know if a software application or function is ripe for automation. Also presented are metrics for test automation ROI.

Published in: Technology
  • Hi Derk-Jan, Thanks for your positive comments. I get the question often, so I thought it might be good to offer some of my own criteria.

    Thanks,

    Randy
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi Randy, Nice overview of factors that determine whether a function, system or test is kandidate to be automated first. Low hanging fruit is important, I agree. So thanks for sharing some way to easily way to identify them and enable the pick
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

What Do We Automate First

  1. 1. THE LOW-HANGING FRUIT OF TEST AUTOMATION - WHAT DO WE AUTOMATE FIRST? RANDALL RICE, CTAL WWW.RICECONSULTING.COM March 19, 2014
  2. 2. 2 BIO - RANDALL W. RICE •  Over 35 years experience in building and testing information systems in a variety of industries and technical environments •  ASTQB Certified Tester – Foundation level, Advanced level (Full) •  Director, American Software Testing Qualification Board (ASTQB) •  Chairperson, 1995 - 2000 QAI’s annual software testing conference •  Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems •  Principal Consultant and Trainer, Rice Consulting Services, Inc. •  www.riceconsulting.com & randallrice.blogspot.com
  3. 3. 3 FEASIBILITY 1.  Is it easy (or even feasible) to automate the function/ component? •  Problematic issues •  Custom objects •  Non-standard objects •  Objects with dynamic attributes (session IDs, etc.) •  Streaming, dynamic data (think stock tickers and other data that is changing even while you are testing it) •  The tool doesn’t handle a certain type of object or component.
  4. 4. 4 CONTROL 2. Who controls/develops the application? •  Are any components (or even the entire application) vendor- developed? •  How stable is the code? •  Are there a lot of hotfixes or maintenance releases •  If so, do you know when they occur? •  Do you have prior knowledge of changes? •  Some vendors have problems in creating and distributing release notes, even to their own testers.
  5. 5. 5 OTHER QUESTIONS 3. Is the function easily repeatable? •  Do you feel like a robot testing it? 4. Is the function easily performed? •  Does it take a lot of set-up? •  Are there a lot of steps to perform? •  Long scripts are hard to maintain? 5. How frequently is it used? •  X times per day (or hour) 6. How long does the function take to test manually, with all test conditions?
  6. 6. 6 NUMBER OF TESTS Timeperfunctionalexecution Number of times executed per test Long term ROI (not so good)
  7. 7. 7 TEST FREQUENCY Timepertestsuiteexecution Number of times tested per day, week, etc. Good choice Long term ROI (not so good) Much depends on the length of the test. 5 test conditions vs. 500 obviously makes a big Difference!
  8. 8. 8 REAL-LIFE EXAMPLE •  A financial services client had a simple function that took about 5 minutes per condition to test. •  However, it took the tester all day to test it due to the number of conditions. •  We spent 3 days to automate it, with the result being 15 minutes to test all conditions!
  9. 9. 9 HELPFUL TEST AUTOMATION METRICS (2) •  Effort saved by test automation (manual vs. automation) •  Effort needed to automate new tests •  Effort needed to analyze failed automated tests •  Effort to maintain automated tests •  Test coverage achieved by automated tests
  10. 10. 10 EQUIVALENT MANUAL TEST EFFORT •  Also known as EMTE •  How much time would it take if the automated tests were performed manually?
  11. 11. 11 EMTE EXAMPLE •  Automated test takes 1 hour to run and are performed for 10 cycles. •  Manual version of test takes 4 days to run. •  EMTE = 40 days •  4 days X 10 cycles
  12. 12. 12 RETURN ON INVESTMENT (ROI) Time Effort Spent on Automating Tests Development Effort on Production Code Initial Effort Increased Effort (Hump) Reduced Effort Source: xUnit Patterns by Gerald Meszaros Saved Effort
  13. 13. 13 RETURN ON INVESTMENT (2) - NOT SO GOOD Time Effort Spent on Automating Tests Development Effort on Production Code Initial Effort Increased Effort (Hump) Ongoing Effort Source: xUnit Patterns by Gerald Meszaros Saved Effort
  14. 14. 14 MORE QUESTIONS 7. Does the function require: •  Human judgment for outcomes? •  Creative types of testing •  “Off the scripted path?” 8. Are you looking for new defects? •  Test automation is more confirmatory than discovery in nature. •  i.e., the defects found by automation are normally regression- type defects.
  15. 15. 15 NEW OR MAINTENANCE? 9. Is this new software or maintenance? •  Great debate: When should the automation be created – during the project or after? •  Since functional changes ripple through automation, great care must be taken not to automate too early. •  However, automation can be helpful during the final phases of a project to speed up regression testing. •  Keep in mind it takes time to create test automation and the distraction can actually turn into a project risk!
  16. 16. 16 WHO AND WHEN? 10. Who will create and use the automation? •  Simple capture/playback (when it works) can be created by testers. •  When problems are encountered, technical help at the developer level is often needed. 11. Which phase of testing? •  Maintenance testing is ideal •  UAT is not good (you want humans testing) •  System testing is a possible win •  Unit test automation is ideal and essential
  17. 17. 17 NEED FOR PRECISION 12. What is the need for precision? •  Exact regression testing requires automation •  One mistake can invalidate the entire test •  Otherwise, you are doing “pseudo- regression” testing •  What is the risk? •  If exact testing is needed, such as medical devices, avionics, etc., automation can help mitigate risks •  WARNING: test automation is “software testing software”, therefore the test scripts may have bugs
  18. 18. 18 KNOWLEDGE AND STABILITY 13. Do you have a sufficient way to compare the test results for pass/fail? •  You must have a source (non-human) 14. How stable is the function to be tested? •  This is an irony •  Functional errors (which you want to find) will cause the scripts to fail •  However, they will make test automation difficult because you will need a way to deal with errors without stopping other tests in the suite.
  19. 19. 19
  20. 20. 20 CONTACT INFORMATION Randall W. Rice, CTAL Rice Consulting Services, Inc. P.O. Box 892003 Oklahoma City, OK 73170 Ph: 405-691-8075 Fax: 405-691-1441 Web site: www.riceconsulting.com e-mail: rrice@riceconsulting.com

×