What Do We Automate First
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

What Do We Automate First

on

  • 396 views

Randy Rice of http://www.riceconsulting.com presents 14 questions you can ask to know if a software application or function is ripe for automation. Also presented are metrics for test automation ROI.

Randy Rice of http://www.riceconsulting.com presents 14 questions you can ask to know if a software application or function is ripe for automation. Also presented are metrics for test automation ROI.

Statistics

Views

Total Views
396
Views on SlideShare
396
Embed Views
0

Actions

Likes
0
Downloads
14
Comments
2

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Hi Derk-Jan, Thanks for your positive comments. I get the question often, so I thought it might be good to offer some of my own criteria.

    Thanks,

    Randy
    Are you sure you want to
    Your message goes here
    Processing…
  • Hi Randy, Nice overview of factors that determine whether a function, system or test is kandidate to be automated first. Low hanging fruit is important, I agree. So thanks for sharing some way to easily way to identify them and enable the pick
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

What Do We Automate First Presentation Transcript

  • 1. THE LOW-HANGING FRUIT OF TEST AUTOMATION - WHAT DO WE AUTOMATE FIRST? RANDALL RICE, CTAL WWW.RICECONSULTING.COM March 19, 2014
  • 2. 2 BIO - RANDALL W. RICE •  Over 35 years experience in building and testing information systems in a variety of industries and technical environments •  ASTQB Certified Tester – Foundation level, Advanced level (Full) •  Director, American Software Testing Qualification Board (ASTQB) •  Chairperson, 1995 - 2000 QAI’s annual software testing conference •  Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems •  Principal Consultant and Trainer, Rice Consulting Services, Inc. •  www.riceconsulting.com & randallrice.blogspot.com
  • 3. 3 FEASIBILITY 1.  Is it easy (or even feasible) to automate the function/ component? •  Problematic issues •  Custom objects •  Non-standard objects •  Objects with dynamic attributes (session IDs, etc.) •  Streaming, dynamic data (think stock tickers and other data that is changing even while you are testing it) •  The tool doesn’t handle a certain type of object or component.
  • 4. 4 CONTROL 2. Who controls/develops the application? •  Are any components (or even the entire application) vendor- developed? •  How stable is the code? •  Are there a lot of hotfixes or maintenance releases •  If so, do you know when they occur? •  Do you have prior knowledge of changes? •  Some vendors have problems in creating and distributing release notes, even to their own testers.
  • 5. 5 OTHER QUESTIONS 3. Is the function easily repeatable? •  Do you feel like a robot testing it? 4. Is the function easily performed? •  Does it take a lot of set-up? •  Are there a lot of steps to perform? •  Long scripts are hard to maintain? 5. How frequently is it used? •  X times per day (or hour) 6. How long does the function take to test manually, with all test conditions?
  • 6. 6 NUMBER OF TESTS Timeperfunctionalexecution Number of times executed per test Long term ROI (not so good)
  • 7. 7 TEST FREQUENCY Timepertestsuiteexecution Number of times tested per day, week, etc. Good choice Long term ROI (not so good) Much depends on the length of the test. 5 test conditions vs. 500 obviously makes a big Difference!
  • 8. 8 REAL-LIFE EXAMPLE •  A financial services client had a simple function that took about 5 minutes per condition to test. •  However, it took the tester all day to test it due to the number of conditions. •  We spent 3 days to automate it, with the result being 15 minutes to test all conditions!
  • 9. 9 HELPFUL TEST AUTOMATION METRICS (2) •  Effort saved by test automation (manual vs. automation) •  Effort needed to automate new tests •  Effort needed to analyze failed automated tests •  Effort to maintain automated tests •  Test coverage achieved by automated tests
  • 10. 10 EQUIVALENT MANUAL TEST EFFORT •  Also known as EMTE •  How much time would it take if the automated tests were performed manually?
  • 11. 11 EMTE EXAMPLE •  Automated test takes 1 hour to run and are performed for 10 cycles. •  Manual version of test takes 4 days to run. •  EMTE = 40 days •  4 days X 10 cycles
  • 12. 12 RETURN ON INVESTMENT (ROI) Time Effort Spent on Automating Tests Development Effort on Production Code Initial Effort Increased Effort (Hump) Reduced Effort Source: xUnit Patterns by Gerald Meszaros Saved Effort
  • 13. 13 RETURN ON INVESTMENT (2) - NOT SO GOOD Time Effort Spent on Automating Tests Development Effort on Production Code Initial Effort Increased Effort (Hump) Ongoing Effort Source: xUnit Patterns by Gerald Meszaros Saved Effort
  • 14. 14 MORE QUESTIONS 7. Does the function require: •  Human judgment for outcomes? •  Creative types of testing •  “Off the scripted path?” 8. Are you looking for new defects? •  Test automation is more confirmatory than discovery in nature. •  i.e., the defects found by automation are normally regression- type defects.
  • 15. 15 NEW OR MAINTENANCE? 9. Is this new software or maintenance? •  Great debate: When should the automation be created – during the project or after? •  Since functional changes ripple through automation, great care must be taken not to automate too early. •  However, automation can be helpful during the final phases of a project to speed up regression testing. •  Keep in mind it takes time to create test automation and the distraction can actually turn into a project risk!
  • 16. 16 WHO AND WHEN? 10. Who will create and use the automation? •  Simple capture/playback (when it works) can be created by testers. •  When problems are encountered, technical help at the developer level is often needed. 11. Which phase of testing? •  Maintenance testing is ideal •  UAT is not good (you want humans testing) •  System testing is a possible win •  Unit test automation is ideal and essential
  • 17. 17 NEED FOR PRECISION 12. What is the need for precision? •  Exact regression testing requires automation •  One mistake can invalidate the entire test •  Otherwise, you are doing “pseudo- regression” testing •  What is the risk? •  If exact testing is needed, such as medical devices, avionics, etc., automation can help mitigate risks •  WARNING: test automation is “software testing software”, therefore the test scripts may have bugs
  • 18. 18 KNOWLEDGE AND STABILITY 13. Do you have a sufficient way to compare the test results for pass/fail? •  You must have a source (non-human) 14. How stable is the function to be tested? •  This is an irony •  Functional errors (which you want to find) will cause the scripts to fail •  However, they will make test automation difficult because you will need a way to deal with errors without stopping other tests in the suite.
  • 19. 19
  • 20. 20 CONTACT INFORMATION Randall W. Rice, CTAL Rice Consulting Services, Inc. P.O. Box 892003 Oklahoma City, OK 73170 Ph: 405-691-8075 Fax: 405-691-1441 Web site: www.riceconsulting.com e-mail: rrice@riceconsulting.com