• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Automation beyond testing tools
 

Automation beyond testing tools

on

  • 421 views

 

Statistics

Views

Total Views
421
Views on SlideShare
421
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Automation beyond testing tools Automation beyond testing tools Document Transcript

    • STEP AUTO Colloquium on Test 12/10/2007Automation ,6th Dec 07 Automation beyond testing tools Srinivasan Desikan © 2007 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice Question How many programmers does it take to change a light bulb? Automation many times gets the wrong set of people for the job • Either a test engineer who can’t program • Or a Programmer who can’t test • Or a person who is neither programmer nor a tester, given to automation saying “Automate if you have nothing else to do” December 10, 2 2007Automation beyond Testing Tools ,Srinivasan Desikan -HP 1
    • STEP AUTO Colloquium on Test 12/10/2007Automation ,6th Dec 07 Automation Survey – Why you are not automating? No belief Not aw are 0 6 Inadequate Features 33 Expensive 61 The job of automation include understanding where tools can help and where it can’t… The job of automation starts in creating value on top of commercially available tools as any tool will not meet 100% of your requirements… December 10, 3 2007 Automation Survey – What did you automate? Easy 10 Suitable 39 Regression 28 All 23 • Only Low-hanging fruits are plugged • 23% got into “trap” as • There is no end to testing ; so 100% is not possible • There may not be ROI if all are automated (Waste of effort) • We have a long way to go to say “we are automating complex test cases” that gives us great business value December 10, 4 2007Automation beyond Testing Tools ,Srinivasan Desikan -HP 2
    • STEP AUTO Colloquium on Test 12/10/2007Automation ,6th Dec 07 Automation Survey – What tools you used? OS Features 35 In house 45 Open source 42 COTS 78 0 20 40 60 80 100 • 35+45+42+78 = 180%; Therefore people on an average use 1.8 tools which is not a good number to get a good coverage on automation • There is heavy dependence on COTS and awareness and usage of Open source/in-house tools to improve December 10, 5 2007 Automation Survey – Difficulties in automation More efforts 29 25 More time 17 35 Skilled people 48 45 Usage 17 0 10 20 30 40 50 60 • Automation is kept in the freezer because it requires more efforts and more complex • Lack of training available on automation. Lack of usage knowledge infact is a great difficulty in automation • Eventhough industry produced the required numbers for automation, the skills of people has a long way to go (Quantity Vs Quality) December 10, 6 2007Automation beyond Testing Tools ,Srinivasan Desikan -HP 3
    • STEP AUTO Colloquium on Test 12/10/2007Automation ,6th Dec 07 Automation Survey – Satisfied with automation No 0 Not yet High 29 27 Just 44 • Only 27% of people satisfied with automation ; More we do automation more we get satisfied with it • This percentage drops further below if we exclude the people we were part of the automation project ; Business value out of automation as perceived by mgmt and outside testing team is very low December 10, 7 2007 Automation Survey – Quality of automation • Only 2% of people share their automation suite with people outside the team (they are not easy to use, and have potential quality issues) • 95% of people say that “They repeat the tests manually when automated tests bring up defects”; this shows − Trust level of test suite is very low − It defeats the purpose of having automated the tests − More needs to be done to make the quality of test suite better − Quality requirements for automation should be better than or equal to those criteria set for products • Only 1% people share their test suite with their customers − Automation is not done with the perspective that “customers also do testing and it should save their time too” − Not many realize that automation is yet another business opportunity (Many of the commercial tools in market today were in-house tools before) December 10, 8 2007Automation beyond Testing Tools ,Srinivasan Desikan -HP 4
    • STEP AUTO Colloquium on Test 12/10/2007Automation ,6th Dec 07 Automation beyond testing tools & Test cases • Test reports are automatically prepared by automation scripts • Test defects are automatically filed in defect tracking system • Test metrics are updated/created by automation scripts • The test results are updated automatically by automation • Traceability matrix updated with mapping to requirements and test cases • Keep looking at defect status and selecting test cases automatically for regression testing (Keeping the track of what defects corresponds to what test cases) • Calculating code coverage and test coverage • Suggesting the improvements in test management − A list of test cases that were never executed for a long time − A list of test cases that always failed (Then why execute them again and again) − A list of test cases that was not executed for long (to execute them atleast now or delete them from database) − December 10, test cases that always passed (Do we need to really execute them A list of 9 again and again) 2007 Survey results – Where test engineers spend time? 5% 5% 10% Automation 5% Test execution 15% 10% Test Setup Reports Waiting for build test case design 10% Test plan 5% Review s 35% Others • Very little investment on automation • Automation in lab setup will give better ROI December 10, 10 2007Automation beyond Testing Tools ,Srinivasan Desikan -HP 5
    • STEP AUTO Colloquium on Test 12/10/2007Automation ,6th Dec 07 References 1. Test automation survey results – Jayakumar. K.R, AmitySoft, Chennai (Necessary approvals obtained) 2. Book: “Software Testing – Principles and Practices”, Srinivasan Desikan & Ramesh Gopalaswamy, ISBN: 817758295X Disclaimer: The views expressed in slides and in the presentation are my own and not that of my employer December 10, 11 2007 Questions? December 10, 12 2007Automation beyond Testing Tools ,Srinivasan Desikan -HP 6
    • STEP AUTO Colloquium on Test 12/10/2007Automation ,6th Dec 07 Thank You ! December 10, 13 2007Automation beyond Testing Tools ,Srinivasan Desikan -HP 7